**Algorithm 2** Differentiable equivariance-aware neural architecture search.

**procedure** EQUINAS<sub>D</sub> (Set of groups [G])
    Initialize network with [G]-mixed equivariant layers, parameterized by Ψ and Z.
    **while** not converged **do**
        Update Z by ∇<sub>Z</sub> L(Ψ, Z).
        Update Ψ by ∇<sub>Ψ</sub> L(Ψ, Z).
    **return** trained network

4.4.1 Evolutionary Equivariance-aware NAS

Towards finding the optimal full equivariance per layer, the equivariance relaxation morphism pre-
sented in Section 4.3.1 is applied as the genetic operator in an evolutionary hill-climbing algorithm.
The Evolutionary Equivariance-Aware NAS (EquiNAS<sub>E</sub>) algorithm, given in Algorithm 1, is similar to
other evolutionary NAS methods such as Elsken et al. [71] with pareto selection as in Falanti et al. [212].
A population of networks, which starts with an individual with all layers equivariant to the largest
possible group, undergoes mutation via equivariance relaxation and selection based on accuracy and
parameter count to optimize neural architecture while learning network parameters.

In each generation, candidate networks are evaluated based on maximizing validation accuracy and minimizing parameter count: the entire Pareto front is kept, then additional high-accuracy individuals are added if necessary until the desired parent population size is reached. Offspring are generated from each parent separately through mutation using the relaxation morphism. This preserves the weights of the parameterized equivariance during mutation, allowing for the continuous training of networks over evolution through inheritance from parent individuals. Specifically, mutation reduces a single layer's parameterized equivariance to a subgroup within the constraint that each layer has parametrized equivariance to a subgroup of all preceding layers. This constraint yields local equivariance properties for the network, as shown in Weiler and Cesa [200] and Elsayed et al. [213] to be empirically favorable in image classification tasks. The resulting individuals are each trained independently for a given training time, and then this process repeats.

The second objective of minimizing parameter count is intended to advance efficient networks,
such as those with large symmetry groups. Accuracy-based selection alone would necessarily prefer
larger networks as mutation via the equivariance relaxation morphism results in two networks with
identical performance but different size, the relaxed network having more parameters, until training;
potentially short-term increases in validation accuracy after training would then result in the selection
of individuals with more parameters. Thus, the proposed strategy of selecting both pareto-front
and high-accuracy individuals is intended to maintain a diverse yet efficient population without
succumbing to overly greedy selections too early.

4.4.2 Differentiable Equivariance-aware NAS

In a contrasting paradigm, the [G]-mixed equivariant layer presented in Section 4.3.2 allows for
smoothly searching across a spectrum of equivariance for each layer via a differentiable NAS algorithm.
Our Differentiable Equivariance-Aware NAS (EquiNAS<sub>D</sub>) algorithm, defined in Algorithm 2, is inspired
by DARTS [19] with significant changes detailed in the following paragraphs. EquiNAS<sub>D</sub> simplifies the
bilevel optimization of the architecture weighting parameters Z and kernel weights Ψ into alternating
independent updates, computing the gradient update for Z with the current, rather than optimal, Ψ
for the current architecture encoded by Z, to boost search efficiency with minimal performance loss
compared to higher order approximations [19].

In most differentiable NAS search spaces, the desired output architecture is discretized to select a subset
of architectural options within constraints, then the weights are re-initialized and trained within the
static architecture. In our formulation, this is not necessary as any mixed operation can be equivalently
expressed as a single layer equivariant to any group G' that is a common subgroup to all groups of
the mixed operation (Eq. 12): in our experimental case, this is a standard translation-equivariant convolutional layer, so the final model can be equivalently expressed as a standard convolutional model with encoded partial equivariance. Thus, the final optimized architecture and trained weights are output from the single search process. We explore the standard NAS paradigm, where only the architecture is output and reused for evaluation such that the weights are retrained in this static architecture, in further experiments.

In order to enforce that the scaling of each kernel does not confound the architecture weighting parameters, we use the weight normalizing reparameterization [214] and do not update the scalar norm parameter of each kernel after initialization.

We do not use disjoint datasets for updating $\Psi$ and $Z$, but rather draw one batch for $\Psi$ and another for $Z$ independently and randomly from the same training split. This allows for a standard dataset split and to use the validation set for hyperparameter tuning.

These two NAS approaches present adaptations of two standard types of NAS, evolutionary and differentiable, to the search for optimal partial equivariance. The equivariance relaxation morphism and the [G]-mixed equivariant layer that enable the evolutionary and differentiable search methods respectively are the main focus; the other characteristics of the NAS methods are adapted from existing methods, but further study could advance specialization of equivariance-aware NAS. We next study empirically the two EquiNAS methods on three datasets, one with known rotational symmetry and two with unknown but visually significant rotational and reflectional symmetry.

## 4.5 Experiments

We focus on the regular representation of groups and show experiments with reflectional and up to 4-fold rotational symmetry groups applied to image classification tasks. Examples of symmetry groups acting on pixel space, which corresponds to $Z^2$, include $T(2)$, which consists of discrete translations in both dimensions; the cyclical groups $C_n$, which consist of n-fold rotations; and the dihedral groups $D_n$, which consist of reflections with n-fold rotations, where $n \in \{1, 2, 4\}$ for exact symmetry without interpolation. The $p4$ group consists of discrete translations and multiples of $90^\circ$ rotations and may be represented as $T(2) \times C_4$. The $p4m$ group consists of discrete translations, reflections, and multiples of $90^\circ$ rotations and may be represented as $T(2) \times D_4$. As standard convolutional layers are already equivariant to $T(2)$, we refer to layers also equivariant to n-fold rotations with or without reflections as $D_n$ or $C_n$-equivariant, respectively. So, a $C_1$ equivariant convolutional layer is a standard translation-equivariant convolutional layer. We use $\{C_1, D_1, C_2, D_2, C_4, D_4\}$ as the set of potential groups for mutation in EquiNAS$_E$ and as [G] in EquiNAS$_D$.

We present experiments on image classification for a variety of datasets. The Rotated MNIST dataset [215, rotMNIST] is a version of the MNIST handwritten digit dataset but with the images rotated by any angle. This task serves as a simple investigational study with known symmetry, while the following two tasks are more realistic and complex. The Galaxy10 DECals dataset [216, Galaxy10] contains galaxy images in 10 broad categories. The ISIC 2019 dataset [217–219, ISIC] contains dermascopic images of 8 types of skin cancer plus a null class. For Galaxy10 and ISIC, we down-sample the images to 64 × 64 due to computational constraints, which adds notable difficulty to the tasks. These tasks exhibit varying levels of rotational and reflectional symmetry, motivating architectural optimization to determine the most effective application of equivariance constraints.

Across all experiments, the architectures are designed to have consistent channel dimensions once expanded to a standard translation-equivariant convolutional layer for each layer across models. Thus, constrained equivariance to a larger symmetry group results in fewer learnable parameters. A layer constrained to $C_4$ equivariance has $|C_4 \setminus D_4| = 2$ times as many independent channels and as many
parameters as a layer constrained to $D_4$ equivariance. This is a notably different paradigm than other
works that equate parameter counts across architectures with different equivariance properties.

As baseline comparisons, we train and test G-CNNs with static architectures. In addition to the static baselines, we re-implement the residual pathway priors (RPP) approach by Finzi et al. [202] as a $C_1$ equivariant layer with regularization in parallel with a $D_4$ equivariant convolutional layer.

**Architecture backbone** For both EquiNAS<sub>E</sub> and EquiNAS<sub>D</sub> experiments, we use the same backbone architecture, such that the static baselines have the same architecture across experiments. The architectures have a lifting layer followed by 7 group convolutional layers, for a total of 8 convolutional layers. After 4 layers, the channel count doubles, from 16 to 32 for a $D_4$ equivariant layer and scaling up for smaller symmetry group equivariance constraints. An average pooling layer is placed after every other layer for all architectures and additionally after the fifth and seventh convolutional layers for Galaxy10 and ISIC. After the final group convolutional layer is a group-dimension average pooling followed by two linear layers to the output dimension. Every convolutional and linear layer except the output layer is immediately followed by a batchnorm then a ReLU.

**Hyperparameters** The hyperparameters for each algorithm are selected such that baselines only differ by training time and optimizers. The learning rates were selected by grid search over baselines on rotMNIST. For all experiments in Section 4.5.1, we use a simple SGD optimizer with learning rate 0.1 to avoid confounding effects such as momentum during the morphism. For EquiNAS<sub>E</sub>, the parent selection size is 5, the training time per generation is 0.5 epochs, and the number of generations is 50 for all tasks. Baselines were trained for the equivalent number of epochs. For all experiments in Section 4.5.2, we use separate Adam optimizers for Ψ and Z, each with a learning rate of 0.01 and otherwise default settings. The total training time is 100 epochs for rotMNIST and 50 epochs for Galaxy10 and ISIC. For RPP, we use a $C_1$-equivariant layer with an L2 regularization parameter of $1 \times 10^{-6}$ in parallel with a $D_4$-equivariant layer without regularization.

For rotMNIST, we use the standard training and test split with a batch size of 64, reserving 10% of the training data as the validation set. For Galaxy10, we set aside 10% of the dataset as the test set, reserving 10% of the remaining training data as the validation set. For ISIC, we set aside 10% of the available training dataset as the test set, reserving 10% of the remaining training data as the validation. For the latter two datasets, we resize the images to 64 × 64 due to computational constraints and use a batchsize of 32. The validation sets were previously used for hyperparameter tuning: for experimental results, they are only used for the experiments in Section 4.5.1 as necessary for the EquiNAS<sub>E</sub> algorithm. No data augmentation is performed, although the datasets are normalized.

**Ablations and Random Search** We implement two kinds of random search for each NAS method. The first ablates smart architecture search: EquiNAS<sub>E</sub> Random Select works as described in Algorithm 1 but with random parent selection (instead of pareto-front selection) and EquiNAS<sub>D</sub> Random Z works as described in Algorithm 2 but with random Z updates (instead of gradient descent) by shuffling Z gradients. The second is more akin to standard NAS random search: for the evolutionary paradigm, we train 30 randomly selected static architectures in the discrete architecture search space for the same training time and selecting the top 5 by validation accuracy, and for the differentiable paradigm, we train 25 randomly selected static architectures in the continuous architecture search space and selecting the top 5 by validation accuracy. 30 and 25 were respectively calculated to be approximately the same compute cost as the trials of EquiNAS<sub>E</sub> and EquiNAS<sub>D</sub>. These are labeled as "Random Static" for both the evolutionary and differentiable paradigms. Since Random Static trains static architectures while EquiNAS<sub>E</sub> and EquiNAS<sub>D</sub> dynamically search for both architectures and parameters, we take the best 5 architectures for each and retrain their parameters from scratch as in the standard NAS
paradigm, labeled as EquiNAS<sub>E</sub> Retrain and EquiNAS<sub>D</sub> Retrain, respectively, for fair comparison to
Random Static.

For each paradigm of experiments, we present their results in the following subsections, with general discussion following in Section 4.6.

**4.5.1 Evolutionary Equivariance-aware NAS**

The classification test errors are listed in Table 3 and visualized in Figure 9. The advantages of equivari-
ance search methods are most apparent in the Galaxy10 benchmark. While EquiNAS<sub>E</sub> outperforms
most baselines on rotMNIST and all baselines on ISIC, it has similar performance on both tasks to the
D<sub>4</sub> baseline, and some of the final architectures are very similar to the D<sub>4</sub> baseline architecture. How-
ever, the D<sub>4</sub> baseline fails at the Galaxy10 task, demonstrating that the same equivariant architecture
can not be naively applied to different tasks. Both search methods, EquiNAS<sub>E</sub> and RPP, outperform all
baseline models on Galaxy10, and by a large margin for EquiNAS<sub>E</sub>.

The evolutionary progress for the trial on each task is shown in Figure 11: the selected population maintains a fully equivariant network in every generation, except for the final generation in both Galaxy10 and ISIC. For RotMNIST, the final selected population originates from two main lineages, one staying fully equivariant until the last generations and the other diverging from the fully equivariant network midway through, showing that training with dynamically constrained parameterizations can produce performant models.

In addition to the normally initialized static baselines, we also train and test baselines that are ini-
tialized with priors to larger symmetry groups. These are implemented by initializing all layers to be
constrained to the prior symmetry group, then using the equivariance relaxation morphism on each
layer. EquiNAS<sub>E</sub> searches for relaxation schedules that yield trained priors on equivariance, while
these additional baselines yield untrained priors. The results in Table 3 show that the C<sub>1</sub>-equivariant
networks generally improve with either equivariance prior, while the C<sub>4</sub> equivariant networks perform
better with D<sub>4</sub> equivariance initialization only when the D<sub>4</sub> constrained baselines also work well. The
untrained prior methods do not perform as well as EquiNAS<sub>E</sub> on rotMNIST, showing the benefit of
investing some training time to the constrained equivariance. For the other tasks, the baselines with
priors have better performances than their constrained baseline counterparts.

Shown in Figure 9, EquiNAS<sub>E</sub> outperforms EquiNAS<sub>E</sub> Random Select, showing the benefit of using
informed selection to guide the relaxation of equivariance constraints over training. Additionally,
EquiNAS<sub>E</sub> Retrain outperforms the Random Static baseline, showing that using compute in an in-
formed search is more beneficial than just randomly searching the space of static architecture con-
straints.

**4.5.2 Differentiable Equivariance-aware NAS**

The classification test errors are listed in Table 4 and visualized in Figure 13. EquiNAS<sub>D</sub> achieves better test accuracy than the other comparable methods on rotMNIST and Galaxy10. Due to differences in training protocol, only comparisons of relative rankings with Table 3 are possible: baseline methods accuracies followed similar patterns to ranking between experimental paradigms, suggesting the benefit of general C<sub>4</sub> equivariance for rotMNIST and Galaxy10 and general D<sub>4</sub> equivariance, including RPP, for ISIC. In this training protocol notably with adaptive optimizers, the results are more consistent across methods and trials.

The architecture weighting parameter dynamics for each trial are shown in Figure 15 for RotMNIST, Figure 16 for Galaxy10, and Figure 17 for ISIC. The general trend of less constrained layers toward the end of the network supports the conjecture of local equivariance being beneficial. However, this effect is less consistent for ISIC with possibly less inherent symmetry: trials on ISIC, the only task where EquiNAS<sub>D</sub> did not exceed baselines, had the most varied architectures. The final mixing of
<table><caption>**Table 3:** Test error percentages (lower is better) across tasks and approaches. Statistics are aggregated over the final selected population of 5 individuals for EquiNAS<sub>E</sub> and across 5 random seeds for all other methods. The **best** and **second best** average errors for each task are highlighted.</caption><thead><tr><th>Method</th><th>rotMNIST</th><th>Galaxy10</th><th>ISIC</th></tr></thead><tbody><tr><td>EquiNAS<sub>E</sub></td><td><b>1.78 ± 0.04</b></td><td><b>20.3 ± 0.9</b></td><td><b>31.0 ± 0.4</b></td></tr><tr><td>RPP [202]</td><td>2.18 ± 0.04</td><td><b>24.3 ± 2.8</b></td><td>32.2 ± 1.7</td></tr><tr><td>D<sub>4</sub> baseline</td><td><b>1.78 ± 0.11</b></td><td>50.8 ± 17.0</td><td>32.1 ± 2.4</td></tr><tr><td>C<sub>4</sub> baseline</td><td><b>1.64 ± 0.22</b></td><td>29.6 ± 5.5</td><td>32.9 ± 1.0</td></tr><tr><td>C<sub>1</sub> baseline</td><td>5.02 ± 1.15</td><td>31.6 ± 4.8</td><td>33.2 ± 1.5</td></tr><tr><td>C<sub>4</sub> (prior: D<sub>4</sub>)</td><td>1.93 ± 0.05</td><td>27.8 ± 5.3</td><td>31.9 ± 1.5</td></tr><tr><td>C<sub>1</sub> (prior: D<sub>4</sub>)</td><td>3.40 ± 0.07</td><td>25.9 ± 2.3</td><td><b>31.4 ± 2.6</b></td></tr><tr><td>C<sub>1</sub> (prior: C<sub>4</sub>)</td><td>2.96 ± 0.05</td><td>30.7 ± 7.1</td><td>32.5 ± 1.1</td></tr></tbody></table>

<figure><img src="image_15.png" /><figcaption>**Figure 9:** Test errors for experiments of Section 4.5.1, including ablation and random search experiments.</figcaption></figure>

architectures for ISIC included a high level of $C_1$, indicating that feature analysis outside of these symmetry groups is important for this benchmark.

Previous differentiable NAS works often used regularization of network size or even architecture weighting parameters themselves to encourage efficient architectures with a single highly weighted choice for each layer. However, our algorithm shows strong preference for a single, more equivariant and thus more expressive layer, notably to $D_4$ or $C_4$ equivariance, without such regularization. This may be due to the bilevel optimization dynamics: more constrained layers may be able to make more effective updates to more closely approximate the correct gradient computation that assumes optimal weights and thus become favorable compared to the lagging larger layers. This conjecture is shown particularly for trials on Galaxy10: increases in the architectural weighting parameter towards $C_1$ equivariance often comes after having strong weights for operations equivariant to larger groups.

EquiNAS<sub>D</sub> finds competitive architectures on average and can find architectures which outperform baseline choices like architectures fully equivariant to $C_1$ or $D_4$. From comparing EquiNAS<sub>D</sub> Retrain to EquiNAS<sub>D</sub> results in Figure 13, retraining a resulting architecture is not consistently better or worse than using the final weights from EquiNAS<sub>D</sub>, showing that there isn't a disadvantage to training weights during search and avoiding the additional cost of a post-search training step.
<figure><img src="image_16.png" /><figcaption>**Figure 10:** Test errors against stored parameter count for experiments of Section 4.5.1. For EquiNAS<sub>E</sub>, parameter counts of the final models are shown, although training begins with the same parameter count as the *D*<sub>4</sub> baselines as shown in Figure 11. Although RPP trains with a higher parameter count, the parameters may be stored in a single filter per layer using Equation 12 for testing.</figcaption></figure>

The use of randomized loss information in Random Static in Figure 13 shows that an informed search for architecture hyperparameters is generally useful. However, experiments on the ISIC benchmark demonstrates that the architecture search can be deceptive and that random loss information can outperform informed loss. This motivates exploration into the use of noise during the search process for architectural parameters.

A sampling of random continuous architectures in Random Static in Figure 13 shows that random architectures can perform well on problems where fully equivariant architectures like the *D*<sub>4</sub> baseline already perform well. However, on the Galaxy10 problem, the *D*<sub>4</sub> and *C*<sub>4</sub> baseline have high variance, suggesting that a fully equivariant architecture is sub-optimal. On this baseline, EquiNAS<sub>D</sub> greatly outperforms a search of random architectures, demonstrating that EquiNAS<sub>D</sub> can discover the appropriate equivariance for a specific dataset over fixed or randomly selected architectures.

The search space for EquiNAS<sub>D</sub> is already well-formed for random architectures, compared to the discrete search space of EquiNAS<sub>E</sub>. This is enabled by the [G]-mixed equivariant layer, which is a contribution of this work. Random non-mixed equivariant architectures do worse on all three benchmarks compared to random architectures which use the [G]-mixed equivariant layer. This can explain why the EquiNAS<sub>D</sub> results are closer to random baselines than the EquiNAS<sub>E</sub> results, as the search space permits easily finding the appropriate mix of equivariances compared to a discretized search space.

To explore the symmetry discovery of EquiNAS<sub>D</sub>, we apply it to six augmentations of the MNIST dataset [9], where each augmentation applies the group actions of each group in {C<sub>1</sub>, C<sub>2</sub>, C<sub>4</sub>, D<sub>1</sub>, D<sub>2</sub>, D<sub>4</sub>} respectively. The resulting architecture dynamics of this experiment are shown in Figure 12, showing that less augmented versions still have some inherent symmetry, while more augmented versions induce stronger architectural changes towards layers that are equivariant to larger groups. Across all augmentations, earlier layers tended towards more constraints to equivariance.

## 4.6 Discussion

To our knowledge, this is the first work which proposes search methods for networks with dynamically constrained equivariance. Many NAS approaches separately search for an architecture and then reinitialize and retrain the weights, while our two proposed approaches find an optimal architecture with
<figure><img src="image_17.png" /><figcaption>**Figure 11:** Historical parameter counts of each selected individual for EquiNAS<sub>E</sub> for each task. The architectures of the final selected population are labeled. Each parameter count history is colored according to the final test accuracy, which is measured of each individual upon removal.</figcaption></figure>

trained weights in a single process, notably with dynamically constrained weights. Gradient-based tuning [220] has shown the benefit not only of optimizing hyperparameters but also of dynamically adjusting them during training [221]. There is an inherent trade-off between accuracy and generalization capabilities with more constrained equivariance: dynamically constrained weights can reap both over the course of training.
<figure><img src="image_18.png" /><figcaption>**Figure 12:** EquiNAS<sub>D</sub> on classification of six augmentations of MNIST, where each row is a trial on MNIST augmented to exhibit symmetry to the labeled group.</figcaption></figure>

<table><caption>**Table 4:** Test error percentages (lower is better) in percent of incorrect classifications across tasks and approaches. The **best** and **second best** average errors for each task are highlighted.</caption><thead><tr><th>Method</th><th>rotMNIST</th><th>Galaxy10</th><th>ISIC</th></tr></thead><tbody><tr><td>EquiNAS<sub>D</sub></td><td><strong><u>2.29 ± 0.27</u></strong></td><td><strong><u>21.8 ± 1.2</u></strong></td><td>32.8 ± 0.6</td></tr><tr><td>RPP [202]</td><td>2.89 ± 0.27</td><td><strong><u>22.0 ± 1.8</u></strong></td><td><strong><u>31.5 ± 0.9</u></strong></td></tr><tr><td><em>D</em><sub>4</sub> Baseline</td><td>2.97 ± 1.50</td><td>22.5 ± 2.0</td><td><strong><u>32.0 ± 1.0</u></strong></td></tr><tr><td><em>C</em><sub>4</sub> Baseline</td><td><strong><u>2.43 ± 0.54</u></strong></td><td>22.2 ± 2.4</td><td>32.8 ± 1.0</td></tr><tr><td><em>C</em><sub>1</sub> Baseline</td><td>3.97 ± 0.75</td><td>26.5 ± 1.5</td><td>32.9 ± 3.1</td></tr></tbody></table>

Our two equivariance-aware NAS approaches have distinct approaches: EquiNAS<sub>E</sub> searches for architectures composed of discretely equivariant layers, while EquiNAS<sub>D</sub> searches for continuous mixtures of equivariance within each layer. The EquiNAS<sub>D</sub> algorithm avoids many known problems in differentiable NAS such as the discretization gap that occurs when searching over a continuous relaxation of a discrete architectural search space [222], such as that of EquiNAS<sub>E</sub>. Towards searching for discretely equivariant layers using the [G]-mixed equivariant layer, proximal NAS algorithms use techniques such as projection [129] and straight-through estimation [223] to avoid the discretization gap and thus may be effective for this application.

The EquiNAS<sub>E</sub> algorithm is innately greedy. At each selection step, the population is evaluated on the known current performance rather than the unknown final performance, so this metric is biased to architectures that train quickly. Networks constrained to higher symmetry group equivariance tend to learn faster, but this could be confounded by the equivariance relaxation causing large gradients for
<figure><img src="image_19.png" /><figcaption>**Figure 13:** Test errors for experiments of Section 4.5.2, including ablation and random search experiments.</figcaption></figure>

<figure><img src="image_20.png" /><figcaption>**Figure 14:** Test errors against stored parameter count for experiments of Section 4.5.2. Although EquiNAS<sub>D</sub> and RPP train with a higher parameter count, the parameters may be stored in a single filter per layer using Equation 12 for testing.</figcaption></figure>

the newly unconstrained parameters and thus potentially fast increases in performance. Further work could utilize other metrics for final performance, such as proxies [224].

All of the theoretical and algorithmic contributions of this work are applicable beyond the image classification experiments presented to architectures with parametrized equivariance to any discrete group. We leave the extension to other group representations and domains as future work, such as the continuous case via careful analysis of the regular representation, still given $G' \setminus G$ is finite.

Our proposed equivariance-aware NAS problems can be practically applied to find effective networks for datasets with hypothesizable symmetry. EquiNAS<sub>E</sub> may particularly work well on tasks that benefit from local equivariance, which can be determined by analyzing the architecture weighting parameters from first applying the more efficient EquiNAS<sub>D</sub>, as well as for finding good discrete architectures within which to retrain weights, based on the ablation and random comparisons. For tasks where EquiNAS<sub>D</sub> models have less consistent equivariance patterns, EquiNAS<sub>E</sub> could be adapted to propose
<figure><img src="image_21.png" /><figcaption>**Figure 15:** Architecture weighting parameters by layer for all trials on rotMNIST.</figcaption></figure>

candidates that relax any layer in the network, removing our constraint of non-increasing equivariance. We thus recommend EquiNAS$_D$ for practical applications if the final model is not restricted to discrete equivariance, in which case it can be used to inform design decisions for applying EquiNAS$_E$.

Beyond NAS, the equivariance relaxation morphism could be used in other applications such as fine-tuning and distillation. Layers of a pre-trained equivariant network could be expanded via equivariance relaxation before fine-tuning on the same or a new task. Similarly, a network could be distilled to a wider architecture for additional performance benefits.

The proposed equivariance-aware NAS algorithms are intentionally simple to focus on studying the
architectural mechanisms presented in Section 4.3, although we have already discussed potential
improvements to these algorithms. We include all valid results, even those that do not favor our own
techniques, for transparency. This work aims to build a foundation for the intersection of equivariant
architectures and NAS, rather than over-engineer the algorithms and experiments for incremental
performance gains on selected benchmarks.

**Conclusion** We present two mechanisms towards equivariance-aware architectural optimization, the equivariance relaxation morphism and the [G]-mixed equivariant layer, as well as two NAS algorithms that implement these mechanisms respectively in an evolutionary approach, EquiNAS<sub>E</sub>, and a differentiable approach, EquiNAS<sub>D</sub>. We investigate how the dynamic equivariance achieved by these algorithms affects the training and performance of networks across multiple image classification tasks of varying complexity and assumed symmetry, demonstrating that these techniques can search for performant architectures and weights even on noisy tasks. The proposed mechanisms and algorithms are extendable beyond vision tasks to any architecture with parametrized equivariance to any discrete group.
<figure><img src="image_22.png" /><figcaption>**Figure 16:** Architecture weighting parameters by layer for all trials on Galaxy10.</figcaption></figure>

The extension of NAS search spaces beyond vanilla convolutions empowers the automation of machine learning, especially for complexly structured data. As convolutions revolutionized computer vision, discovering appropriate architectures that capitalize on both explicit and implicit structure and symmetries in data may potentiate models in much broader domains.
<figure><img src="image_23.png" /><figcaption>**Figure 17:** Architecture weighting parameters by layer for all trials on ISIC.</figcaption></figure>
# 5 Improving Constrained NAS

Neural architecture search (NAS) recently emerged as a path towards automating the structural design of networks. Initial methods used evolutionary algorithms [225] or reinforcement learning [180] to search for networks from building blocks of operators. By relaxing the search space over operators to be continuous, the original Differentiable Architecture Search (DARTS) algorithm was the first to use gradient descent for searching across network architectures [19]. This one-shot search builds a supernetwork with every possible operator at every possible edge connecting activation states within the network, using trainable structural weights across operators within an edge to determine which edges and operators will be used in the final architecture.

While DARTS was able to drastically reduce search time while finding high-performing networks, it suffers from several drawbacks. Among them, contributions pointed out the lack of early search decisions [226–228], failure to approach the final target constraint [161], performance collapse due to skip-connections [229–231], and inefficiency due to unconstrained search [129].

In this work, we posit that difficulties arise in DARTS due to a lack of informed schedule and regularization befitting a constrained bilevel optimization problem. We formulate our contributions in a new augmentation called **DARTS-PRIME** (Differentiable Architecture Search with Proximity Regularization and Fisher Informed Schedule). Specifically, DARTS-PRIME is a combination of two additions to DARTS: a dynamic update schedule, yielding less frequent architecture updates with respect to weight updates, using gradient information to guide the update schedule, and a novel proximity regularization towards the desired discretized architecture, which promotes clearer distinctions between operator and edge choices at the time of discretization.

We analyze these two additions together as DARTS-PRIME, as well as each independently in ablation studies, over standard benchmarks and search spaces with CIFAR-10, CIFAR-100, and Penn TreeBank. We find that these additions independently improve DARTS and combine as DARTS-PRIME to make significant accuracy improvements over the original DARTS method and more recently published variants across all three datasets.

## 5.1 Background and Related Work

In this section we offer an explanation of the DARTS algorithm formulated as a constrained bilevel optimization problem, focusing on the issues which arise due to this complex search. We also cover related variants of DARTS which propose various solutions to the difficulties arising from this constrained bilevel problem.

DARTS sets up a supergraph neural network that contains all possible options in all possible connections within a framework architecture derived from that of other NAS works [180, 225]. For image classification tasks, this supernetwork consists of repeated cells of two different types, normal and reduce. Each cell of the same type shares the same architecture. This formulation allows for a shallower network with fewer normal cells to be searched over and a deeper one to be used for evaluation, such that both networks fit maximally on a standard GPU. The structure within each cell is a fully-connected directed acyclic graph, receiving input from the two previous cells and using a fixed number of intermediate activation states that are concatenated to produce that cell's output. Each of these edges contains all operators, each of which is a small stack of layers such as a skip-connection or a nonlinearity followed by convolution followed by batchnorm. In order to match the search space to that of the preceding NAS works, the search objective is to find exactly two operators from unique sources, either one of the two previous cells or one of the previous activation states within the cell, to go to each activation state. DARTS accomplishes this by keeping the two strongest operators, as
measured by trained continuous architecture weights, from unique sources for each state within the cell.

The DARTS search can be formalized as follows. Let $\alpha_{ijp}$ be the architecture weight for operator $p$ from node $j$ to node $i$, so $\alpha_{ij}$ represents the set of architecture weights for all operators from node $j$ to node $i$ and $\alpha_i$ represents the set of architecture weights for all operators from all nodes to node $i$. The discretization for the DARTS search space may be stated as the intersection of the following sets:

$$S = \bigcap_{i} S_{i}^{1} \cap S_{i}^{2} \cap S_{i}^{3} \qquad (13)$$

$$S_{i}^{1} = \{\alpha_{i} | \forall j \ \mathrm{card}(\alpha_{ij}) \le 1\} \qquad (14)$$

$$S_{i}^{2} = \{\alpha_{i} | \mathrm{card}(\alpha_{i}) = 2\} \qquad (15)$$

$$S_{i}^{3} = \{\alpha_{i} | \alpha_{ijp} \in \{0, 1\}\}, \qquad (16)$$

where $S_i^1$ states that there is at most one active operator coming in to each intermediate state from any given source node, $S_i^2$ states that there are exactly two total active operators coming in to each intermediate state, and $S_i^3$ states that all $\alpha_{ijp}$ values are binary, so all operators are either active or inactive. Thus, $S$ contains the encoding for all possible discretized architectures $\alpha$ in the search space of DARTS.

During the DARTS search, $S$ is relaxed so that $\alpha$ is not constrained. A softmax operation is applied over continuous $\alpha$ across operators in an edge as an activation. During a forward pass, the output of each edge is computed as the weighted sum of each operator on that edge multiplied by its activated architecture weight; then, the output of each state is computed as the sum of all incoming edges; finally, the output of the entire cell is computed as the channel-wise concatenation of each intermediate state. After searching for a set number of epochs, the softmaxed $\alpha$ is projected onto $S$ to derive the final architecture, as shown in Figure 18.

In order to train the search supernetwork with architecture weights $\alpha$ and network parameters $w$, DARTS optimizes the following objective:

$$\min_{\alpha} L_{val}(w^{*}(\alpha), \alpha) \qquad (17a)$$

$$\textbf{s.t.} \quad w^{*}(\alpha) = \arg\min_{w} L_{train}(w, \alpha), \quad \alpha \in S. \qquad (17b)$$

Even with the relaxed space of $\alpha$, the remaining bilevel optimization problem cannot be directly optimized with standard techniques due to the gradient of the outer optimization (17a) depending on the gradient of the inner optimization (17b), so it is not computationally feasible. Instead, DARTS alternates a single step of gradient descent for $\alpha$, the outer variable, using either first-order or second-order approximation of the gradient with a single step of gradient descent for $w$, the inner variable. This naive schedule of alternating steps may not be optimal, even in practice: if the parameterized operators are not trained enough at the current architecture encoding for their parameters to be a reasonable approximation of an optima, their architecture weights will be discounted unfairly. Architectures with more skip-connections converge faster even though they tend to have a worse performance at convergence [230], so this naive schedule may unfairly and undesirably favor skip-connections.

Since the publication of DARTS [19], many variations have been proposed to improve aspects of the original algorithm. We focus on those that are more relevant to the issues we tackle with DARTS-PRIME and study the same DARTS search space in the following.

RobustDARTS [231] investigates failure modes of DARTS, including avoiding the collapse towards excessive skip-connections. Skip-connection mode collapse is a known issue with DARTS, and is demonstrated in Figure 19. The authors propose several regularizations of the inner optimization (17b): in particular, their proposed DARTS-ADA utilizes Hessian information to impose an adaptively
<figure><img src="image_24.png" /><figcaption>(a) Normal cell $\alpha$ over search training</figcaption></figure>

<figure><img src="image_25.png" /><figcaption>(b) Reduce cell $\alpha$ over search training</figcaption></figure>

<figure><img src="image_26.png" /><figcaption>(c) Final normal cell</figcaption></figure>

<figure><img src="image_27.png" /><figcaption>(d) Final reduce cell</figcaption></figure>

**Figure 18:** Training progression and final results of our highest performing trial of DARTS on CIFAR-10. Each subplot in (a) and (b) represents an edge in the supernetwork cell: each column is the source of the edge, and each row is the destination of the edge. The softmaxed $\alpha$ for each operator is plotted over the search training. To derive the architecture shown in (c) and (d) respectively, the two largest activated $\alpha$ values from unique columns are selected in each row at the end of search. The "none" operator is not shown, as it cannot be selected at discretization.
<figure><img src="image_28.png" /><figcaption>**Figure 19:** Extended trials of DARTS and DARTS-PRIME on CIFAR-10. A single extended trial for each algorithm was run to 150 epochs, and the architecture checkpointed every 10 generations was evaluated once. In DARTS, the architecture eventually has only skip-connections for the searchable part of the architecture, resulting in a high-error performance. DARTS-PRIME avoids this collapse.</figcaption></figure>

increasing regularization, which bares similarities to both our proposed dynamic informed scheduling and increasing proximity regularization. However, this specific variant is not analyzed in the same standard DARTS search space. DARTS+ [232] uses early stopping to avoid skip-connection mode collapse. DARTS- [229] avoids skip-connection domination by adding an auxiliary skip-connection in parallel to every edge in addition to the skip-connection operation already being included for each edge in the search. The auxiliary skip-connection is linearly decayed over training, similar to our linear schedule of regularization.

NASP [129] tackles the constraint problem in Differentiable NAS with a modified proximal algorithm [233]. A partial discretization is applied before each $\alpha$ step and each $w$ step, so forward and backward passes only use and update the current strongest edges of the supernetwork. While this does improve the efficiency of the search without decreasing benchmark performance, it may lead to unselected operators lagging in training compared to selected ones and thus having an unfair disadvantage during search.

## 5.2 Towards Constrained Optimization

Neural Architecture Search, particularly when encoded as in DARTS, is a constrained bilevel optimization problem. Both being constrained and being bilevel separately make the optimization less straightforward, such that standard techniques are less applicable. We propose new approaches to the dimensions of bilevel scheduling and regularization towards the constraints to tackle this combined difficult problem in a more dynamic and informed fashion.

### 5.2.1 Dynamic FIMT Scheduling

Fisher Information of a neural network is closely related to the Hessian matrix of the loss of the network and can be used as a measure of information contained in the weights with respect to the training data as well as a proxy for the steepness of the gradient landscape. While true Fisher Information is expensive to calculate in neural networks, the empirical Fisher Information matrix diagonal can be computed using gradient information already calculated for each batch [234]. This form of the Fisher Information has been used for updating the momentum term in the Adam optimizer [235], overcoming catastrophic forgetting [236], and pruning parameters [237, 238].
We compute the trace of the empirical Fisher Information matrix with each minibatch:

$$ \bar{F}(w) = \mathrm{tr}(\nabla_w L_{\mathrm{train}}(w, \alpha) \nabla_w L_{\mathrm{train}}(w, \alpha)^{\mathrm{T}}) \qquad (18) $$

Because the network is changing at each weight update and this measure is subject to variability between minibatches, we use an exponentially weighted moving average of the Fisher Information Matrix Trace (FIMT), similar to the decay factor in the Adam optimizer [235]:

$$ \bar{F}_n(w) = \lambda \bar{F}(w) + (1 - \lambda) \bar{F}_{n-1}(w), \qquad (19) $$

for the n-th minibatch, where $\lambda$ is a hyperparameter.

We use the FIMT as a moving estimate of information contained within the weights of the network and the curvature of the gradient landscape. In particular, as noted in DARTS, when the gradient of $w$ is 0, then $w$ is a local optimum and thus $w^* = w$, as desired. As a local optima is approached, the FIMT should decrease and $\alpha$ updates can be triggered more frequently. To achieve this effect, we utilize an exponentially adaptive threshold denoted by $h$, parametrized by an initial threshold $h_0$, threshold increasing factor $h_i \ge 1$, expected step ratio $r$, and moving average weighting factor $\lambda$. A threshold decreasing factor $h_i \le 1$ is computed such that the expected ratio between $w$ and $\alpha$ updates, given constant $\bar{F}_n(w)$, is equal to $r$. The protocol of this schedule is detailed in 3. The adaptive threshold causes the $\alpha$ update frequency to increase when the FIMT is generally decreasing over batches and decrease when the FIMT is generally increasing over batches. In turn, this adaptive threshold yields a schedule of $\alpha$ and $w$ updates that is more informed about the convergence of the inner optimization given in Equation 17b while maintaining efficiency. This avoids unfair discount of relevant architectures. An example of our dynamic FIMT schedule is shown in Figure 20.

<figure><img src="image_29.png" /><figcaption>**Figure 20:** Example of dynamic FIMT schedule, visualized over the first epoch. $\alpha$ updates occur at an interval of roughly every 10 $w$ updates, adjustable by the expected ratio scheduling hyperparameter, but do not occur at the beginning of training when the FIMT is constantly decreasing.</figcaption></figure>

### 5.2.2 Regularization

In DARTS, the final $\alpha$ values can be far from the resultant discretization, leading to disparity between the search process and its outcome. In order to reward proximity to discretization without losing training signal to inactive portions of the supernetwork, we utilize a proximity regularization that increases in strength over the course of training. The outer objective now becomes

$$ \min_{\alpha} \left( L_{\mathrm{val}}(w^*(\alpha), \alpha) + \frac{c \rho_p}{2} \| (\alpha - \Pi_S(\alpha)) \|_2 \right), \qquad (20) $$
**Algorithm 3 DARTS-PRIME**

**Input:** Initial threshold $h_0$, threshold increasing factor $h_i$, expected step ratio $r$, moving average weighting factor $\lambda$, regularization parameter $\rho_p$, and learning rates $\gamma_w, \gamma_\alpha$

**Output:** Discrete architecture $\Pi_S(\alpha)$

1. Initialize $\alpha, w$; let $h \leftarrow h_0, h_d \leftarrow \exp(-r \ln h_i), t \leftarrow 1$

2. **while** not converged **do**

3. $G \leftarrow \nabla_w L_{train}(w, \alpha)$

4. $w \leftarrow w - \gamma_w G$

5. $\tilde{F}_n(w) \leftarrow \lambda \operatorname{tr}(GG^T) + (1-\lambda)\tilde{F}_{n-1}(w)$

6. **if** $\tilde{F}_n(w) < h$ **then**

7. $\alpha \leftarrow \alpha - \gamma_\alpha \nabla_\alpha (L_{val}(w, \alpha) + \frac{c\rho_p}{2} \|(\alpha - \Pi_S(\alpha))\|_2)$

8. $h \leftarrow h \cdot h_d$

9. **else**

10. $h \leftarrow h \cdot h_i$

11. $n \leftarrow n + 1$

12. **return** $\Pi_S(\alpha)$

where $c$ increases linearly from 0 to 1 over the course of search training, $\rho_p$ is a regularization hyperparameter, and $\Pi_S$ is the projection onto set $S$, in this case equivalent to deriving a discretized architecture. This formulation allows for more exploration of $\alpha$ and training of $w$ in parameterized operations particularly at the beginning of the search training, progressively increasing the regularization pressure towards discretization as training continues.

### 5.2.3 DARTS-PRIME

We combine dynamic FIMT scheduling and proximity regularization with DARTS as DARTS-PRIME. The procedure for DARTS-PRIME is given in Algorithm 3.

In addition to these contributions, we also study a regularization based on the Alternating Direction Method of Multipliers (ADMM), an algorithm that uses dual ascent and the method of multipliers to solve constrained optimization problems [239]. We propose as well a non-competitive activation function to naturally permit progressive discretization.

### 5.2.4 ADMM Regularization

The Alternating Direction Method of Multipliers (ADMM) is an algorithm that uses dual ascent and the method of multipliers to solve constrained optimization problems [239]. ADMM has previously been applied to training sparse networks [240], pruning weights [241, 242], and pruning channels [243], using the version of the algorithm designed for nonconvex constraints such as maximal cardinality of weight matrices. When applied in practice the non-convex objective of neural network objective functions, the optimization variable of the network parameters is usually not evaluated to convergence within every iteration. The implementation of ADMM functions similarly to a dynamic regularization, so we apply it to differentiable NAS in comparison to our proposed proximity regularization.

Directly applying ADMM for nonconvex constraints [239] to the optimization given in Problem 17 using $\alpha \in S$ as the constraint, the iterative solution is

$$ \alpha^{k+1} := \operatorname*{arg\,min}_{\alpha} \left( L_{val}(w^*(\alpha), \alpha) + \frac{\rho_a}{2} \|\alpha - z^k + u^k\|_2^2 \right) \quad (21) $$

$$ \textbf{s.t.} \quad w^*(\alpha) = \operatorname*{arg\,min}_{w} L_{train}(w, \alpha) \quad (22) $$

$$ z^{k+1} := \Pi_S(\alpha^{k+1} + u^k) \quad (23) $$

$$ u^{k+1} := u^k + \alpha^{k+1} - z^{k+1}, \quad (24) $$

where $\Pi_S$ is the projection onto $S$, $z$ and $u$ are introduced variables, and $\rho_a$ is a regularization hyperparameter.
Equation 21 is solved by using the DARTS 1st order approximation for n minibatches before updating z and u. The number of w steps (Equation 22) for each α step (Equation 21) is determined by the α schedule.

Because neither level of the bilevel optimization is evaluated to convergence within each iteration, we discount past u values with each update using a discount factor, $\lambda_u$. Thus, Equation 24 becomes

$$u^{k+1} := \lambda_u u^k + \alpha^{k+1} - z^{k+1}. \qquad (25)$$

### 5.2.5 CRB Activation

We propose a new activation of α that improves fairness towards discretization and allows for progressive discretization, called Clipped ReLU Batchnorm (CRB) activation. For CRB activation of α, the softmax activation is replaced by unit-clipped ReLU.

This activation function does not enforce all operator weights within an edge to sum to 1. So, the distribution of the output of an edge is not as regulated. To account for this in activation distributions, a non-affine batchnorm layer is placed after summing across edges for each state within the cell. This permits the α values across edges to still be comparable, as necessary for discretization. Additionally, the "none" operator can be removed, since the activated α values are no longer dependent on each other.

CRB activation provides bounds within [0, 1] and a natural progressive pruning heuristic. Over training, when any architecture weight becomes non-positive, the corresponding operator is pruned from that edge. This progressive pruning allows the network training to focus on the remaining operators and saves computational time.

## 5.3 Experiments

We test the two presented modifications to DARTS together as DARTS-PRIME as well as each independently as ablation studies. Specifically, we test the dynamic α update schedule using Fisher Information (FIMT), and the proximity regularization (PR), in addition to DARTS-PRIME. We compare these methods to DARTS and its variants on three benchmark datasets: the image classification datasets CIFAR-10 and CIFAR-100, and the text prediction dataset Penn TreeBank.

We also rerun DARTS using the code provided by [19]<fnref n="2" />. To better understand the impact of an informed schedule, we study a slight modification of DARTS with a constant schedule of exactly 10 w steps per α step (CS10). The value of 10 was selected for all scheduled experiments as we hypothesize that more w steps should be taken per α step to aid the bilevel optimization approach local minima in w before optimizing α.

### 5.3.1 Datasets and Tasks

CIFAR-10 and CIFAR-100 [247] are image datasets with 10 and 100 classes, respectively, and each consisting of 50K training images and 10K testing images for image classification. Applying the operator and cell-based search space from [19] to CIFAR-10 has been heavily studied in related NAS works. We additionally perform both search and evaluation on CIFAR-100 using identical protocols and hyperparameters to explore their applicability to similar tasks.

Penn TreeBank [248] (PTB) is a text corpus consisting of over 4.5M American English words for the task of word prediction. We use the standard pre-processed version of the dataset [19, 245]. The DARTS search space for RNNs for text prediction searches over activation functions of dense layers within an RNN cell.

<footnote n="2">²https://github.com/quark0/darts</footnote>
<table><caption>**Table 5:** Results for CNN architecture search on CIFAR-10 and CIFAR-100. Test errors (lower is better) are listed in percent of incorrect classifications on the test set after retraining each search architecture. Network sizes of the evaluation network are listed in millions of parameters. Search costs are listed in unnormalized GPU days and do not include evaluation costs. Each of our trials were run on a single V100. Mean errors and search costs are listed with standard deviations.</caption><thead><tr><th>Task</th><th>Method</th><th>Test Error (%)</th><th>Parameters (M)</th><th>Search Time (GPU days)</th></tr></thead><tbody><tr><td rowspan="10">CIFAR-10</td><td>NASNet-A<sup>†</sup> [180]</td><td>2.65</td><td>3.3</td><td>1800</td></tr><tr><td>AmoebaNet-A*<sup>†</sup> [225]</td><td>2.55 ± 0.05</td><td>2.8</td><td>3150</td></tr><tr><td>PNAS<sup>†</sup> [244]</td><td>3.41 ± 0.09</td><td>3.2</td><td>225</td></tr><tr><td>ENAS<sup>†</sup> [245]</td><td>2.89</td><td>4.6</td><td>0.5</td></tr><tr><td>Random Search [19]</td><td>3.29 ± 0.15</td><td>3.2</td><td>4</td></tr><tr><td>DARTS 1st* [19]</td><td>3.00 ± 0.14</td><td>3.3</td><td>1.5</td></tr><tr><td>DARTS 2nd* [19]</td><td>2.76 ± 0.09</td><td>3.3</td><td>4</td></tr><tr><td>P-DARTS [227]</td><td>2.5</td><td>-</td><td>0.3</td></tr><tr><td>PD-DARTS [226]</td><td>2.57 ± 0.12</td><td>3.2</td><td>0.3</td></tr><tr><td>FairDARTS [161]</td><td>2.54 ± 0.05</td><td>3.32 ± 0.46</td><td>0.4</td></tr><tr><td>R-DARTS (L2) [231]</td><td>2.95 ± 0.21</td><td>-</td><td>1.6</td></tr><tr><td>DARTS- [229]</td><td>2.59 ± 0.08</td><td>3.5 ± 0.13</td><td>0.4</td></tr><tr><td>NASP [129]</td><td>2.83 ± 0.09</td><td>3.3</td><td>0.1</td></tr><tr><td rowspan="10">CIFAR-100</td><td>DARTS reimplementation</td><td>2.82 ± 0.19</td><td>2.78 ± 0.28</td><td>0.4</td></tr><tr><td>+CS10</td><td>2.80 ± 0.09</td><td>3.41 ± 0.40</td><td>0.4</td></tr><tr><td>+FIMT</td><td>2.79 ± 0.12</td><td>3.49 ± 0.31</td><td>0.6</td></tr><tr><td>+PR</td><td>2.77 ± 0.03</td><td>3.50 ± 0.15</td><td>0.5</td></tr><tr><td>DARTS-PRIME</td><td>2.62 ± 0.07</td><td>3.70 ± 0.27</td><td>0.5</td></tr><tr><td>+CRB</td><td>3.58 ± 0.79</td><td>1.77 ± 0.25</td><td>0.4</td></tr><tr><td>+FIMT+CRB</td><td>2.91 ± 0.13</td><td>2.34 ± 0.35</td><td>0.5</td></tr><tr><td>DARTS-PRIME +CRB</td><td>2.98 ± 0.22</td><td>3.20 ± 0.33</td><td>0.5</td></tr><tr><td>+ADMM</td><td>2.89 ± 0.05</td><td>3.13 ± 0.30</td><td>0.4</td></tr><tr><td>+ADMM+FIMT</td><td>2.85 ± 0.15</td><td>4.43 ± 0.99</td><td>0.5</td></tr><tr><td>+ADMM+FIMT+CRB</td><td>3.18 ± 0.29</td><td>2.06 ± 0.26</td><td>0.4</td></tr><tr><td rowspan="5">CIFAR-100</td><td>P-DARTS [227]</td><td>15.92 ± 0.18</td><td>3.7</td><td>0.4</td></tr><tr><td>DARTS+ [232]</td><td>15.42 ± 0.30</td><td>3.8</td><td>0.2</td></tr><tr><td>R-DARTS (L2) [231]</td><td>18.01 ± 0.26</td><td>-</td><td>-</td></tr><tr><td>DARTS (our trials)</td><td>19.49 ± 0.60</td><td>2.02 ± 0.15</td><td>0.5</td></tr><tr><td>+FIMT</td><td>19.44 ± 1.26</td><td>2.22 ± 0.22</td><td>0.5</td></tr><tr><td>+PR</td><td>19.04 ± 0.85</td><td>2.24 ± 0.31</td><td>0.4</td></tr><tr><td>DARTS-PRIME</td><td>17.44 ± 0.57</td><td>3.16 ± 0.42</td><td>0.5</td></tr><tr><td>DARTS-PRIME +CRB</td><td>18.41 ± 0.48</td><td>2.28 ± 0.24</td><td>0.4</td></tr></tbody></table>

*: Previously reported results where the single best search architecture was evaluated multiple times rather than reporting results across multiple searches. Selecting the best architecture across searches is reflected in the search time.

†: Architectures searched in a slightly different search space in operator types and network meta-structure.

### 5.3.2 Hyperparameters

We use all protocols and hyperparameters from [19] for each domain of image classification and text prediction and for both search and evaluation unless otherwise stated. We use the first order approximation of the gradient of $\alpha$. Hyperparameters for CIFAR-100 are maintained directly from CIFAR-10.

When the dynamic FIMT schedule is used, the expected ratio of $w$ steps per $\alpha$ step, $r$, is set to 10, as in CS10. The initial threshold, $h_0$, is set to 1.0. The threshold increasing factor, $h_i$, is set to 1.05. The
<table><caption>**Table 6:** Results for RNN architecture search on PTB. Performance is measured by perplexity (lower is better). All previous works report the single best result from multiple runs rather than statistics across multiple trials. For our trials, convergence at 1200 epochs was sufficient to differentiate between different methods.</caption><thead><tr><th rowspan="2">Method</th><th colspan="2">Perplexity</th><th rowspan="2">Evaluation Epochs</th><th rowspan="2">Parameters (M)</th><th rowspan="2">Search Time (GPU Days)</th></tr><tr><th>Valid</th><th>Test</th></tr></thead><tbody><tr><td>NAS<sup>†</sup> [180]</td><td>-</td><td>64</td><td>-</td><td>25</td><td>10000</td></tr><tr><td>ENAS<sup>††</sup> [245]</td><td>60.8</td><td>58.6</td><td>8000</td><td>24</td><td>0.5</td></tr><tr><td>Random search [19]</td><td>61.8</td><td>59.4</td><td>8000</td><td>23</td><td>2</td></tr><tr><td>DARTS 1st [19]</td><td>60.2</td><td>57.6</td><td>8000</td><td>23</td><td>0.5</td></tr><tr><td>DARTS 2nd [19]</td><td>58.1</td><td>55.7</td><td>8000</td><td>23</td><td>1</td></tr><tr><td>GDAS [246]</td><td>59.8</td><td>57.5</td><td>2000</td><td>23</td><td>0.4</td></tr><tr><td>DARTS (our trials)</td><td>68.5 ± 6.2</td><td>66.2 ± 6.1</td><td>1200</td><td>23</td><td>0.1</td></tr><tr><td>DARTS-PRIME</td><td>63.1 ± 2.1</td><td>61.1 ± 2.2</td><td>1200</td><td>23</td><td>0.1</td></tr></tbody></table>

†: Architectures searched in a slightly different search space in operator types and network meta-structure.

‡: Reevaluated and reported by [19] using the current standard evaluation pipeline.

moving average weighting factor, $\lambda$, is set to 0.2. These values are all set to intuitive values without formal tuning.

For proximity regularization (PR), the regularization parameter, $\rho_p$, is set to 0.1 for image classification and 1.0 for text prediction. This was exponentially searched over 0.001 to 1.0 on CIFAR-10 and PTB, respectively, during early development.

For ADMM regularization, $z$ and $u$ are updated after every 10 $\alpha$ steps. The decay factor, $\lambda_u$, is set to 0.8. The regularization parameter, $\rho_a$, is set to 0.1.

When CRB activation is used, the cosine annealing of the architecture learning rate and the weight decay of $\alpha$ are both turned off.

### 5.3.3 Data and Training Configuration

After search, we follow the same evaluation procedure as DARTS for image classification and text prediction. After the search training completes and the final discretization selects the cell architecture(s), the evaluation network is reinitialized at the full size, trained on the training set, and tested on the heldout test set.

For trials with the original DARTS schedule of alternating $w$ and $\alpha$ steps, the training and validation split was maintained at 50/50 during search, as done in [19]. For trials with either the constant or dynamic schedule with more $w$ steps per $\alpha$ step, the dataset was split according to the expected schedule of steps. For the ratio of 10 $w$ steps per $\alpha$ step, this results in a dataset split of 90.91% in the training set and 9.09% in the validation set.

A difference from the experiment protocol in DARTS is that we do not select the best architecture from multiple searches based on validation performance. Instead, we evaluate each architecture fully once and report the mean test error across 4 search trials with different random seeds. 4 was chosen as the number of trials due to the prohibitive cost of the evaluation phase which can take up to 3 GPU days per trial on a V100 GPU. While this is a small number of independent trials, we believe that showing the evaluation results from multiple search trials, instead of the best search architecture, is a better measure of search algorithm performance and reliability.
## 5.4 Results and Discussion

We conduct experiments for each of the contributions independently as well as combined with each other in various configurations in the standard DARTS search space for CIFAR-10. Our experiments also include the original DARTS algorithm. We also apply DARTS, DARTS-PRIME, and a subset of ablation configurations to CIFAR-100 and PTB. The mean and standard deviation across searches of the resulting test error or perplexity, final architecture size, and search time for each are reported in Table 5 for CIFAR-10 and CIFAR-100 and in Table 6 for PTB, compared to reported results from related works. The $\alpha$ progressions over training and final architectures for the best trial on CIFAR-10 are shown in Figure 18 for DARTS and Figure 21 for DARTS-PRIME. The best architectures for DARTS-PRIME are shown in Figure 23.

The DARTS-PRIME trials show significant improvements over DARTS and match state-of-the-art results within the standard CIFAR-10 search space and task while also yielding one of the lowest variances in performance across our ablation studies. The training progress and final cell architectures for the best trial of DARTS-PRIME are shown in Figure 21. This architecture has an evaluation size of 3.76M parameters and a test error of 2.53.

DARTS-PRIME avoids skip-connection mode collapse, as shown in Figure 19. This is important in domains with variance in parameterization across operators. Avoiding skip-connection mode collapse in the CNN search space decreases the need for tuning the length of search, which improves generalizability across tasks.

We see a clear correlation between network size and performance for this search space in image classification tasks, further visualized in Figure 26. This is one of the reasons why the skip-connection mode collapse is catastrophic.

### 5.4.1 Benefits of Scheduling

Using either CS10 or dynamic FIMT scheduling increases the network size in all paired experiments and tends to improve performance across datasets. This is tied to avoiding the skip-connection mode collapse. Even with many fewer $\alpha$ updates overall, trials using the less frequent schedules have comparable or larger ranges of $\alpha$ values at discretization. Thus, using more informed architecture steps helps prevent unfair suppression of parameterized operations.

Both the constant and dynamic schedules show similar improvement over the original schedule. One characterization of the dynamic schedule is that in the first epoch, the $\alpha$ updates are delayed significantly beyond the expected schedule, as shown in Figure 20. This appears beneficial, as this gives parameterized operations more time to train before $\alpha$ updates start to occur. Characterizing further benefits of dynamic scheduling over constant scheduling is left as future work. Additionally, other information metrics and dynamic scheduling techniques beyond Fisher Information may be explored.

### 5.4.2 Regularization Improves Reliability

Proximity regularization yields better and more consistent test error and perplexity results, whether applied alone or with FIMT as in our proposed DARTS-PRIME, especially for CIFAR-10. This is very beneficial for one-shot tasks to improve reliability.

Using ADMM regularization generally improves performance in each paired experiment. However, proximity regularization was more performant in all cases. ADMM is a very powerful algorithm but with many hyperparameters that require careful tuning when applied in practice. This leaves further development as open work.
<figure><img src="image_30.png" /><figcaption>(a) Normal cell $\alpha$ over search training</figcaption></figure>

<figure><img src="image_31.png" /><figcaption>(b) Reduce cell $\alpha$ over search training</figcaption></figure>

<figure><img src="image_32.png" /><figcaption>(c) Final normal cell</figcaption></figure>

<figure><img src="image_33.png" /><figcaption>(d) Final reduce cell</figcaption></figure>

<caption>**Figure 21:** Best trial of DARTS-PRIME on CIFAR-10.</caption>
<figure><img src="image_34.png" /><figcaption>**Figure 22:** Best final architecture of DARTS on CIFAR-100.</figcaption></figure>

<figure><img src="image_35.png" /><figcaption>**Figure 23:** Best final architecture of DARTS-PRIME on CIFAR-100.</figcaption></figure>

<figure><img src="image_36.png" /><figcaption>**Figure 24:** Best final architecture of DARTS on PTB.</figcaption></figure>

<figure><img src="image_37.png" /><figcaption>**Figure 25:** Best final architecture of DARTS-PRIME on PTB.</figcaption></figure>
<figure><img src="image_38.png" /><figcaption>**Figure 26:** Comparison across CIFAR-10 trials of test error vs. network size, recolored by (a) activation, (b) schedule, and (c) regularization.</figcaption></figure>
The linear scheduling of the increase of proximity regularization strength is a simple implementation
choice that yields sufficient performance improvement. It may conflict with applying early stopping as
done in other DARTS variants, but only minorly, in a similar way as standard training protocols such as
the cosine annealing of architecture weights used in DARTS. This leaves other informed and dynamic
schedules, such as similar to our FIMT schedule, as an open question.

5.4.3 CRB Activation Produces Smaller Networks

For all combinations, the CRB activation finds smaller networks than the corresponding experiment
with softmax activation, albeit with a small degradation in performance. This pattern is shown in
Figure 26a. We noted that only trials using CRB activation selected any pooling operators in the normal
cell architecture, where other methods use convolutional operators. This could cause performance
degradation due to the difference in depth between the search and evaluation networks, particularly
since only normal cells are added to reach the final depth.

We also note that CRB activation has high variance without regulation or informed scheduling. Particu-
larly in the DARTS+CRB experiment, one trial had a very high error of 4.95, which greatly increased the
mean and variance. This trial fell into skip-connection mode collapse. As discussed in the following,
scheduling and especially regularization help prevent this collapse from occurring, which is beneficial
if the search is applied in a truly one-shot task. On the other hand, if multiple searches can be com-
pleted and small network size is beneficial to the application, this protocol may actually be the best
approach to apply.

An example of progressive pruning provided by CRB activation is shown in Figure 27(a-b).

5.4.4 DARTS-PRIME Performance Beyond CIFAR-10

Our findings in CIFAR-10 are also generally supported by the results on CIFAR-100 and PTB: DARTS-
PRIME consistently and significantly outperforms DARTS. Specifically on PTB, DARTS-PRIME also
has higher reliability in performance. The architecture from our best trial of DARTS-PRIME on CIFAR-
100, shown in Figure 23, has an evaluation size of 3.56M parameters and a test error of 16.82. The
architecture from our best trial of DARTS-PRIME on PTB, shown in Figure 25, has an validation
perplexity of 60.4 and a test perplexity of 58.2 in the shortened evaluation time compared to previous
works.

We note that no further hyperparameter tuning was completed for CIFAR-100, but we anticipate that
tuning directly on CIFAR-100 may have allowed DARTS-PRIME to get similar low-variance performance
results as the other tasks. Only minor tuning was completed for PTB, which differs from CIFAR in
application domain and search space, as the operators of the RNN search space are activation functions,
which are not parameterized and do not affect the overall size of the network. The performance of
DARTS-PRIME on CIFAR-100 and PTB shows the applicability of the methods and hyperparameters
within DARTS-PRIME without extensive searching on target datasets and tasks.

5.5 Conclusion

In this work, we propose DARTS-PRIME with alternative approaches to two dimensions of differen-
tiable NAS algorithms: informed dynamic scheduling of gradient descent steps and regularization
towards discretization. Each of these are analyzed with ablation studies to isolate the effects of each
approach across domains and tasks. For practitioners, we recommend dynamic FIMT scheduling
for improved network performance and proximity regulation for both improved and more reliable
network performance.

The benefits of these approaches are apparent in results on the CIFAR-10, CIFAR-100, and PTB bench-
marks yet also offer insight for different applications of differentiable NAS. The majority of DARTS
<figure><img src="image_39.png" /><figcaption>(a) Normal cell $\alpha$ over search training</figcaption></figure>

<figure><img src="image_40.png" /><figcaption>(b) Reduce cell $\alpha$ over search training</figcaption></figure>

<figure><img src="image_41.png" /><figcaption>(c) Final normal cell</figcaption></figure>

<caption>(d) Final reduce cell</caption>

<caption>**Figure 27:** Best trial of DARTS-PRIME +CRB.</caption>
variants, including this work, focus on narrow and static benchmark problems, achieving similar values of minimal error or perplexity through extensive weight training and hyperparameter tuning. However, more dynamic tasks, such as shifting data distributions or semi-supervised learning as in reinforcement learning, could highlight the benefits of using dynamic optimization schemes, such as our proposed improvements of scheduling and regularization.

The two improvements of DARTS-PRIME, proximity regularization and a FIMT informed schedule, are modular and could be added to other differentiable NAS algorithms beyond DARTS [19]. In particular, proximity regularization may be applied to any algorithm utilizing a continuous relaxation of the architecture, and dynamic FIMT scheduling may be applied to any NAS method posed as a multilevel optimization. While DARTS-PRIME shows a significant improvement over DARTS, we believe that the contributions of a dynamic FIMT scheduling and proximity regularization can be beneficial for one-shot NAS methods in general.

Dynamic scheduling of structural steps permits more flexible learning algorithms that adapt to the training trajectory and make architectural changes when appropriate. This work contributes towards the challenges of automating machine learning and dynamic adaptation during the learning process, initiating investigation of dynamic scheduling for structural changes. These ideas will be further explored in another structural paradigm, neural creation, in the following section.
# 6 Neurogenesis

Most existing structural learning approaches such as neural architecture search (NAS) [80] or pruning [78] prune away from a predefined search space, but growing architectures is not yet well understood. By dynamically building ANNs throughout learning, we can aim towards networks which learn not only parameters but also their architectures for specific tasks, potentially improving learning or performance of the final architecture and avoiding expensive hand-tuning of architectures. Furthermore, dynamic ANN architectures can allow continual growth to include more information and tasks as necessary. Dynamic architecture changes may add or remove structural units or connections, operating on the basis of neurons, channels, layers, or other structures. We consider neurogenesis to be the form of structural learning that adds new neurons or channels within existing layers, complementary to structural pruning often used by channel-searching NAS techniques such as [107], [117] and to network deepening such as [146]. Our focus in this chapter is neurogenesis, which is a naturally difficult and less well-studied form of structural learning that invokes multiple questions, specifically when to add neurons, where to add them in the network, and how to initialize the parameters of new neurons; questions which have an undefined search space that must be artificially constrained for optimization.

While considering the trade-off of performance versus efficiency of the network, including training time, inference time, and memory size, we aim to grow ANNs without a preset final size. We propose that an effective neurogenesis algorithm should converge to an efficient number of neurons in each layer during the course of training.

We present the following contributions towards understanding and utilizing neurogenesis.

* We introduce a framework for neurogenesis, decomposing it into triggers and initializations.

* We present novel triggers and initializations based on orthogonality of either post-activations or weights that together form the NORTH* suite of strategies, as well as a gradient-based trigger to complement existing gradient-based initializations. These algorithmically answer when, where, and how to add new neurons to ANNs.

* We compare these contributions with existing initialization methods from the literature and baselines in a variety of tasks and architecture search spaces, showing that NORTH* strategies achieve compact yet performant architectures.

## 6.1 Background

### 6.1.1 Problem Statement and Notations

An artificial neural network (ANN) may be optimized through empirical risk minimization:

$$ \arg\min_{f} \mathbb{E}_{x, y \sim D} L(f(\mathbf{x}), \mathbf{y}) \qquad (26) $$

for a neural network $f$, a loss function $L$, and a dataset $D$ consisting of inputs $x$ and outputs $y$. In the simple case of a dense multi-layer perceptron (MLP) with $d$ hidden layers, $f$ may be expressed as $f(\mathbf{x}) = \sigma_{d+1}(W_{d+1}\sigma_d(...W_2\sigma_1(W_1\mathbf{x})...))$, where $\sigma_l$ is a nonlinear activation function, adding a row for the bias, and $W_l \in \mathbb{R}^{M_l \times (M_{l-1}+1)}$ is the weight matrix for the $l$th layer, including the bias parameters. We distinguish between input and output weights of neurons in $W_l$ with $M_l$ rows that each represent the fan-in weights of a neuron in layer $l$ receiving input from each of the $M_{l-1}$ preceding neurons and equivalently $M_{l-1} + 1$ columns for which each of $M_{l-1}$ are the fan-out weights of a previous-layer neuron and the last column is the biases.
We define $z_l = W_l \sigma_{l-1}(W_{l-1} \sigma_{l-2}(\dots W_2 \sigma_1(W_1 x)\dots))$ as the pre-activations and $h_l = \sigma_l(z_l)$ as the post-activations of layer $l$. For $n$ samples, i.e. a mini-batch, $Z_l \in \mathbb{R}^{M_l \times n}$ is the pre-activation matrix and $H_l = \sigma(Z_l)$ is the post-activation matrix.

To perform neurogenesis, the addition of $k$ neurons to the $l^{th}$ layer is accomplished by appending $k$ rows of fan-in weights to $W_l$ and $k$ columns of fan-out weights to $W_{l+1}$. Utilizing this operation in the empirical risk minimization of Equation (26) defines the neurogenesis optimization. We restrict the search space of where to add new neurons to within existing layers.

### 6.1.2 Neurogenesis

We consider neurogenesis through the following dimensions.

* When: Standard learning algorithms naturally discretize the learning process into steps, so neurogenesis may occur at any step of training.

* How many: When neurogenesis is triggered at a step, multiple neurons can be added at once.

* Where: In standard ANN architectures, this amounts to which layer neurogenesis occurs in.

* How: The fan-in and fan-out weights and bias of each new neuron must be initialized.

Most existing neurogenesis works focus on "How", or the initialization of neurons, using a fixed schedule of additions. Some works have approached global scheduling, such as [147, 249, 250] using convergence as a global trigger for neurogenesis. However, this leads to neurogenesis happening later in training and does not explore the questions of "Where" or "How many". We posit that, in order to be competitive with static networks in terms of training cost, neurogenesis should happen throughout training, including before convergence.

To answer "How", most existing non-random neurogenesis initializations in recent literature are gradient-based. Firefly from [110] generates candidate neurons that either split existing neurons with noise or are completely new and selects those with the highest gradient norm. [101] also create noisy copies, selecting from high-saliency neurons. Both GradMax from [142] and NeST from [68] compute an auxiliary gradient of zero-weighted bridging connections across layers over a mini-batch. GradMax uses left singular vectors of this auxiliary gradient as fan-out weights to maximise the gradient norm, while NeST initializes a sparse neuron connecting neuron pairs with a high gradient value in their auxiliary connection.

No known works add new neurons that are explicitly more different to existing neurons than randomly selected weights. In the context of non-growing and linear networks, [251] proposes orthogonal weight initialization. In the case of MLPs with nonlinear activation functions, a layer with orthogonal weights only provides orthogonal pre-activations, and only if the output of the previous layer is orthogonal. [252] finds pre-activation orthogonality may actually be detrimental in non-linear networks and presents a weight initialization scheme for layers with equal static widths that maximises post-activation orthogonality, but does not have a clear extension to neurogenesis. [133] proposes a NAS heuristic based on post-activation distances between samples, computationally similar to the orthogonality gap metric by [252]. Maintaining orthogonality of neuron activations in static networks has especially been studied in reinforcement learning [253] and self-supervised learning [254], but becomes relevant in all tasks when considering dynamically growing networks.

Neurogenesis during the learning process necessarily expands the parameter search space and thus the loss landscape's dimensionality. Neurogenesis may either immediately preserve the ANN's function, such as by using zero fan-in or fan-out weights as in [142, 152], or may change the function and current performance. Function-preserving neurogenesis is a form of network morphism [137, 255],
**Algorithm 4 Neurogenesis framework.**

**procedure NEUROGENESIS(TRIGGER, INITIALIZATION, initial ANN f)**
**while** *f* not converged **do**
Gradient descent step on current existing weights
**for** each hidden layer *l* **do**
**if** TRIGGER(*f*, *l*) > 0 **then**
Add TRIGGER(*f*, *l*) neurons using INITIALIZATION(*f*, *l*)
**return** trained *f*

<table><caption>**Table 7:** Specification of initialization and trigger pairs used for each strategy.</caption><thead><tr><th>Strategy</th><th>Trigger</th><th>Initialization</th></tr></thead><tbody><tr><td>NORTH-Select</td><td><i>T</i><sub>act</sub> eq. (28)</td><td>Select sec. 6.2.2</td></tr><tr><td>NORTH-Pre</td><td><i>T</i><sub>act</sub> eq. (28)</td><td>Pre-activation eq. (33)</td></tr><tr><td>NORTH-Random</td><td><i>T</i><sub>act</sub> eq. (28)</td><td>RandomInit sec. 6.2.2</td></tr><tr><td>NORTH-Weight</td><td><i>T</i><sub>weight</sub> eq. (31)</td><td>Weight eq. (34)</td></tr><tr><td>GradMax</td><td><i>T</i><sub>grad</sub> eq. (32)</td><td>GradMax sec. 6.2.2, [142]</td></tr><tr><td>Firefly</td><td><i>T</i><sub>grad</sub> eq. (32)</td><td>Firefly sec. 6.2.2, [110]</td></tr><tr><td>NeST</td><td><i>T</i><sub>grad</sub> eq. (32)</td><td>NeST sec. 6.2.2, [68]</td></tr></tbody></table>

maintaining the location in the newly expanded loss landscape but changing the direction of the descent path into the new dimensions. [142] selects the steepest direction by maximising the gradient norm. Function-altering neurons jump from the descent path: [110, 250, 256] escape convergence at saddle points, while [68] performs a backpropagation-like jump.

Previous neurogenesis works take many approaches across a range of motivations and applications to tackle this difficult problem. In order to unify neurogenesis strategies and study their components, we propose a framework within which we formulate our contributions.

## 6.2 A Framework for Studying Neurogenesis

Our neurogenesis framework decomposes neurogenesis strategies into *triggers*, which are heuristics that determine when, where, and how many neurons to add, and *initializations*, which determine how to set their weights before training them. We present the basic framework in Algorithm 4. After every gradient step, for each layer, the trigger is evaluated to compute if and how many neurons to add, then the initialization is used to add these new neurons. As the trigger is evaluated after each gradient step, the decision of when is based on non-zero outputs from the trigger. Similarly, the question of where to add neurons is treated by evaluating the trigger on each layer independently.

We introduce our contributions for the case of MLPs, although they generalize to more complex cases like convolutions, detailed in Section 6.2.3. Our strategies with their triggers and initializations are summarized in Table 7. We define triggers in Section 6.2.1 and initializations in Section 6.2.2.

### 6.2.1 Triggers

We explore three sources of information for triggers for neurogenesis: neural activations, weights, and gradients. These triggers determine if and how many neurons to add for each layer after each gradient step. We propose that a useful neurogenesis trigger should add neurons when doing so will efficiently improve the capacity of the network or learning.
**Activation-based** When building an efficient network through neurogenesis, we intuitively want to add neurons yielding novel features that may improve the direction of descent, lower asymptotical empirical risk, or avoid unnecessary redundancy in the network. Thus, we need to measure how different or orthogonal the post-activations are from each other. To measure orthogonality of a layer, we use the $\epsilon$-numerical rank of the post-activation matrix [253, 257]. For layer $l$ and $n$ samples generating the post-activation $\mathbf{H}_l$, the effective dimension metric may be estimated by

$$ \phi_a^{ED}(f, l) = \frac{1}{M_l} \left| \left\{ \sigma \in \text{SVD}\left(\frac{1}{\sqrt{n}}\mathbf{H}_l\right) \middle| \sigma > \epsilon \right\} \right|, \quad (27) $$

where SVD($\frac{1}{\sqrt{n}}\mathbf{H}_l$) is the set of singular values of $\frac{1}{\sqrt{n}}\mathbf{H}_l$ and $\epsilon > 0$ is a small threshold. Due to the SVD decomposition, evaluating this metric has a constraint on the dimensions of $\mathbf{H}_l$ of $n > M_l$.

When the orthogonality metric of a layer's current post-activations is high, there may be additional useful features beyond the currently saturated directions so we add neurons. Conversely, a low metric value indicates redundancy in the layer, which may benefit from differentiation between neurons via gradient steps before reconsidering neurogenesis. Adding a new neuron will increase the metric when its activation is more orthogonal to that of other neurons and decrease it if it is redundant. We use each layer's metric value at network initialization as the baseline value, which accounts for orthogonality loss as the input is propagated through layers the network. We aim to at least maintain this initial metric value over training, even as the layer grows: if new neurons do not increase the rank and thus lower the metric value, then neurogenesis pauses until the rank increases via gradient descent. We multiply the baseline value by a threshold hyperparameter $\gamma_a$ close to 1. The number of neurons triggered is the difference of the current metric value and the threshold, scaled by the current number of neurons:

$$ T_{act}(f, \phi_a, l) = \min(0, |M_l(\phi_a(f, l) - \gamma_a\phi_a(f_0, l))|), \quad (28) $$

where $f_0$ is the ANN at initialization and $\phi_a$ is the orthogonality metric.

We additionally study a metric based on the orthogonality gap from [252] that measures the gap of the covariance matrix of post-activations to the identity matrix. We use an estimate of the orthogonality gap

$$ \phi_a^{OG}(f, l) = 1 - \left\| \left( \frac{1}{\|\mathbf{H}_l\|_F^2} \right) \mathbf{H}_l^\top \mathbf{H}_l - \left( \frac{1}{\|\mathbf{I}_n\|_F^2} \right) \right\|_F, \quad (29) $$

where $\mathbf{I}_n \in \mathbb{R}^{n \times n}$ is the identity matrix. $\mathbf{H}_l^\top \mathbf{H}_l \in \mathbb{R}^{n \times n}$ represents the covariance of $\mathbf{H}_l$ across samples and thus approaches the identity matrix if the post-activation vector of each neuron is orthogonal to all others. However, this metric has a dependency on layer width that may confound neurogenesis strategies. Also, this metric does not extend to the weight-based strategy presented next.

**Weight-based** We include weight matrix orthogonality as a comparison to activation-based methods. The trigger, $T_{weight}$, is computed as in equations (27)-(28) but with $\mathbf{W}_l$ instead of $\mathbf{H}_l$:

$$ \phi_w^{ED}(f, l) = \frac{1}{M_l} \left| \left\{ \sigma \in \text{SVD}\left(\frac{1}{\sqrt{n}}\mathbf{W}_l\right) \middle| \sigma > \epsilon \right\} \right|, \quad (30) $$

$$ T_{weight}(f, \phi_w, l) = \min(0, |M_l(\phi_w(f, l) - \gamma_w\phi_w(f_0, l))|). \quad (31) $$

**Gradient-based** In order to study gradient-based neurogenesis initializations such as [68], [110], and [142] in the context of dynamic neurogenesis, we propose a trigger based on the auxiliary gradient. As shown by [142], the maximum increase in gradient norm by adding $k$ neurons to the $l$th layer is the
sum of the largest k singular values of the auxiliary gradient matrix $\frac{\partial L}{\partial z_{l+1}} h_{l-1}^T$. We use these singular values and compare them to the gradient norms of all existing neurons in that layer over the same n samples. The number of neurons triggered is the number of singular values larger than the sum of gradient norms:

$$T_{grad}(f, L, l) = \left| \left\{ \sigma \in \text{SVD}\left(\frac{\partial L}{\partial Z_{l+1}} H_{l-1}^T\right) \middle| \sigma > \sum_{m=1}^{M_l} \left\| \frac{\partial L}{\partial w_m^{\text{in}}} \right\|_F + \left\| \frac{\partial L}{\partial w_m^{\text{out}}} \right\|_F \right\} \right|, \quad (32)$$

where $w_m^{\text{in}}$ is the m-th row of $W_l$ and $w_m^{\text{out}}$ is the m-th column of $W_{l+1}$, respectively representing the fan-in and fan-out weights of the m-th neuron in layer l. We intuit this threshold creates neurons that could have initial gradients significantly stronger than existing neurons and will be harder to surpass as the layer grows, thus leading to convergence in layer width.

### 6.2.2 Initializations

For the initialization of new neurons, we aim to reuse information computed for the corresponding trigger in each method to reduce computational overhead. For all NORTH* initializations, we add function-preserving neurons which do not immediately change the output of the network, or more specifically the activation of any downstream layers. We propose that the role of initialization for neurogenesis during training is rather to add a new neuron that is useful locally to the triggered layer which will then be integrated into the network's functionality through gradient descent; we measure this utility based on activation, weights, and gradients. We rescale all new neurons to match the weight norm of existing neurons, so scaling is ignored in the following calculations for simplicity.

**Activation-based** We present multiple approaches for initializing neurons towards orthogonal post-activations. Due to the nonlinearity of a neuron, we only have a closed-form solution for a set of fan-in weights yielding not the desired orthogonal post-activation but an orthogonal pre-activation, similar to related works on orthogonality in static networks [251]. The following approaches generate candidate neurons and select those that independently maximise the orthogonality metric of the post-activations with the existing neurons in that layer. We initialize fan-out weights to 0.

* Select: generate random candidate neurons.

* Pre-activation: generate candidate neurons with pre-activations maximally orthogonal to existing pre-activations. For n samples and $M_l$ neurons in layer l, a candidate's fan-in weights are

$$w = (H_{l-1}^T)^{-1} V_{Z_l}^T a, \quad (33)$$

where $(H_{l-1}^T)^{-1}$ is the left inverse of the transpose of post-activations $H_{l-1}$ for layer l-1, $V_{Z_l}^T$ is comprised of orthogonal vectors of the kernel of pre-activations $Z_l$, and $a \in \mathbb{R}^{n-M_l}$ is a random vector resampled for each candidate.

As a baseline to these selection-based techniques, NORTH-Random uses RandomInit as the initialization strategy with random fan-in weights and zeroed fan-out weights.

For pre-activation based initialization, we compute the Moore-Penrose pseudoinverse $(H_{l-1}^T)^+$, which approximates the left inverse when $n \gg M_{l-i}$ and the rows of $H_{l-1}$ are approximately orthogonal. The orthogonal vectors of the kernel of pre-activations $Z_l$ are computed as the right $n-M_l$ columns of $V_{Z_l}$ from the full singular value decomposition of $Z_l$.

In addition to the two methods presented for creating candidates for activation-based selection at neuron initialization, we also present an optimization-based approach. We perform projected gradient
descent [258] on randomly generated candidate neurons, optimizing the orthogonality metric of each set with the existing neurons and constrained by weight norm.

When the effective dimension metric is used, we need a continuous proxy for selection across all three methods as well as a differentiable proxy for the projected optimization. We use the sum of singular values as the differentiable proxy as well as the tie-breaker for neurons that yield equivalent effective dimensionalities for the layer.

However, the optimization method was found to be prohibitively expensive, performing an inner optimization for every growth event with an additional projection step for every gradient step. Further study into post-activation orthogonality metrics and proxies is warranted, particularly balancing performance with efficiency.

**Weight-based** For the weight-based strategy NORTH-Weight, maximally orthogonal weights for new neurons can be solved for directly. We initialize $k$ neurons each with fan-out weights of 0 and fan-in weights projected onto the kernel of $W_l$:

$$w = \text{proj}_{\text{ker}W_l}(w_i), \qquad (34)$$

where $w_i \in \mathbb{R}^{M_{l-1}+1}$ are the random initial fan-in weights from the base weight initialization and the projection is computed using $V'_{W_l}$, comprised of orthogonal vectors of the kernel.

**Gradient-based** We reimplement the neurogenesis initialization portions of NeST from [68], Firefly from [110], and GradMax from [142] within our framework to investigate them in the dynamic context and be able to compare them with our dynamic methods.

* NeST from [68]: initialize $k$ neurons, using sums of squares of the largest auxiliary gradient matrix values with a random sign to the respective fan-in and fan-out weights as connections. As NeST is only specified for a single neuron, we adapt to $k$ neurons by using different random signs for each neuron. We use the same recommended top quantile of 0.4 for filtering the largest auxiliary gradient matrix values. In the convolutional case, generate random candidate neurons and add those that independently lower the loss the most.

* Firefly from [110]: generate candidate neurons from two strategies: split existing neurons by halving the fan-out weights and adding and subtracting random uniform noise from the two copies, and add new candidates with random fan-in weights and small fan-out weights. We then keep the $k$ neurons with the largest gradient norm. We do not perform gradient steps on the candidates, as done in Firefly, in order to focus on neurogenesis rather than pruning after training; Firefly does both. The magnitude of the noise for split candidates and the fan-out weights for new candidates is controlled by hyperparameter $\epsilon_{\text{Firefly}}$, for which we use a value of 1e-4.

* GradMax from [142]: initialize $k$ neurons with zero fan-in weights and the top $k$ left singular vectors of the auxiliary gradient matrix as the fan-out weights. We note that this method is limited to $M_{l+1}$ candidate neurons, capping neurogenesis at the size of the next layer.

### 6.2.3 Neurogenesis for Convolutions

For convolutional layers, we consider that a channel is analogous to a neuron in a dense layer. However, the activation for a channel and a single sample as well as the parameterization of the connection between two channels are both matrices instead of single values. Thus, $H_l \in \mathbb{R}^{H_l \times W_l \times M_l \times n}$ and $W_l \in \mathbb{R}^{k_l \times k_l \times M_l \times (M_{l-1}+1)}$. We detail how our methods are adapted to convolutions as follows.
* Effective Dimension: $H_l$ is flattened to $\mathbb{R}^{M_l \times (H_l W_l n)}$, so that the metric is relative to the number of neurons in layer $l$. This permits the use of a smaller buffer size.

* Orthogonality Gap: $H_l$ is flattened to $\mathbb{R}^{(H_l W_l M_l) \times n}$, so that $H_l^\top H_l \in \mathbb{R}^{n \times n}$.

* Activation-based Trigger: Because the orthogonality metric at initialization is very close to 0 for the last layers of a deep convolutional network and increases as the network learns, we instead use the running maximum of orthogonality metric values for the threshold. Thus,

$$T_{act}^{conv}(f, \phi_a, l) = \min(0, \lfloor M_l (\phi_a(f, l) - \gamma_a \max_t \phi_a(f_t, l)) \rfloor). \quad (35)$$

In the case of residual networks, we turn off residual connections when evaluating activation-based metrics to isolate the contribution by each non-identity layer. We also grow groups of layers with interdependent channel constraints due to these residual connects based on metrics of the first layer of the group.

* Weight-based Trigger and Initialization: $W_l$ is flattened to $\mathbb{R}^{M_l \times ((M_l-1)k_l k_l)}$.

* Gradient-based Trigger: While we did not have enough resources to include results for all dynamic gradient-based neurogenesis strategies in convolutional neural networks, the implementation for these are included in our code. Computing the auxiliary gradient is detailed in [142].

* Activation-based Initialization: We do not have a closed-form solution for orthogonal pre-activations of convolutions, so we do not implement NORTH-Pre for architectures with convolutions.

## 6.3 Experiments

We study the impact of the different trigger and initialization methods detailed in Table 7 over a variety of tasks, with dynamic schedules to study triggers and independently studying the importance of initialization by also using fixed schedules. We also compare with static networks of various sizes to understand the relative performance of networks grown with neurogenesis. Specifically, we focus on three settings: MLPs [259] on generated toy datasets in Section 6.3.2, MLPs on MNIST [260] in Section 6.3.3, and convolutional neural networks (CNNs) on image classification: VGG-11 [261] and WideResNet-28 [262] on CIFAR10/CIFAR100 in Section 6.3.4. For all experiments, plots show mean and standard deviation in the form of error bars, clouds, or ellipses (which may result in artifacts outside of the layer size bounds) or quartiles in the form of boxplots across 5 seeded trials. We include further experiment details in Section 6.3.1 and a study on hyperparameters in 6.3.5. We only compare our results within our framework rather than take reported results in order to avoid confounding differences in implementation and focus on understanding neurogenesis. By studying neurogenesis across different tasks, architectures, and layer types, we aim to draw general conclusions about when, where, and how to add new neurons to ANNs.

### 6.3.1 Experiment Details

We run 5 trials with random seeds 1-5 in each study for each configuration and present aggregated results. We list hyperparameters in Table 8. For initializations that generate candidates, the number of candidates is set to either 1000 for dense layers or 100 for convolutional layers, plus the number of neurons to add. For all initializations, the bias is initially set to 0 and non-zero fan-in weights and fan-out weights of new neurons are scaled to match the current weight norm of existing neurons in the same layer for all initializations, except for Firefly for which we scale noise vectors to $\epsilon_{Firefly} = 1e-4$ times the current weight norm of existing neurons. We use a modified ReLU activation function as GradMax
<table><caption>**Table 8:** Default hyperparameters.</caption><thead><tr><th></th><th>MLP - Generated</th><th>MLP - MNIST</th><th>VGG-11 - CIFAR10</th><th>VGG-11 - CIFAR100</th><th>WRN-28 - CIFAR10</th></tr></thead><tbody><tr><td>Optimizer</td><td>ADAM [263]</td><td></td><td></td><td>ADAM, CosineAnnealing</td><td></td></tr><tr><td>Learning Rate</td><td></td><td>3e-4</td><td></td><td></td><td>3e-3</td></tr><tr><td>Batchsize</td><td>128</td><td>512</td><td></td><td>128</td><td></td></tr><tr><td>Epochs</td><td>Convergence</td><td>20</td><td>100</td><td></td><td>50</td></tr><tr><td>Input dimensions</td><td>64</td><td>784</td><td></td><td>32 x 32 x 3</td><td></td></tr><tr><td>Output dimension</td><td>2</td><td>10</td><td>10</td><td>100</td><td>10</td></tr><tr><td>Hidden layers/groups</td><td>1 or 2</td><td>2</td><td>10</td><td></td><td>4</td></tr><tr><td>Initial layer width</td><td>4</td><td>64</td><td></td><td>0.25x</td><td></td></tr><tr><td>Medium Static width</td><td>N/A</td><td>256</td><td></td><td>1x</td><td></td></tr><tr><td>Maximum layer width</td><td>512</td><td>784</td><td>2x</td><td></td><td>6x</td></tr><tr><td>Buffer size</td><td>1024</td><td>1568</td><td></td><td>128</td><td></td></tr><tr><td>Base initialization</td><td></td><td></td><td>Xavier uniform [264]</td><td></td><td></td></tr><tr><td>Orthogonality metric</td><td></td><td></td><td>φ<sub>ED</sub></td><td></td><td></td></tr><tr><td>ε, eq. (27),(30)</td><td></td><td></td><td>0.01</td><td></td><td></td></tr><tr><td>γ<sub>a</sub>, eq. (28)</td><td>0.97</td><td></td><td>0.9,0.97,0.99</td><td></td><td>0.9</td></tr><tr><td>γ<sub>w</sub>, eq. (31)</td><td></td><td></td><td>0.99</td><td></td><td></td></tr><tr><td>Device</td><td>CPU</td><td></td><td></td><td>GPU</td><td></td></tr></tbody></table>

requires that the gradient at 0 is 1; in other words, $σ(x) = 0$ if $x < 0$ and $σ(x) = x$ if $x ≥ 0$, resulting in the same activation but a different gradient at 0 compared with standard ReLU $σ(x) = max(0, x)$. We noted no empirical difference in results between the standard and modified ReLU and thus use the modified ReLU across all configurations [265].

We compare growing networks against static networks: Small Static is the same size as the initial size of growing networks, Medium Static is the same size as the final size for preset schedules and is also the baseline used for preset schedules, and Big Static is the same size as the maximum size of growing networks.

All plots except 32 average across the 5 random seeds and use standard deviation for error bars, clouds, or ellipses. The ellipses represent the contour line of a Gaussian density function matching the mean and covariance at 1 standard deviation.

We ran all CPU experiments on Intel Xeon Gold 6136 computing nodes and the GPU experiments on GTX 1080TIs, RTX8000s, and Tesla V100s, all provided by local clusters. The experiments to generate all plots and tables in this work consumed 299.53 CPU days and 85.17 GPU days in total, with an estimated carbon footprint of 236.3 kgCO₂e [266, 267].

### 6.3.2 Generated Data

<figure><img src="image_42.png" /><figcaption>**Figure 28:** Hidden neurons upon convergence on for MLPs on generated toy data versus the number of independent features in a 1 (left) and 2 (middle, right) hidden layer MLPs.</figcaption></figure>

We first analyze all strategies on supervised learning for toy generated binary classification datasets. All have 64 input features but differ in how many of these inputs are linearly independent, demonstrating how neurogenesis responds to the effective dimensionality of the input. The datasets are created for
a given number of independent features $n_f$ by generating 5000 samples of $n_f$ normally distributed features and filling the remaining 64 – $n_f$ features with random linear combinations. The output binary class for each sample is generated by summing all features and adding 10% random noise, then thresholding by the mean value. 10% of samples are reserved as the test split for final accuracy. We train MLPs with 1 or 2 hidden layers using the different neurogenesis strategies in all layers, shown in Figure 28. In this simple task, all strategies achieve similarly high test accuracy across datasets: above 99% for all feature sizes except 97% for 32 independent features.

Comparing final layer widths shows the dependency of each strategy on the effective input dimensionality and base architecture. The first layers across the two architectures exhibit the unsupervised nature of NORTH\* strategies but also the strength of gradient-based strategies to adapt width to depth. Width of the second hidden layer has higher variance across all strategies, indicative of the layer-wise input distribution: the input to the second layer is changing in dimension and distribution, whereas the first hidden layer has a static input from the dataset in this task.

NORTH-Weight is restricted by the input dimensionality due to the upper bound on number of singular values by the minimum of the matrices' dimensions. Thus, it cannot explore networks wider than the input dimensionality, which may be problematic for tasks with low-dimensional inputs. All other NORTH\* methods grow to the maximum size allowed in the first layer and to larger sizes than NORTH-Weight in the second layer.

### 6.3.3 MLP MNIST

<figure><img src="image_43.png" /><figcaption>**Figure 29:** Dynamic (left) and preset scheduled (right) comparisons of network performance across neurogenesis strategies for MLPs on MNIST, evaluated by test accuracy (top) and training time (bottom). Preset scheduled are compared against the Medium Static network, which is the same architecture as the final size of the grown networks.</figcaption></figure>

We continue our study on neurogenesis in dense MLP layers on the task of MNIST classification. The 28 × 28 images of the MNIST dataset are flattened to 784 features. This is fed into an MLP with 2 hidden layers which grow using neurogenesis, then classified into 10 classes representing the digit in the image. We use the standard training and test split.

We test dynamic strategies as well as isolate the initializations in preset growth schedules. To better understand the contribution of the trigger versus initialization, we compare initialization methods using fixed neurogenesis schedules. We use two predetermined schedules: Linear, which adds one
<figure><img src="image_44.png" /><figcaption>**Figure 30:** Layer width for MLPs on MNIST over training for dynamic strategies, across training (far-left for Layer 1, center-right for Layer 2) and at the end of training (center-left for Layer 1, far-right for Layer 2).</figcaption></figure>

neuron per gradient step, and Batched, which adds batches of neurons after each gradient step. Both schedules are defined by initial and final layer widths. We grow neurons for the first 75% of epochs. The linear schedule adds neurons one at a time so it has many small growth events, while the batched schedule adds neurons in 8 regular intervals so it has only a few large growth events. We compare with a Medium Static network which starts training with the final size of the growing networks, 256 neurons per hidden layer. We compare test accuracy and training time against final network size or schedule in Figure 29, detailing the layer widths over training for dynamic schedules in Figure 30.

For dynamic neurogenesis, the Pareto front of accuracy versus network size is dominated by the NORTH\* orthogonality-based methods, demonstrating the advantage of using activation and weight orthogonality to trigger neurogenesis.

The gradient-based methods, especially GradMax and NeST, yield comparable but not quite competitive results, possibly confounded by the adaptations used to implement them within our framework. However, they are faster in this CPU implementation than some activation-based NORTH\* strategies, benefiting from thorough code optimization of deep learning libraries for gradient descent, while the larger SVD calculations used by NORTH\* strategies hinder efficiency.

From Figure 30, no dynamic strategy reached the maximum possible network size. Thus, our triggers do respond to the "When" and "How many" questions, converging towards smaller networks. Furthermore, NORTH-Select and NORTH-Random reach networks which are competitive with the largest non-growing network with fewer neurons. NORTH\* methods tend to converge towards networks with large first layers and smaller second layers, a common pattern in hand-designed networks. Thus, the triggers can also respond to the "Where" question by dynamically adding neurons to each layer independently. However, using a maximum layer size is still needed for these methods to avoid exploding layer widths, particularly without extensive hyperparameter tuning.

As for preset schedules, multiple NORTH\* strategies exceed the performance of the static network of the same final architecture for this task and hyperparameter setting, demonstrating that dynamically growing a network over learning can result in better performance than starting with a network of the maximum final size. NORTH-Pre slightly under-performed the less-informed approaches of NORTH-Select and NORTH-Random, as well as NORTH-Weight which has a similar premise towards orthogonal pre-activations.

### 6.3.4 Deep CNNs on CIFAR10/CIFAR100

We extend neurogenesis to deep convolutional networks and test them on CIFAR10 and CIFAR100 classification with a VGG-11 or WideResNet-28 backbone. We use the standard training and test splits of CIFAR10 and CIFAR100, which have input images of size 32 × 32 × 3 and 10 or 100 classes, respectively. The standard VGG-11 hidden layer widths are 64, 128, 256, 256, 512, 512, 512, and 512 channels in the
<figure><img src="image_45.png" /><figcaption>**Figure 31:** Comparisons of test accuracy (top) and training time (bottom) vs. network size across neurogenesis strategies and values of γ for VGG-11 on CIFAR10 (left) and CIFAR100 (center) and WideResNet-28 on CIFAR10 (right).</figcaption></figure>

<table><caption>**Table 9:** Average GPU seconds per mini-batch for trigger evaluation for all layers, including GPU memory garbage collection time, during the 1st epoch of VGG-11 training on CIFAR10 on a Tesla V100.</caption><thead><tr><td>Strategy</td><td>Time</td></tr></thead><tbody><tr><td>NORTH-Select</td><td>5.97 ± 0.18</td></tr><tr><td>NORTH-Rand</td><td>5.77 ± 0.09</td></tr><tr><td>NORTH-Weight</td><td>10.05 ± 1.01</td></tr><tr><td>GradMax</td><td>14.00 ± 7.03</td></tr><tr><td>Firefly</td><td>18.52 ± 11.59</td></tr><tr><td>NeST</td><td>23.79 ± 8.78</td></tr><tr><td>Small Static</td><td>5.49 ± 0.05</td></tr><tr><td>Medium Static</td><td>5.49 ± 0.05</td></tr><tr><td>Big Static</td><td>5.14 ± 0.05</td></tr></tbody></table>

convolutional layers, then 4096 and 4096 neurons in the dense layers. The standard WideResNet-28-1 has an initial convolution layer with 16 channels, 3 groups with 16, 32, and 64 channels each and 4 residual blocks per group, and a final dense layer connecting global mean pooling to the output. The backbone used as the starting point for neurogenesis has $\frac{1}{4} \times$ the standard layer widths, also used for the Small Static architecture. The layer sizes for growing networks were limited to $2 \times$ the standard layer widths for VGG-11 and $6 \times$ the standard layer widths for WideResNet-28-1 (which is WideResNet-28-6 in their notation), also used for the Big Static architecture. hlDue to the interdependence of layer widths in WideResNet-28, we measure the first layer of each group to determine when, where, and how to add neurons to all layers of the same group. We also turn off the residual connections when evaluating activation and weight-based metrics. We use batch-normalization in WideResNet-28, but otherwise do not perform any other “tricks” such as data augmentation or dropout. Due to restricted computational time, we could not perform a formal sweep of hyperparameters. We selected the largest learning rate that we tested that allowed all networks up to Big Static to increase accuracy during the first 5 epochs. Similarly, we selected the largest batchsize that we tested that could fit any gradient computation in memory on all GPU devices. The cluster GPUs available to us were not identical, so we ran each base architecture and dataset combination on the same GPU in order to compare within
<figure><img src="image_46.png" /><figcaption>**Figure 32:** Layer widths for VGG-11 on CIFAR10, scaled relative to standard VGG-11 widths, for NORTH-Select, NORTH-Rand, and NORTH-Weight, over the course of training in (a) and at the end of training in (b).</figcaption></figure>

experiments, but not across.

The gradient-based strategies require computing auxiliary gradients in addition to the gradients of existing parameters and any gradients further required in gradient-based initialization, shown in Table 9. Because batchsizes are generally selected to maximise GPU memory of each gradient evaluation and the gradient-based neurogenesis methods use multiple passes per batch, they are less efficient than NORTH* in the case of VGG-11.

Test accuracy and training cost of the different methods is compared against network size for each base architecture and dataset combination in Figure 31. On VGG-11, our NORTH* methods find networks in the same order of size of the standard network, here represented by Medium Static, while being competitive or improved in accuracy for both datasets. On CIFAR10, we search across $\gamma_a$, showing the tunability of NORTH-Select and NORTH-Rand across the trade-off for size and performance. Notably, NORTH-Select with $\gamma_a = 0.9$ outperforms even Big Static with less than half as many parameters on both CIFAR10 and CIFAR100. As in the dense case, NORTH-Select has a slightly higher cost for the benefit of smarter candidate generation compared to NORTH-Rand. For WideResNet-28, NORTH* methods tended to grow to the at or near the maximum size, likely due to the complicated dynamics of residual connections on the trigger metrics. NORTH-Weight slightly beats Big Static at this maximum size while NORTH-Select matches performance, and GradMax beats Medium Static at a similar size.

Layer width during and at the end of training for VGG-11 on CIFAR10 is presented in Figure 32, showing how different NORTH* methods respond to the "Where" question. All three methods keep the dense layers relatively small, which has a large effect on the overall parameter count. NORTH-Weight grows some intermediate convolutional layers to maximum size, almost opposite to the width patterns of NORTH-Rand. Deeper convolutional layers tend to have later innovations, likely in response to innovations of the earlier layers.
### 6.3.5 Hyperparameter studies on MLPs for MNIST

In the following experiments, we study the effect of NORTH\* hyperparameters independently in the MNIST task. These experiments were used to inform our default settings for the experiments in 6.3.3. We discuss our findings in the context of this specific task and setup. We only use the standard training split of MNIST in these experiments, holding out 10% of this split as the validation set to measure final validation accuracy.

We study the effects of activation orthogonality measures Effective Dimension $\phi_a^{ED}$ from Equation 27 versus Orthogonality Gap $\phi_a^{OG}$ from Equation 29 on performance and training time and their sensitivities to $\gamma_a$ in Figure 33. $\phi_a^{ED}$ generally yields higher accuracies given the network size, although $\phi_a^{OG}$ is generally more efficient. $\phi_a^{ED}$ is more robust to varying $\gamma_a$ and yields less extreme network sizes, although both metrics are rather sensitive to its value. Thus, applying activation-based neurogenesis to new tasks may require tuning of $\gamma_a$ to the specific task configuration. $\phi_a^{ED}$ has the additional hard constraint on sample size being greater than the current layer width, which may make scaling to very wide networks difficult.

We study the effect of batchsize in Figure 34a. We note that the size of the buffer used for assessing the orthogonality metric is independent of batch size. The dynamic networks generally benefit more from larger batchsizes than static networks. NORTH-Select particularly matches or exceeds the validation performance of larger static networks at larger batchsizes. We hypothesize that the relative frequency of neurogenesis trigger and initialization steps for larger batchsizes is beneficial.

The results of various learning rates are shown in Figure 34b. The general trend is that larger networks, both static and growing, benefit from smaller learning rates. NORTH-Select matches NORTH-Random in validation performance at smaller learning rates, even with a smaller network size. We conjecture that larger learning rates cause large, noisy gradient steps that hinder the activation-based triggering of neurogenesis while increasing randomness and thus orthogonality for weight-based triggering.

## 6.4 Discussion

NORTH\* strategies grow efficient networks which achieve comparable performance to static architectures, even surpassing the performance of static architectures of the same final size. We posit that the use of growth with informed triggers leads to efficient networks as redundant neurons have a low chance of being created in the small network initialization and are not added by NORTH\* neurogenesis. This highlights a benefit of neurogenesis: dynamically creating an architecture adapted to the target task.

We find that activation and weight orthogonality can be useful triggers for dynamic neurogenesis and weight initializations of new neurons. Previous methods use gradient information for these decisions, explicitly considering training dynamics and particularly the outputs via the loss. Our NORTH\* methods only implicitly use this information via network updates changing orthogonality metrics and explicitly use the input distribution. This could allow extension to unsupervised and semi-supervised contexts, where gradient information may be unavailable or unreliable.

As neurogenesis triggers are evaluated after every gradient step, they can have significant computational cost. NORTH\* strategies were more efficient than gradient-based strategies in our GPU experiments, but less for CPU. The number of candidates, frequency of trigger evaluations, and the size of the activation buffer are hyperparameters of NORTH\* which can be reduced for efficiency or increased for performance. Comparing the cost of growing networks to that of static networks is difficult: although dynamically grown networks incur higher training costs than predetermined schedules or non-growing networks, they have the benefit of algorithmically determining the architecture widths instead of hand-engineering them through expensive trial-and-error.
We focus on function-preserving neuron addition. Such neurons do not immediately impact network output while still receiving gradient information to become functional through training. This can be easily achieved by zero-ing either the fan-in weights (as in NORTH\*) or the fan-out weights (as in GradMax). Firefly and NeST, however, initialize functional neurons that change the position in the loss landscape; this could hinder performance by moving the network in an undesired direction, but could also complement gradient descent through neuron initialization. Optimizing the fan-out weights of NORTH\* strategies to make helpful movements in the loss landscape is a direction for future work.

This work serves as a study to isolate neurogenesis from other structural learning techniques in order to understand it more and open the door to further neurogenesis research. We pair complimentary triggers and activations; other combinations are possible albeit with careful consideration. We show that neurogenesis alone can already find compact yet performant networks. We leave dynamic structural learning, incorporating neurogenesis along other structural operators like layer addition and pruning, as future work. For layer addition, we hypothesize that similar activation and weight-based triggering applies, then layer addition candidates could be compared to neuron addition candidates; for gradient-based methods, evaluating the auxiliary gradient on directly neighboring layers rather than alternating could signal layer addition. As for pruning, we hypothesize that unstructured pruning may not be necessary with our strategies that avoid redundancy at initialization, but structured pruning could remove neurons that become redundant over training. This could bypass the need for hyperparameters such as maximum layer width and be even more effective towards searching for networks that optimize both performance and cost.

**Limitations and Broader Impact Statement:** In addition to the limitations and broader impacts already discussed, the efficient and adaptive architectures grown by neurogenesis have reduced environmental footprint, avoiding cycles of hand-tuning static architectures or training expensive super-networks to prune or apply NAS to. We do not identify any potential negative societal impacts beyond those associated with general-purpose machine learning methodologies.

## 6.5 Conclusion

We present the NORTH\* suite of neurogenesis strategies, comprised of triggers that use orthogonality metrics of either activations or weights to determine when, where, and how many neurons to add and informed initializations that optimize the metrics. NORTH\* strategies achieve dynamic neurogenesis, growing effective networks that converge in size and can outperform baseline static networks in compactness or performance. The orthogonality-based neurogenesis methods in NORTH\* could be further used to grow networks in a variety of settings, such as continual learning, shifting distributions, or combined synergistically with other structural learning methods. Neurogenesis can grow network architectures that dynamically respond to learning.

Understanding neurogenesis in ANNs permits its application as an operation in structural learning algorithms, particularly as a complement to pruning. This expands the potential search space used during structural learning, permitting more powerful and adaptive methods. Furthermore, the investigated dynamic scheduling aims to perform neural creation exactly when necessary and useful, removing the need for manually designed scheduling and increasing adaptivity. This study on neurogenesis is extended with neural pruning for dynamic learning environments such as transfer learning in the following section.
<figure><img src="image_47.png" /><figcaption>**Figure 33:** Comparison of validation performance and sensitivity to $\gamma_a$ of $\phi_a^{ED}$ and $\phi_a^{OG}$ in activation-based strategies.</figcaption></figure>
<figure><img src="image_48.png" /><figcaption>**Figure 34:** Effects of (a) batchsize and (b) learning rate on NORTH-Select and NORTH-Random, compared to dynamic and static baselines.</figcaption></figure>
# 7 Growing and Pruning ANNs

The preceding work on filling foundational gaps in structural learning has naturally led to investigation beyond the simple paradigm of statically distributed supervised learning of a single task from scratch, which provides an *in vitro* test bed for developing new techniques, towards more comprehensive and realistic problems. The typical paradigm of hand-designing static architectures is not scalable with the growing desire to apply ANNs in more complex and even dynamic tasks, where the training data distribution may change over time. The simplest form of a dynamic learning environment is transfer learning, where an ANN may be generally pre-trained on a source dataset before fine-tuning a specific copy on the target dataset, improving computational reuse and even performance on the target task. Fine-tuning usually consists of the same generic training approach of gradient descent with a static architecture.

As intermediate activations are effective for out-of-distribution (OoD) detection [268], the inverse suggests they could also inform neural growth [20] and pruning. During fine-tuning on the target task, some existing features may be more useful or adaptable to the target task than others: neurons that are redundant with others or show OoD activation patterns such as dormancy [269] may be pruned and new neurons with incoming distribution-aware initialization may be added. This strategically adapts the model to the new domain by augmenting the most useful components of the inductive bias transferred from pre-training.

We propose and investigate the utility of growing and pruning neurons during transfer learning, resulting in a fine-tuned model with a customized architecture for the target task. We study the use of informed scheduling for the growth and pruning operations, removing the hand-designed schedules and algorithmic details seen in other comparable approaches. We define the DYNO (Dynamic Neural Optimization) grow-and-prune algorithm: the proposed strategies that comprise DYNO only use activation information from just the forward pass, without any masking or gradient calculations. We study DYNO in multiple transfer learning scenarios, analyzing how different layers respond in across various transfer contexts. DYNO yields dynamically shaped efficient models with tuned layer widths and tuned parameters, without predetermined architectural scheduling.

## 7.1 Problem Definition

We adopt the same notation as Section 6.1. An artificial neural network (ANN) $f$ may be optimized through empirical risk minimization of a loss function $L$ for a dataset $D$ consisting of inputs $x$ and outputs $y$:

$$ \arg\min_{f} \mathbb{E}_{x,y \sim D} L(f(\mathbf{x}), \mathbf{y}). \qquad (36) $$

For a dense multi-layer perceptron (MLP) with $d$ hidden layers, $f$ may be expressed as $f(\mathbf{x}) = \sigma_{d+1}(\mathbf{W}_{d+1}\sigma_d(...\mathbf{W}_2\sigma_1(\mathbf{W}_1\mathbf{x})...))$, where $\sigma_l$ is a nonlinear activation function that also adds a row for the bias, and $\mathbf{W}_l \in \mathbb{R}^{M_l \times (M_{l-1}+1)}$ is the weight matrix for the $l^{th}$ layer, including the bias parameters. We define $\mathbf{h}_l = \sigma_l(\mathbf{W}_l\sigma_{l-1}(...\mathbf{W}_2\sigma_1(\mathbf{W}_1\mathbf{x})...))$ as the post-activations of layer $l$. For $n$ samples, $\mathbf{H}_l \in \mathbb{R}^{M_l \times n}$ is the post-activation matrix.

For convolutional layers, we consider that a channel is analogous to a neuron in a dense layer. However, both the activation for a channel and a single sample as well as the parameterization of the connection between two channels are matrices instead of single values. Thus, $\mathbf{H}_l \in \mathbb{R}^{H_l \times W_l \times M_l \times n}$ and $\mathbf{W}_l \in \mathbb{R}^{k_l \times k_l \times M_l \times (M_{l-1}+1)}$.

Standard ANN training procedures pre-define the size and structure of the weight matrices $W$, updating their parameters via stochastic gradient descent over mini-batches sampled from the current dataset towards optimizing Equation (36).
**Algorithm 5** A general framework for neural grow-and-prune optimization.

**procedure** GROW-AND-PRUNE(TRIG<sub>GROW</sub>, INIT<sub>GROW</sub>, TRIG<sub>PRUNE</sub>, SELECT<sub>PRUNE</sub>, initial ANN *f*)

**while** *f* not converged **do**

**for** each hidden layer *l* **do**

**if** TRIG<sub>PRUNE</sub>(*f*, *l*) > 0 **then**

Remove TRIG<sub>PRUNE</sub>(*f*, *l*) neurons by SELECT<sub>PRUNE</sub>(*f*, *l*)

**if** TRIG<sub>GROW</sub>(*f*, *l*) > 0 **then**

Add TRIG<sub>GROW</sub>(*f*, *l*) neurons using INIT<sub>GROW</sub>(*f*, *l*)

**for** *n* steps **do**

Gradient descent step on current existing weights and dataset

**return** trained *f*

To perform neurogenesis or neural pruning, the addition or removal of *k* neurons to the *l*th layer is accomplished by appending or removing *k* rows of fan-in weights in *W*<sub>*l*</sub> and *k* columns of fan-out weights in *W*<sub>*l*+1</sub>. Utilizing these operations in the empirical risk minimization of Equation (36) defines the neural grow-and-prune optimization. We restrict the search space of where to add new neurons to within existing layers.

## 7.2 Grow and Prune for Transfer Learning

Our general proposed framework for neural grow-and-prune is presented in Algorithm 5. This framework cycles through each optimization operation of pruning, growth, and parameter optimization via gradient descent, allowing the grow-and-prune details such as the trigger strategies, growth initialization strategy, and pruning selection strategies to be defined alongside other training details such as the gradient descent optimizer. We propose the **Dynamic Neural Optimization (DYNO)** grow-and-prune algorithm with the strategies defined as follows.

Following [20], we define triggers and initialization/selection strategies for each operation of growing neurons and pruning neurons. The trigger is evaluated regularly to determine, for each layer, how many neurons to grow or prune. The initialization strategy for growing or selection strategy is then applied to carry out the structural change.

The trigger for both operations is based on the orthogonality metric of *effective dimensionality*, the *ε*-numerical rank of the post-activation matrix [253, 257]. For layer *l* and *n* samples generating the post-activation *H*<sub>*l*</sub>, the effective dimension metric *φ*<sub>ED</sub> may be estimated by

$$ \phi_{ED}(f, l) = \frac{1}{M_l} \left| \left\{ \sigma \in \text{SVD}\left(\frac{1}{\sqrt{n}}\mathbf{H}_l\right) \middle| \sigma > \epsilon \right\} \right|, \quad (37) $$

where SVD($\frac{1}{\sqrt{n}}\mathbf{H}_l$) is the set of singular values of $\frac{1}{\sqrt{n}}\mathbf{H}_l$ and $\epsilon > 0$ is a small threshold. For convolutional layers, $\mathbf{H}_l$ is flattened to $\mathbb{R}^{M_l \times (H_l W_l n)}$, so that the metric is relative to the number of neurons or channels in layer *l*.

A significant increase in the orthogonality metric is used to trigger neurogenesis, while a significant decrease is used to trigger pruning:

$$ \text{TRIG}_{\text{PRUNE}}(f, l) = \min(0, \lfloor M_l ((1 - \delta)\phi_{\text{ED}}^* - \phi_{\text{ED}}(f, l)) \rfloor), \quad (38) $$

$$ \text{TRIG}_{\text{GROW}}(f, l) = \min(0, \lfloor M_l (\phi_{\text{ED}}(f, l) - (1 + \delta)\phi_{\text{ED}}^*) \rfloor), \quad (39) $$

where $\delta > 0$ is a small threshold and $\phi_{ED}^*$ is the value of $\phi_{ED}(f, l)$ at the last structural change of *f*. This combination of trigger strategies slims the layer when its activations are expressible in fewer
dimensions, and expands it when the feature space in near expressible capacity. The updating baseline
allows structural operations to occur dynamically in response to changes in the training distribution,
with minimal dependence on the frequency of trigger evaluation.

Neurons to prune are selected based on a greedy cosine similarity algorithm, `SELECT PRUNE(f, l)`. In the case of ReLU-activated layers, all completely dormant neurons are first pruned. Then, until the total number of neurons to prune is reached, the neuron with the highest norm of cosine similarities of normalized post-activations with other remaining neurons is greedily pruned.

New neurons are grown with the initialization strategy, `INIT GROW(f, l)`, as in NORTH-Select [20]: from candidates with scaled random fan-in vectors and zeroed fan-out vectors, select neurons that independently maximize $\phi_{ED}(f, l)$ with the existing neurons in that layer.

## 7.3 Experiments

We apply neural grow-and-prune transfer learning on three classification scenarios, each beginning
with VGG11 pretrained on Imagenet [270]. The target datasets are:

* **Imagenette**: 10 easily distinguishable classes of Imagenet [271], thus making high-level features useful but not detailed features that may distinguish more similar classes.

* ** Imagewoof**: 10 dog breed classes of Imagenet [271], thus requiring minute details within a subspace of the original distribution.

* **Galaxy10**: satellite images of galaxies in 10 shape-based categories [216], so the underlying distribution is very different from natural images as in Imagenet.

We additionally implemented Surgical Fine-Tuning (SFT) [272], which dynamically adjusts layer-wise
learning rates based on the layer's relative gradient norm, as an orthogonal approach to transfer
learning.

All experiments begin with the same Imagenet-pretrained VGG11 backbone, with standard layer widths. The results show the mean, standard deviation, and (when applicable) covariance across 5 trials each with different random seeds. For Galaxy10, the final classification layer is replaced with a reinitialized 10-way classification layer. For Imagenette and Imagewoof, the final classification layer is replaced with a 10-way classification layer consisting of the 10 respective classification neurons corresponding to each class. All trials are run for 1000 iterations with 64 samples per iteration, performing pruning and growth trigger evaluations every 50 iterations. The effective dimensionality threshold $\epsilon$ is 1e-2 and the trigger threshold $\delta$ is 3e-2. The optimizer is Adam with a learning rate of 3e-4 and weight decay of 1e-3. All metrics are evaluated on post-activations, which are tracked for each layer in a FIFO buffer that is dynamically sized to contain more samples than neurons, as necessary for the effective dimensionality calculation. Only forward pass information is required. The number of candidates generated for neural growth is $100 \times \text{TRIGGROW}(f, l)$. ImageNet, Imagenette, and Imagewoof are available freely for research purposes. Galaxy10 is available under the MIT license.

The results of applying DYNO compared to standard static fine-tuning are shown in Figure 35. When comparing test accuracy to inference FLOPs, the fine-tuned models have slightly lower average accuracy but are consistently more efficient in compute cost. Network sizes are consistent within tasks: models for Galaxy10 are about half as expensive as the original pre-trained model. DYNO causes immediately large architectural changes, then progressively smaller later in training, approaching architectural convergence but allowing small amounts of turnover well after the distribution change. Considering layer-wise architectural changes of DYNO, models across all three tasks had most change
<figure><img src="image_49.png" /><figcaption>**Figure 35:** Experimental results for transfer to Imagenette (top), Imagewoof (middle), and Galaxy10 (bottom), including combinations of DYNO and SFT [272]. First column: final test accuracy against final model inference FLOPs. Second column: progression of inference cost over the course of training. Third and fourth columns: total neurons grown and pruned per layer over the course of DYNO (third column) and DYNO+SFT (fourth column), relative to the original layer width. All covariance ellipses, error clouds, and error bars show standard deviation across the 5 trials.</figcaption></figure>

occur in the later layers of the network, particularly the final convolution (layer 7) and final dense layer before the classification layer (layer 9). Models for Galaxy10 had significantly more architectural change, both for pruning and growing, and in more layers.

To fine-tune on a similar task, such as transferring to Imagenette, DYNO made very few changes, similar to what would happen with standard pruning of a final layers. However, when distribution shifts, such as transferring to Galaxy10, the architecture changes throughout: major modifications are made in the final layer, and there are even structural modifications in the early layers. This shows that DYNO responds to distribution shift based on the learned task to appropriately modify the architecture.

DYNO was not able to surpass the test accuracy of larger static models. This may be due to the simple linear schedule of architectural evaluation without a long final tuning phase or any learning rate decay, nor extensive hyperparameter tuning. Such "tricks" are often employed for improving performance but may confound results so thus were avoided in this preliminary study.

The strategies of DYNO are intended to minimize the number and impact of hyperparameters. For example, the moving baseline used in the trigger functions reduces dependence on the trigger evaluation frequency. Further work on hyperparameter optimization and specialization is warranted, such as learnable thresholds [273] and layer-wise thresholds, which could improve test accuracy compared to static fine-tuning.

Towards structurally heterogeneous learning rates, we additionally reimplemented Surgical Fine-Tuning (SFT) [272] as an orthogonal transfer learning tool. We specifically use their Auto-RGN method, which dynamically scales the learning rate of each layer based on the relative gradient norm. The experimental results are shown in Figure 35. All hyperparameters were kept the same, with SFT hyperparameters used from Lee et al. [272].
With SFT, DYNO performs as well or better than static architectures across all tasks. However, DYNO+SFT
has significantly less architectural change: virtually none during transfer to Imagewoof, only late
changes for Imagenette, and immediate pruning but then subtle growth for Galaxy10. A potential
explanation is that SFT reduces some signaled need for grow-and-prune by heterogeneously and
dynamically tweaking learning rates. Further hyperparameter tuning towards synergy could yield
improved results: implementing SFT on a neuronal level could be even more beneficial.

A notable potential confounder is that SFT only reduces the learning rate from the baseline value used, so all trials using SFT had smaller learning rates overall. This explains why performance was improved for Imagenette and Imagewoof, which are very close in distribution to Imagenet, but was worse for Galaxy10.

7.4 Discussion

The DYNO grow-and-prune algorithm shows dynamic layer-specific responsiveness for architectural
changes during fine-tuning without the need for any hand-engineered bias, such has predetermined
schedules of architectural change. This is a benefit of dynamically informed scheduling and minimal
hyper-parameterization. This direction is important for the motivation of AutoML towards more
automation and less human engineering.

Grow-and-prune algorithms have a very complex design space due to the interplay of growing, pruning,
and training a model. This preliminary study only considered one intuition-guided instance within this
space by defining each of TRIG<sub>GROW</sub>, INIT<sub>GROW</sub>, TRIG<sub>PRUNE</sub>, and SELECT<sub>PRUNE</sub>, but the vastness prompts
algorithmic optimization to find useful combinations of strategies. Previous automated algorithmic
discovery methods such as [274, 275] could be extended to include neural growth and pruning as
potential operators during training.

Transfer learning is the first step from static supervised learning: further in the same direction include
continual learning and multi-task learning. These dynamic paradigms benefit from domain general-
ization and avoiding catastrophic forgetting, which can be supported architecturally. Architectural
optimization could help find efficient parameter sharing between distributions or tasks. Grow-and-
prune may naturally be effective for alleviating the decay in plasticity normally observed in ANNs
[276]. To avoid catastrophic forgetting, the pruning mechanism may be modified: activation informa-
tion could be used to mark which existing neurons could either be used with locked parameters or
be copied and fine-tuned. These architectural techniques may additionally detect new contexts, as
demonstrated by [268], to trigger appropriate architectural adjustment, potentially incorporated with
structurally aware representation disentanglement. These in tandem could ameliorate problems such
as representation collapse [253] and lack of generalization [277], in addition to finding dynamically
constructed architectures for improved efficiency.

7.5 Additional Related Works

Previous neural grow-and-prune works often focus on the "where" and "how" of pruning, using
generic schedules and neurogenesis initializations. The most comparable works that separately
consider informed neural creation and pruning are Firefly [110] and NeST [68], which use gradient-
based information. Further works perform less informed neural creation [101, 278] or ephemeral
pruning via masking [117]. [279] studies synaptic grow-and-prune for incremental learning and [280]
performs unstructured pruning during fine-tuning, or fine-pruning. Structured fine-pruning has
mostly been studied in the case of large language models (LLMs) driven by their prohibitively large
training and inference costs when unpruned [281]. [282] performs structured pruning driven by
redundancy metrics. No other known neural grow-and-prune works that allow dynamic layer widths
consider the transfer learning case nor use only activation-based information.
7.6 Conclusion

Grow-and-prune during transfer learning and other dynamic learning environments is an intuitively
useful yet challenging addition that warrants further investigation. With DYNO, we propose a frame-
work for triggering pruning and growing strategies and demonstrate that this can result in structural
changes during the learning process which respond to a downstream task distribution. This work
synthesizes dynamic scheduling from Sections 5 and 6 and neural creation from Section 6 towards
dynamic structural adjustment during distribution shift, beginning with structural adaptation for the
simple case of transfer learning.
# 8 Discussion

My experimental work has thus far focused on improving aspects of structural learning that are not well studied as identified via literature analysis for structural learning (Section 3), particularly extending NAS to architectural representations of equivariance (Section 4), scheduling and improving differentiable NAS (Section 5), isolating and improving methods for neural creation (Section 6), and studying the interplay of neural growth and pruning in dynamic learning environments (Section 7).

These contributions provide tools for the automation of architectural optimization at various structural levels, particularly focusing on more informed scheduling and expanded algorithmic applications. Scheduling, which comprises what structural changes to perform and when, where, how, and why to perform them, is important for achieving automation and adaptivity. Contributions in this manuscript, particularly of Sections 5-7, utilize dynamic heuristics of learning to inform scheduling so that these questions may be automatically answered. Expanding the applicability of structural learning is crucial for escaping the tight bounds of well-studied yet artificial benchmarks towards more realistic problems. Section 4 directly accomplishes this through a novel domain of structures that yield optimizably partial equivariance, while Section 6 investigates novel methods for adding to architectures, rather than just pruning from them. As Section 7 begins to demonstrate, the proposed tools fill multiple foundational gaps in structural learning such that methods may now be synergistically combined towards comprehensive and adaptive systems with real-world applications.

The more immediate and practical goals of this research direction are towards automated machine learning. The preceding chapters focus on dynamic architectural design, which can work synergistically with automated hyperparameter tuning and even algorithmic optimization. This path may eventually go towards automation of machine learning as a tool: rather than an expert machine learning engineer hand-designing the architecture, hyperparameters, and other aspects of the pipeline, a scientist could directly get a trained structurally-optimized model simply by defining their problem and providing potential data in an intuitive programming language.

Other long-term goals of this work are to break the convenient yet restricting conventions of deep learning towards more organic methods. The presented contributions are loosely inspired by neuroscience: the human brain achieves the best-known example of human-defined intelligence. However, the brain is still a dark gray box to us: we've only scratched the surface in elucidating its learning mechanisms even with exponential progress in neural imaging and modeling. The fields of neuroscience and neuro-inspired artificial intelligence may work interactively towards furthering the both understanding and capabilities of intelligent systems, whether biological or artificial.

Sparsity is a multi-dimension spectrum that overlaps with structure and priors: it can occur at different levels and even implicitly, such as the relative sparsity of convolutions or the dynamic sparsity of attention. The organization, modularity, and other human-comprehensible properties of such ANN structures allows us to work with more orders of magnitude of dimensions at a time. This has already enabled models up to billions of parameters in size but in a human-constrained search space, which contains only known structures and feasibly simple patterns of connectivity. The beauty of architectural optimization is potentially breaking these patterns beyond what humans can conceive, although at the possible, yet temporary, cost of interpretability.

## 8.1 Future Work

The cumulation of these contributions within the context of the current state of literature provides a basis for the upcoming interdisciplinary research directions presented here, in addition to those specifically discussed within each chapter.
### 8.1.1 Structural Plasticity in Reinforcement Learning

Reinforcement learning (RL) removes the assumption of immediate supervision, in the context of an ANN-controlled agent acting over time within an environment. Distribution shift may occur when the agent encounters a new domain of states, such as a new level of a video game or a faulty sensor. Applying a structural grow-and-prune algorithm, such as extending the work of Section 7, to this context could allow the agent to be more flexible and adapt to the given environment, balancing plasticity with stability. However, the RL paradigm poses additional challenges of measuring heuristics and gradient information with less supervision than in fully-supervised tasks.

### 8.1.2 Dynamic Equivariance Search

The tools and algorithms presented in Section 4 permit the relaxation of equivariance constraints during architectural optimization. By further developing a complementary operation to increase equivariance constraints, the discrete structural equivariance level could be searched more dynamically, in a similar fashion as neural pruning complementing neural creation in Section 7. Such an operation may utilize an equivariance distance metric both for scheduling as well as to determine the optimal projection to equivariance of a supergroup. It would necessarily not be a function-preserving morphism, similar to neural pruning. This equivariance-constraining operation could be used in tandem with the equivariance relaxation morphism in an evolutionary NAS algorithm similar to EquiNAS<sub>E</sub>, or in an algorithm more akin to that presented in Section 7 using scheduling heuristics such as an equivariance distance metric for each operation.

### 8.1.3 Structural Learning for Transformers

Most existing architectural optimization works have focused on architectural search spaces for relatively simple computer vision tasks. However, machine learning in the 2020s is shifting towards models adept at complex or even multi-modal structures of data. The attention-based transformer architecture has become the state-of-the-art for many domains, from simple computer vision to much more complex tasks. Structurally augmenting such models requires defining appropriate search spaces and adjusting the algorithms for navigating them efficiently due to their massive size. The grow-and-prune work of Section 7 may be extended to investigate transformers, where pre-training is standard practice: the structural strategies can be directly applied to different components such as hidden neurons, attention heads and matrix columns, and even entire blocks. This, along with extending geometric techniques as mentioned in Section 8.1.2 will further empower architectural search spaces on current hot topics such as specializing large language models (LLMs) and building geometric models for complex structured data and tasks.

### 8.1.4 Woke LLMs: Equivariance Towards Socially Unbiased Models

At the intersection of the preceding directions is evading social bias in LLM training [283]: the contributions towards optimizing partial equivariance from Section 4 may be extended towards achieving theoretically valid appropriate symmetry of LLMs to representations of social issues, being aware of constructs such as gender, race, and sexual orientation without being blind nor biased to them. The architectural constraints may be algorithmically proposed and interactively guided by human feedback, rather than completely learned from training data, to ensure latent biases in data are not learned.

## 8.2 Conclusion

Structural learning is a unique sub-field of machine learning with close ties to neuroscience, posited towards further automation and increasing artificial intelligence through the concurrent optimization of ANN architectures with their parameters. The work within this manuscript progresses our
understanding of structural learning, particularly dynamic scheduling and expanded tools to effectu-
ate structural change across various granularities of the architecture of an ANN, towards improved
automation, adaptivity, and applicability. These contributions complete a foundation for structural
learning, empowering the automation of machine learning towards more powerful and adaptable
models.
References

[1] W. S. McCulloch and W. Pitts, "A logical calculus of the ideas immanent in nervous activity," *The bulletin of mathematical biophysics*, vol. 5, no. 4, pp. 115-133, 1943.

[2] G. E. Moore *et al.*, "Progress in digital integrated electronics," in *Electron devices meeting*, vol. 21. Washington, DC, 1975, pp. 11–13.

[3] H. N. Mhaskar and T. Poggio, "Deep vs. shallow networks: An approximation theory perspective," *Analysis and Applications*, vol. 14, no. 06, pp. 829–848, 2016.

[4] T. Poggio, H. Mhaskar, L. Rosasco, B. Miranda, and Q. Liao, "Why and when can deep-but not shallow-networks avoid the curse of dimensionality: a review," *International Journal of Automation and Computing*, vol. 14, no. 5, pp. 503–519, 2017.

[5] G. Cybenko, "Approximation by superpositions of a sigmoidal function," *Mathematics of control, signals and systems*, vol. 2, no. 4, pp. 303–314, 1989.

[6] K. Hornik, "Approximation capabilities of multilayer feedforward networks," *Neural networks*, vol. 4, no. 2, pp. 251–257, 1991.

[7] B. G. Rash, J. B. Ackman, and P. Rakic, "Bidirectional radial Ca2+ activity regulates neurogenesis and migration during early cortical column formation," *Science advances*, vol. 2, no. 2, p. e1501733, 2016.

[8] K. Fukushima, "Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position," *Biological cybernetics*, vol. 36, no. 4, pp. 193–202, 1980.

[9] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," *Proceedings of the IEEE*, vol. 86, no. 11, pp. 2278–2324, 1998.

[10] T. S. Cohen, M. Geiger, and M. Weiler, "A general theory of equivariant cnns on homogeneous spaces," *Advances in neural information processing systems*, vol. 32, 2019.

[11] L. J. Ba and R. Caruana, "Do deep nets really need to be deep?" in *Proceedings of the 27th International Conference on Neural Information Processing Systems-Volume 2*, 2014, pp. 2654–2662.

[12] J. Frankle and M. Carbin, "The lottery ticket hypothesis: Finding sparse, trainable neural networks," in *International Conference on Learning Representations*, 2018.

[13] D. Zou, Y. Cao, D. Zhou, and Q. Gu, "Gradient descent optimizes over-parameterized deep relu networks," *Machine Learning*, vol. 109, no. 3, pp. 467–492, 2020.

[14] D. H. Hubel and T. N. Wiesel, "Receptive fields, binocular interaction and functional architecture in the cat's visual cortex," *The Journal of physiology*, vol. 160, no. 1, p. 106, 1962.

[15] S. Kanjlia, C. Lane, L. Feigenson, and M. Bedny, "Absence of visual experience modifies the neural basis of numerical thinking," *Proceedings of the National Academy of Sciences*, vol. 113, no. 40, pp. 11 172–11 177, 2016.

[16] K. Maile, H. Luga, and D. G. Wilson, "Structural learning in artificial neural networks: A neural operator perspective," *Transactions on Machine Learning Research*, 2022.

[17] K. Maile, D. G. Wilson, and P. Forré, "Equivariance-aware architectural optimization of neural networks," *11th International Conference on Learning Representations*, 2023.
[18] K. Maile, E. Lecarpentier, H. Luga, and D. G. Wilson, "DARTS-PRIME: Regularization and scheduling improve constrained optimization in differentiable NAS," *arXiv preprint arXiv:2106.11655*, 2021.

[19] H. Liu, K. Simonyan, and Y. Yang, "DARTS: Differentiable architecture search," *arXiv preprint arXiv:1806.09055*, 2018.

[20] K. Maile, E. Rachelson, H. Luga, and D. G. Wilson, "When, where, and how to add new neurons to anns," in *International Conference on Automated Machine Learning*. PMLR, 2022, pp. 18–1.

[21] K. Maile, H. Luga, and D. G. Wilson, "Neural growth and pruning in dynamic learning environments," *International Conference on Automated Machine Learning*, 2023.

[22] F. Rosenblatt, "The perceptron: a probabilistic model for information storage and organization in the brain." *Psychological review*, vol. 65, no. 6, p. 386, 1958.

[23] I. Goodfellow, Y. Bengio, and A. Courville, *Deep Learning*. MIT Press, 2016, http://www.deeplearningbook.org.

[24] V. Nair and G. E. Hinton, "Rectified linear units improve restricted boltzmann machines," in *ICML*, 2010.

[25] K. Janocha and W. M. Czarnecki, "On loss functions for deep neural networks in classification," *arXiv preprint arXiv:1702.05659*, 2017.

[26] S. Ruder, "An overview of gradient descent optimization algorithms," *arXiv preprint arXiv:1609.04747*, 2016.

[27] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors," *Nature*, vol. 323, no. 6088, pp. 533–536, 1986.

[28] Y. Bengio, D.-H. Lee, J. Bornschein, T. Mesnard, and Z. Lin, "Towards biologically plausible deep learning," *arXiv preprint arXiv:1502.04156*, 2015.

[29] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, 2016, pp. 770–778.

[30] T. M. Hospedales, A. Antoniou, P. Micaelli, and A. J. Storkey, "Meta-learning in neural networks: A survey," *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 2021.

[31] X. He, K. Zhao, and X. Chu, "AutoML: A survey of the state-of-the-art," *Knowledge-Based Systems*, vol. 212, p. 106622, 2021.

[32] N. Cook, "The neuron-level phenomena underlying cognition and consciousness: Synaptic activity and the action potential," *Neuroscience*, vol. 153, no. 3, pp. 556–570, 2008.

[33] R. D. Fields, A. Araque, H. Johansen-Berg, S.-S. Lim, G. Lynch, K.-A. Nave, M. Nedergaard, R. Perez, T. Sejnowski, and H. Wake, "Glial biology in learning and cognition," *The neuroscientist*, vol. 20, no. 5, pp. 426–431, 2014.

[34] H. Kennedy, D. C. Van Essen, and Y. Christen, *Micro-, meso- and macro-connectomics of the brain*. Springer Nature, 2016.

[35] A. M. Fjell and K. B. Walhovd, "Structural brain changes in aging: courses, causes and cognitive consequences," *Reviews in the Neurosciences*, vol. 21, no. 3, pp. 187–222, 2010.

[36] G. Radnikow and D. Feldmeyer, "Layer-and cell type-specific modulation of excitatory neuronal activity in the neocortex," *Frontiers in neuroanatomy*, vol. 12, p. 1, 2018.
[37] G. Z. Tau and B. S. Peterson, "Normal development of brain circuits," *Neuropsychopharmacology*, vol. 35, no. 1, pp. 147-168, 2010.

[38] Y. LeCun, J. S. Denker, and S. A. Solla, "Optimal brain damage," in *Advances in Neural Information Processing Systems*, 1990, pp. 598-605.

[39] C. E. Martin and P. K. Willy, "Probabilistic Program Neurogenesis," in *The 2018 Conference on Artificial Life: A Hybrid of the European Conference on Artificial Life (ECAL) and the International Conference on the Synthesis and Simulation of Living Systems (ALIFE)*. MIT Press, 2019, pp. 440-447.

[40] P. Li, I. Farkas, and B. MacWhinney, "Early lexical development in a self-organizing neural network," *Neural Networks*, vol. 17, no. 8, pp. 1345-1362, 2004, new Developments in Self-Organizing Systems. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0893608004001534

[41] R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, and T. Tuytelaars, "Memory aware synapses: Learning what (not) to forget," in *Proceedings of the European Conference on Computer Vision (ECCV)*, 2018, pp. 139-154.

[42] F. A. Azevedo, L. R. Carvalho, L. T. Grinberg, J. M. Farfel, R. E. Ferretti, R. E. Leite, W. J. Filho, R. Lent, and S. Herculano-Houzel, "Equal numbers of neuronal and nonneuronal cells make the human brain an isometrically scaled-up primate brain," *Journal of Comparative Neurology*, vol. 513, no. 5, pp. 532-541, 2009.

[43] S. Malik, G. Vinukonda, L. R. Vose, D. Diamond, B. B. Bhimavarapu, F. Hu, M. T. Zia, R. Hevner, N. Zecevic, and P. Ballabh, "Neurogenesis continues in the third trimester of pregnancy and is suppressed by premature birth," *Journal of neuroscience*, vol. 33, no. 2, pp. 411-423, 2013.

[44] N. Urbán and F. Guillemot, "Neurogenesis in the embryonic and adult brain: same regulators, different roles," *Frontiers in cellular neuroscience*, vol. 8, p. 396, 2014.

[45] W. Wu, K. Wong, J.-h. Chen, Z.-h. Jiang, S. Dupuis, J. Y. Wu, and Y. Rao, "Directional guidance of neuronal migration in the olfactory system by the protein slit," *Nature*, vol. 400, no. 6742, pp. 331-336, 1999.

[46] S. Ackerman, "The development and shaping of the brain," *Discovering the Brain*, pp. 86-103, 1992.

[47] G.-l. Ming and H. Song, "Adult neurogenesis in the mammalian brain: Significant answers and significant questions," *Neuron*, vol. 70, no. 4, pp. 687-702, 2011.

[48] A. Ernst, K. Alkass, S. Bernard, M. Salehpour, S. Perl, J. Tisdale, G. Possnert, H. Druid, and J. Frisén, "Neurogenesis in the striatum of the adult human brain," *Cell*, vol. 156, no. 5, pp. 1072-1083, 2014.

[49] H. T. Ghashghaei, C. Lai, and E. Anton, "Neuronal migration in the adult brain: are we there yet?" *Nature Reviews Neuroscience*, vol. 8, no. 2, pp. 141-151, 2007.

[50] R. Mozzachiodi and J. H. Byrne, "More than synaptic plasticity: role of nonsynaptic plasticity in learning and memory," *Trends in neurosciences*, vol. 33, no. 1, pp. 17-26, 2010.

[51] C. L. Waites, A. M. Craig, and C. C. Garner, "Mechanisms of vertebrate synaptogenesis," *Annu. Rev. Neurosci.*, vol. 28, pp. 251-274, 2005.

[52] L. Erskine and E. Herrera, "The retinal ganglion cell axon's journey: Insights into molecular mechanisms of axon guidance," *Developmental biology*, vol. 308, no. 1, pp. 1-14, 2007.
[53] C. Pfeiffenberger, T. Cutforth, G. Woods, J. Yamada, R. C. Rentería, D. R. Copenhagen, J. G. Flanagan, and D. A. Feldheim, "Ephrin-As and neural activity are required for eye-specific patterning during retinogeniculate mapping," *Nature neuroscience*, vol. 8, no. 8, p. 1022, 2005.

[54] T. J. Nelson and D. L. Alkon, "Molecular regulation of synaptogenesis during associative learning and memory," *Brain research*, vol. 1621, pp. 239-251, 2015.

[55] W. Kelsch, S. Sim, and C. Lois, "Watching synaptogenesis in the adult brain," *Annual review of neuroscience*, vol. 33, pp. 131-149, 2010.

[56] M. M. Riccomagno and A. L. Kolodkin, "Sculpting neural circuits by axon and dendrite pruning," *Annual review of cell and developmental biology*, vol. 31, pp. 779-805, 2015.

[57] D. O. Hebb, *The organisation of behaviour: a neuropsychological theory*. Science Editions New York, 1949.

[58] L. C. Katz and C. J. Shatz, "Synaptic activity and the construction of cortical circuits," *Science*, vol. 274, no. 5290, pp. 1133-1138, 1996.

[59] D. H. Hubel, T. N. Wiesel, S. LeVay, H. B. Barlow, and R. M. Gaze, "Plasticity of ocular dominance columns in monkey striate cortex," *Philosophical Transactions of the Royal Society of London. B, Biological Sciences*, vol. 278, no. 961, pp. 377-409, 1977.

[60] D. K. Wilton, L. Dissing-Olesen, and B. Stevens, "Neuron-glia signaling in synapse elimination," *Annual review of neuroscience*, vol. 42, pp. 107-127, 2019.

[61] B. Stevens, N. J. Allen, L. E. Vazquez, G. R. Howell, K. S. Christopherson, N. Nouri, K. D. Micheva, A. K. Mehalow, A. D. Huberman, B. Stafford et al., "The classical complement cascade mediates cns synapse elimination," *Cell*, vol. 131, no. 6, pp. 1164-1178, 2007.

[62] D. L. Bishop, T. Misgeld, M. K. Walsh, W.-B. Gan, and J. W. Lichtman, "Axon branch removal at developing synapses by axosome shedding," *Neuron*, vol. 44, no. 4, pp. 651-661, 2004.

[63] A. W. Harrington and D. D. Ginty, "Long-distance retrograde neurotrophic factor signalling in neurons," *Nature Reviews Neuroscience*, vol. 14, no. 3, pp. 177-187, 2013.

[64] P. Vanderhaeghen and H.-J. Cheng, "Guidance molecules in axon pruning and cell death," *Cold Spring Harbor perspectives in biology*, vol. 2, no. 6, p. a001859, 2010.

[65] W. Yeo and J. Gautier, "Early neural cell death: dying to become neurons," *Developmental biology*, vol. 274, no. 2, pp. 233-244, 2004.

[66] R. R. Buss, W. Sun, and R. W. Oppenheim, "Adaptive roles of programmed cell death during nervous system development," *Annu. Rev. Neurosci.*, vol. 29, pp. 1-35, 2006.

[67] M. Fricker, A. M. Tolkovsky, V. Borutaite, M. Coleman, and G. C. Brown, "Neuronal cell death," *Physiological reviews*, vol. 98, no. 2, pp. 813-880, 2018.

[68] X. Dai, H. Yin, and N. K. Jha, "NeST: A neural network synthesis tool based on a grow-and-prune paradigm," *IEEE Transactions on Computers*, vol. 68, no. 10, pp. 1487-1497, 2019.

[69] M. Kang and B. Han, "Operation-aware soft channel pruning using differentiable masks," in *International Conference on Machine Learning*. PMLR, 2020, pp. 5122-5131.

[70] P. T. Quinlan, "Structural change and development in real and artificial neural networks," *Neural Networks*, vol. 11, no. 4, pp. 577-599, 1998.
[71] T. Elsken, J.-H. Metzen, and F. Hutter, "Simple and efficient architecture search for convolutional neural networks," *arXiv preprint arXiv:1711.04528*, 2017.

[72] Y. Ci, C. Lin, M. Sun, B. Chen, H. Zhang, and W. Ouyang, "Evolving search space for neural architecture search," *arXiv preprint arXiv:2011.10904*, 2020.

[73] Z. Lu, K. Deb, E. Goodman, W. Banzhaf, and V. N. Boddeti, "Nsganetv2: Evolutionary multi-objective surrogate-assisted neural architecture search," in *European Conference on Computer Vision*. Springer, 2020, pp. 35–51.

[74] K. O. Stanley and R. Miikkulainen, "Evolving neural networks through augmenting topologies." *Evolutionary computation*, vol. 10, no. 2, pp. 99–127, 2002.

[75] S. Hochreiter and J. Schmidhuber, "Long short-term memory," *Neural computation*, vol. 9, no. 8, pp. 1735–1780, 1997.

[76] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, "Attention is all you need," *Advances in Neural Information Processing Systems*, vol. 30, 2017.

[77] G. W. Lindsay, "Attention in psychology, neuroscience, and machine learning," *Frontiers in computational neuroscience*, p. 29, 2020.

[78] L. Deng, G. Li, S. Han, L. Shi, and Y. Xie, "Model compression and hardware acceleration for neural networks: A comprehensive survey," *Proceedings of the IEEE*, vol. 108, no. 4, pp. 485–532, 2020.

[79] T. Hoefler, D. Alistarh, T. Ben-Nun, N. Dryden, and A. Peste, "Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks," *arXiv preprint arXiv:2102.00554*, 2021.

[80] T. Elsken, J. H. Metzen, and F. Hutter, "Neural architecture search: A survey," *Journal of Machine Learning Research*, vol. 20, pp. 1–21, 2019.

[81] L. Xie, X. Chen, K. Bi, L. Wei, Y. Xu, Z. Chen, L. Wang, A. Xiao, J. Chang, X. Zhang et al., "Weight-sharing neural architecture search: A battle to shrink the optimization gap," *arXiv preprint arXiv:2008.01475*, 2020.

[82] K. O. Stanley, J. Clune, J. Lehman, and R. Miikkulainen, "Designing neural networks through neuroevolution," *Nature Machine Intelligence*, vol. 1, no. 1, pp. 24–35, 2019.

[83] J. F. Miller, "Improbed: Multiple problem-solving brain via evolved developmental programs," *Artificial Life*, vol. 27, no. 3–4, pp. 300–335, 2022.

[84] G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter, "Continual lifelong learning with neural networks: A review," *Neural Networks*, 2019.

[85] Y. Han, G. Huang, S. Song, L. Yang, H. Wang, and Y. Wang, "Dynamic neural networks: A survey," *arXiv preprint arXiv:2102.04906*, 2021.

[86] S. Han, J. Pool, J. Tran, and W. Dally, "Learning both weights and connections for efficient neural network," *Advances in Neural Information Processing Systems*, vol. 28, 2015.

[87] B. R. Bartoldson, A. S. Morcos, A. Barbu, and G. Erlebacher, "The generalization-stability tradeoff in neural network pruning," *arXiv preprint arXiv:1906.03728*, 2019.

[88] L. Liebenwein, C. Baykal, B. Carter, D. Gifford, and D. Rus, "Lost in pruning: The effects of pruning neural networks beyond test accuracy," *Proceedings of Machine Learning and Systems*, vol. 3, pp. 93–138, 2021.
[89] E. Dayan and L. G. Cohen, "Neuroplasticity subserving motor skill learning," *Neuron*, vol. 72, no. 3, pp. 443-454, 2011.

[90] S. Liu, D. C. Mocanu, A. R. R. Matavalam, Y. Pei, and M. Pechenizkiy, "Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware," *Neural Computing and Applications*, vol. 33, pp. 2589-2604, 2021.

[91] J. Peng, B. Tang, H. Jiang, Z. Li, Y. Lei, T. Lin, and H. Li, "Overcoming long-term catastrophic forgetting through adversarial neural pruning and synaptic consolidation," *IEEE Transactions on Neural Networks and Learning Systems*, 2021.

[92] D. C. Mocanu, E. Mocanu, P. Stone, P. H. Nguyen, M. Gibescu, and A. Liotta, "Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science," *Nature communications*, vol. 9, no. 1, p. 2383, 2018.

[93] G. Bellec, D. Kappel, W. Maass, and R. Legenstein, "Deep rewiring: Training very sparse deep networks," *International Conference on Learning Representations*, 2018.

[94] M. Wortsman, A. Farhadi, and M. Rastegari, "Discovering neural wirings," *Advances in Neural Information Processing Systems*, vol. 32, pp. 2684-2694, 2019.

[95] B. Hassibi, D. G. Stork, and G. J. Wolff, "Optimal brain surgeon and general network pruning," in *IEEE international conference on neural networks*. IEEE, 1993, pp. 293-299.

[96] M. Zhu and S. Gupta, "To prune, or not to prune: exploring the efficacy of pruning for model compression," *arXiv preprint arXiv:1710.01878*, 2017.

[97] Y. Guo, A. Yao, and Y. Chen, "Dynamic network surgery for efficient DNNs," *arXiv preprint arXiv:1608.04493*, 2016.

[98] N. Lee, T. Ajanthan, and P. H. S. Torr, "SNIP: Single-shot Network Pruning based on Connection Sensitivity," *arXiv:1810.02340 [cs]*, feb 2019.

[99] W. J. Puma-Villanueva, E. P. dos Santos, and F. J. Von Zuben, "A constructive algorithm to synthesize arbitrarily connected feedforward neural networks," *Neurocomputing*, vol. 75, no. 1, pp. 14-32, jan 2012.

[100] X. Dai, H. Yin, and N. K. Jha, "Incremental Learning Using a Grow-and-Prune Paradigm with Efficient Neural Networks," *IEEE Transactions on Emerging Topics in Computing*, may 2019. [Online]. Available: http://arxiv.org/abs/1905.10952

[101] X. Du, Z. Li, Y. Ma, and Y. Cao, "Efficient network construction through structural plasticity," *IEEE Journal on Emerging and Selected Topics in Circuits and Systems*, vol. 9, no. 3, pp. 453-464, 2019.

[102] D. B. Chklovskii, B. Mel, and K. Svoboda, "Cortical rewiring and information storage," *Nature*, vol. 431, no. 7010, pp. 782-788, 2004.

[103] C. Scholl, M. E. Rule, and M. H. Hennig, "The information theory of developmental pruning: Optimizing global network architectures using local synaptic rules," *PLoS computational biology*, vol. 17, no. 10, p. e1009458, 2021.

[104] C. Laurent, C. Ballas, T. George, N. Ballas, and P. Vincent, "Revisiting loss modelling for unstructured pruning," *arXiv preprint arXiv:2006.12279*, 2020.

[105] P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz, "Pruning convolutional neural networks for resource efficient inference," in *International Conference on Learning Representations*, 2017.
[106] S. Yan, B. Fang, F. Zhang, Y. Zheng, X. Zeng, M. Zhang, and H. Xu, "HM-NAS: Efficient neural architecture search via hierarchical masking," in *Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops*, 2019.

[107] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, "Pruning filters for efficient convnets," *International Conference on Learning Representations*, 2017.

[108] A. G. Rust, R. Adams, S. George, and H. Bolouri, "Activity-based pruning in developmental artificial neural networks," in *Proc. of the European Conf. on Artificial Life (ECAL'97)*, 1997, pp. 224-233.

[109] C. Siegel, J. Daily, and A. Vishnu, "Adaptive neuron apoptosis for accelerating deep learning on large scale systems," in *2016 IEEE International Conference on Big Data (Big Data)*. IEEE, 2016, pp. 753-762.

[110] L. Wu, B. Liu, P. Stone, and Q. Liu, "Firefly neural architecture descent: a general approach for growing neural networks," in *Advances in Neural Information Processing Systems*, 2020.

[111] S. Anwar, K. Hwang, and W. Sung, "Structured pruning of deep convolutional neural networks," *ACM Journal on Emerging Technologies in Computing Systems (JETC)*, vol. 13, no. 3, p. 32, 2017.

[112] A. Gordon, E. Eban, O. Nachum, B. Chen, H. Wu, T.-J. Yang, and E. Choi, "Morphnet: Fast & simple resource-constrained structure learning of deep networks," in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, 2018.

[113] P. L. Narasimha, W. H. Delashmit, M. T. Manry, J. Li, and F. Maldonado, "An integrated growing-pruning method for feedforward network training," *Neurocomputing*, vol. 13, no. 71, pp. 2831-2847, 2008.

[114] M. Ye, C. Gong, L. Nie, D. Zhou, A. Klivans, and Q. Liu, "Good subnetworks provably exist: Pruning via greedy forward selection," in *Proceedings of the 37th International Conference on Machine Learning*, ser. Proceedings of Machine Learning Research, H. D. III and A. Singh, Eds., vol. 119. PMLR, 13-18 Jul 2020, pp. 10820-10830. [Online]. Available: http://proceedings.mlr.press/v119/ye20b.html

[115] W. Pan, H. Dong, and Y. Guo, "DropNeuron: Simplifying the structure of deep neural networks," *arXiv:1606.07326 [cs, stat]*, jul 2016.

[116] S. Chen, L. Lin, Z. Zhang, and M. Gen, "Evolutionary NetArchitecture Search for deep neural networks pruning," in *Proceedings of the 2019 2nd International Conference on Algorithms, Computing and Artificial Intelligence*, 2019, pp. 189-196.

[117] A. Wan, X. Dai, P. Zhang, Z. He, Y. Tian, S. Xie, B. Wu, M. Yu, T. Xu, K. Chen et al., "Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions," in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2020, pp. 12965-12974.

[118] S. Guo, Y. Wang, K. Yuan, and Q. Li, "Differentiable network adaption with elastic search space," *arXiv preprint arXiv:2103.16350*, 2021.

[119] K. A. Laube and A. Zell, "Prune and replace NAS," in *2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA)*. IEEE, 2019, pp. 915-921.

[120] X. Li, Y. Zhou, T. Wu, R. Socher, and C. Xiong, "Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting," in *International Conference on Machine Learning*. PMLR, 2019, pp. 3925-3934.
[121] J. Mei, Y. Li, X. Lian, X. Jin, L. Yang, A. Yuille, and J. Yang, "Atomnas: Fine-grained end-to-end neural architecture search," *arXiv preprint arXiv:1912.09640*, 2019.

[122] K. Bi, L. Xie, X. Chen, L. Wei, and Q. Tian, "GOLD-NAS: Gradual, One-Level, Differentiable," *arXiv preprint arXiv:2007.03331*, 2020.

[123] A. Noy, N. Nayman, T. Ridnik, N. Zamir, S. Doveh, I. Friedman, R. Giryes, and L. Zelnik, "Asap: Architecture search, anneal and prune," in *International Conference on Artificial Intelligence and Statistics*. PMLR, 2020, pp. 493-503.

[124] R. Wang, M. Cheng, X. Chen, X. Tang, and C.-J. Hsieh, "Rethinking architecture selection in differentiable NAS," in *International Conference on Learning Representations*, 2021.

[125] N. Roberts, M. Khodak, T. Dao, L. Li, C. Ré, and A. Talwalkar, "Rethinking neural operations for diverse tasks," *arXiv preprint arXiv:2103.15798*, 2021.

[126] H. Cai, L. Zhu, and S. Han, "ProxylessNAS: Direct neural architecture search on target task and hardware," *arXiv*, pp. 1-13, dec 2018. [Online]. Available: http://arxiv.org/abs/1812.00332

[127] T. Veniat and L. Denoyer, "Learning time/memory-efficient deep architectures with budgeted super networks," in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, 2018, pp. 3492-3500.

[128] P. Guo, C.-Y. Lee, and D. Ulbricht, "Learning to branch for multi-task learning," in *Proceedings of the 37th International Conference on Machine Learning*, ser. Proceedings of Machine Learning Research, H. D. III and A. Singh, Eds., vol. 119. PMLR, 13-18 Jul 2020, pp. 3854-3863. [Online]. Available: http://proceedings.mlr.press/v119/guo20e.html

[129] Q. Yao, J. Xu, W.-W. Tu, and Z. Zhu, "Efficient neural architecture search via proximal iterations," in *Proceedings of the AAAI Conference on Artificial Intelligence*, 2020.

[130] X. Yuan, P. H. P. Savarese, and M. Maire, "Growing efficient deep networks by structured continuous sparsification," in *International Conference on Learning Representations*, 2021.

[131] Z. Liu, M. Sun, T. Zhou, G. Huang, and T. Darrell, "Rethinking the value of network pruning," in *International Conference on Learning Representations*, 2019.

[132] M. S. Abdelfattah, A. Mehrotra, Ł. Dudziak, and N. D. Lane, "Zero-cost proxies for lightweight NAS," in *International Conference on Learning Representations*, 2020.

[133] J. Mellor, J. Turner, A. Storkey, and E. J. Crowley, "Neural architecture search without training," in *International Conference on Machine Learning*. PMLR, 2021, pp. 7588-7598.

[134] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Dropout: a simple way to prevent neural networks from overfitting," *The Journal of Machine Learning Research*, vol. 15, no. 1, pp. 1929-1958, 2014.

[135] R. Liu, J. Gao, J. Zhang, D. Meng, and Z. Lin, "Investigating bi-level optimization for learning and vision from a unified perspective: A survey and beyond," *arXiv preprint arXiv:2101.11517*, 2021.

[136] Y. Yoshihara, M. De Roo, and D. Muller, "Dendritic spine formation and stabilization," *Current opinion in neurobiology*, vol. 19, no. 2, pp. 146-153, 2009.

[137] T. Wei, C. Wang, Y. Rui, and C. W. Chen, "Network morphism," in *International Conference on Machine Learning*. PMLR, 2016, pp. 564-572.

[138] J. Kim, M.-S. Kim, and H. Yoon, "Tweaking deep neural networks," *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 2021.
[139] A. Holtmaat and K. Svoboda, "Experience-dependent structural synaptic plasticity in the mammalian brain," *Nature Reviews Neuroscience*, vol. 10, no. 9, pp. 647-658, 2009.

[140] T. E. Faust, G. Gunner, and D. P. Schafer, "Mechanisms governing activity-dependent synaptic pruning in the developing mammalian cns," *Nature Reviews Neuroscience*, pp. 1-17, 2021.

[141] A. Alunni and L. Bally-Cuif, "A comparative view of regenerative neurogenesis in vertebrates," *Development*, vol. 143, no. 5, pp. 741-753, 2016.

[142] U. Evci, B. van Merriënboer, T. Unterthiner, F. Pedregosa, and M. Vladymyrov, "GradMax: Growing neural networks using gradient information," in *10th International Conference on Learning Representations*, 2022.

[143] K. M. Christian, H. Song, and G.-l. Ming, "Functions and dysfunctions of adult hippocampal neurogenesis," *Annual review of neuroscience*, vol. 37, p. 243, 2014.

[144] Q. Liu, L. Wu, and D. Wang, "Splitting steepest descent for growing neural architectures," *Advances in neural information processing systems*, 2019.

[145] C. Dong, L. Liu, Z. Li, and J. Shang, "Towards adaptive residual network training: A neural-ode perspective," in *International Conference on Machine Learning*. PMLR, 2020, pp. 2616-2626.

[146] W. Wen, F. Yan, Y. Chen, and H. Li, "AutoGrow: Automatic Layer Growing in Deep Convolutional Networks," *Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining*, vol. 20, pp. 833-841, 2020. [Online]. Available: https://doi.org/10.1145/3394486.3403126

[147] T. Ash, "Dynamic Node Creation in Backpropagation Networks," *Connection Science*, vol. 1, no. 4, pp. 365-375, jan 1989. [Online]. Available: https://sci-hub.do/10.1080/09540098908915647

[148] S. E. Fahlman and C. Lebiere, "The cascade-correlation learning architecture," in *Advances in Neural Information Processing Systems*, 1990, pp. 524-532.

[149] M. Frean, "The upstart algorithm: A method for constructing and training feedforward neural networks," *Neural computation*, vol. 2, no. 2, pp. 198-209, 1990.

[150] M. Lehtokangas, "Modelling with constructive backpropagation," *Neural Networks*, vol. 12, no. 4-5, pp. 707-716, 1999.

[151] L. Ma and K. Khorasani, "A new strategy for adaptively constructing multilayer feedforward neural networks," *Neurocomputing*, vol. 51, pp. 361-385, 2003.

[152] M. M. Islam, M. A. Sattar, M. F. Amin, X. Yao, and K. Murase, "A new constructive algorithm for architectural and functional adaptation of artificial neural networks," *IEEE transactions on systems, man, and cybernetics, part B (cybernetics)*, vol. 39, no. 6, pp. 1590-1605, 2009.

[153] C. Cortes, X. Gonzalvo, V. Kuznetsov, M. Mohri, and S. Yang, "Adanet: Adaptive structural learning of artificial neural networks," in *International Conference on Machine Learning*. PMLR, 2017, pp. 874-883.

[154] S. Liang, R. Sun, J. D. Lee, and R. Srikant, "Adding one neuron can eliminate all bad local minima," *Advances in Neural Information Processing Systems*, vol. 31, 2018.

[155] Y. Lu, A. Kumar, S. Zhai, Y. Cheng, T. Javidi, and R. Feris, "Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification," in *Proceedings of the IEEE conference on Computer Vision and Pattern Recognition*, 2017, pp. 5334-5343.
[156] Y. N. Dauphin, R. Pascanu, C. Gulcehre, K. Cho, S. Ganguli, and Y. Bengio, "Identifying and attacking the saddle point problem in high-dimensional non-convex optimization," *Advances in Neural Information Processing Systems*, vol. 27, 2014.

[157] G. G. Turrigiano, "Homeostatic plasticity in neuronal networks: the more things change, the more they stay the same," *Trends in neurosciences*, vol. 22, no. 5, pp. 221-227, 1999.

[158] T. Miconi, A. Rawal, J. Clune, and K. O. Stanley, "Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity," in *International Conference on Learning Representations*, 2019.

[159] J. C. Magee and C. Grienberger, "Synaptic plasticity forms and functions," *Annual review of neuroscience*, vol. 43, pp. 95-117, 2020.

[160] A. D. Milstein, Y. Li, K. C. Bittner, C. Grienberger, I. Soltesz, J. C. Magee, and S. Romani, "Bidirectional synaptic plasticity rapidly modifies hippocampal representations," *eLife*, vol. 10, 2021.

[161] X. Chu, T. Zhou, B. Zhang, and J. Li, "Fair DARTS: Eliminating unfair advantages in differentiable architecture search," in *European Conference on Computer Vision*. Springer, 2020, pp. 465-480.

[162] R. K. Srivastava, K. Greff, and J. Schmidhuber, "Training very deep networks," *Advances in Neural Information Processing Systems*, vol. 28, 2015.

[163] L. J. Herstel and C. J. Wierenga, "Network control through coordinated inhibition," *Current Opinion in Neurobiology*, vol. 67, pp. 34-41, 2021.

[164] E. Marder, "Neuromodulation of neuronal circuits: back to the future," *Neuron*, vol. 76, no. 1, pp. 1-11, 2012.

[165] W. Maass, "Networks of spiking neurons: the third generation of neural network models," *Neural networks*, vol. 10, no. 9, pp. 1659-1671, 1997.

[166] A. Tavanaei, M. Ghodrati, S. R. Kheradpisheh, T. Masquelier, and A. Maida, "Deep learning in spiking neural networks," *Neural networks*, vol. 111, pp. 47-63, 2019.

[167] D. Kappel, S. Habenschuss, R. Legenstein, and W. Maass, "Network plasticity as bayesian inference," *PLoS Computational Biology*, vol. 11, no. 11, p. e1004485, 2015.

[168] S. Tian, L. Qu, L. Wang, K. Hu, N. Li, and W. Xu, "A neural architecture search based framework for liquid state machine design," *Neurocomputing*, vol. 443, pp. 174-182, 2021.

[169] N. K. Kasabov, "Neucube: A spiking neural network architecture for mapping, learning and understanding of spatio-temporal brain data," *Neural Networks*, vol. 52, pp. 62-76, 2014.

[170] E. Olivares, E. J. Izquierdo, and R. D. Beer, "A neuromechanical model of multiple network rhythmic pattern generators for forward locomotion in c. elegans," *Frontiers in Computational Neuroscience*, vol. 15, p. 572339, 2021.

[171] M. Lechner, R. Hasani, A. Amini, T. A. Henzinger, D. Rus, and R. Grosu, "Neural circuit policies enabling auditable autonomy," *Nature Machine Intelligence*, vol. 2, no. 10, pp. 642-652, 2020.

[172] G. F. Miller, P. M. Todd, and S. U. Hegde, "Designing neural networks using genetic algorithms." in *ICGA*, vol. 89, 1989, pp. 379-384.

[173] N. Richards, D. E. Moriarty, and R. Miikkulainen, "Evolving neural networks to play Go," *Applied Intelligence*, vol. 8, no. 1, pp. 85-96, 1998.
[174] J. Secretan, N. Beato, D. B. D'Ambrosio, A. Rodriguez, A. Campbell, J. T. Folsom-Kovarik, and K. O. Stanley, "Picbreeder: A case study in collaborative evolutionary exploration of design space," *Evolutionary computation*, vol. 19, no. 3, pp. 373-403, 2011.

[175] M. Hausknecht, J. Lehman, R. Miikkulainen, and P. Stone, "A neuroevolution approach to general atari game playing," *IEEE Transactions on Computational Intelligence and AI in Games*, vol. 6, no. 4, pp. 355-366, 2014.

[176] F. Gruau, "Neural network synthesis using cellular encoding and the genetic algorithm," *PhD thesis*, Laboratoire de l'Informatique du Parallilisme, Ecole Normale Supirieure de Lyon, 1994.

[177] J. F. Miller and D. G. Wilson, "A developmental artificial neural network model for solving multiple problems," in *Proceedings of the Genetic and Evolutionary Computation Conference Companion*. ACM, 2017, pp. 69-70.

[178] G. S. Hornby and J. B. Pollack, "Creating high-level components with a generative representation for body-brain evolution," *Artificial Life*, vol. 8(3), 2002.

[179] M. Huisman, J. N. Van Rijn, and A. Plaat, "A survey of deep meta-learning," *Artificial Intelligence Review*, vol. 54, no. 6, pp. 4483-4541, 2021.

[180] B. Zoph and Q. V. Le, "Neural architecture search with reinforcement learning," *International Conference on Learning Representations*, 2017.

[181] Y. Chen, X. Song, C. Lee, Z. Wang, Q. Zhang, D. Dohan, K. Kawakami, G. Kochanski, A. Doucet, M. Ranzato et al., "Towards learning universal hyperparameter optimizers with transformers," *arXiv preprint arXiv:2205.13320*, 2022.

[182] C. Finn, P. Abbeel, and S. Levine, "Model-agnostic meta-learning for fast adaptation of deep networks," in *International conference on machine learning*. PMLR, 2017, pp. 1126-1135.

[183] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., "Language models are few-shot learners," *Advances in Neural Information Processing Systems*, vol. 33, 2020.

[184] S. Niu, J. Wu, G. Xu, Y. Zhang, Y. Guo, P. Zhao, P. Wang, and M. Tan, "Adaxpert: Adapting neural architecture for growing data," in *Proceedings of the 38th International Conference on Machine Learning*, ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR, 18-24 Jul 2021, pp. 8184-8194. [Online]. Available: http://proceedings.mlr.press/v139/niu21a.html

[185] G. S. Hornby and J. B. Pollack, "Body-brain co-evolution using L-systems as a generative encoding," in *Proceedings of the 3rd Annual Conference on Genetic and Evolutionary Computation*, 2001, pp. 868-875.

[186] Y. Fu, Z. Yu, Y. Zhang, and Y. Lin, "Auto-agent-distiller: Towards efficient deep reinforcement learning agents via neural architecture search," *arXiv preprint arXiv:2012.13091*, 2020.

[187] Y. Miao, X. Song, D. Peng, S. Yue, E. Brevdo, and A. Faust, "RL-DARTS: Differentiable architecture search for reinforcement learning," *arXiv preprint arXiv:2106.02229*, 2021.

[188] M. Botvinick, J. X. Wang, W. Dabney, K. J. Miller, and Z. Kurth-Nelson, "Deep reinforcement learning and its neuroscientific implications," *Neuron*, 2020.

[189] A. Banerjee, R. V. Rikhye, and A. Marblestone, "Reinforcement-guided learning in frontal neocortex: emerging computational concepts," *Current Opinion in Behavioral Sciences*, vol. 38, pp. 133-140, 2021.
[190] F. Huang and H. Huang, "Biadam: Fast adaptive bilevel optimization methods," *arXiv preprint arXiv:2106.11396*, 2021.

[191] M. M. Bronstein, J. Bruna, T. Cohen, and P. Veličković, "Geometric deep learning: Grids, groups, graphs, geodesics, and gauges," *arXiv preprint arXiv:2104.13478*, 2021.

[192] D. Blalock, J. J. G. Ortiz, J. Frankle, and J. Guttag, "What is the state of neural network pruning?" *arXiv preprint arXiv:2003.03033*, 2020.

[193] Y. Bian, Q. Song, M. Du, J. Yao, H. Chen, and X. Hu, "Sub-architecture ensemble pruning in neural architecture search," *arXiv preprint arXiv:1910.00370*, 2019.

[194] N. Sinha and K.-W. Chen, "Evolving neural architecture using one shot model," in *Proceedings of the Genetic and Evolutionary Computation Conference*, 2021, pp. 910–918.

[195] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, "Backpropagation applied to handwritten ZIP code recognition," *Neural Computation*, vol. 1, no. 4, pp. 541–551, 1989.

[196] T. Cohen and M. Welling, "Group equivariant convolutional networks," in *International Conference on Machine Learning*. PMLR, 2016, pp. 2990–2999.

[197] S. Ravanbakhsh, J. Schneider, and B. Poczos, "Equivariance through parameter-sharing," in *International Conference on Machine Learning*. PMLR, 2017, pp. 2892–2901.

[198] R. Kondor and S. Trivedi, "On the generalization of equivariance and convolution in neural networks to the action of compact groups," in *International Conference on Machine Learning*. PMLR, 2018, pp. 2747–2755.

[199] M. Weiler, P. Forré, E. Verlinde, and M. Welling, "Coordinate independent convolutional networks - isometry and gauge equivariant convolutions on riemannian manifolds," *arXiv preprint arXiv:2106.06020*, 2021.

[200] M. Weiler and G. Cesa, "General E(2)-equivariant steerable CNNs," *Advances in Neural Information Processing Systems*, vol. 32, 2019.

[201] R. Wang, R. Walters, and R. Yu, "Approximately equivariant networks for imperfectly symmetric dynamics," in *International Conference on Machine Learning*. PMLR, 2022.

[202] M. Finzi, G. Benton, and A. G. Wilson, "Residual pathway priors for soft equivariance constraints," *Advances in Neural Information Processing Systems*, vol. 34, pp. 30037–30049, 2021.

[203] A. Zhou, T. Knowles, and C. Finn, "Meta-learning symmetries by reparameterization," in *International Conference on Learning Representations*, 2020.

[204] R. A. Yeh, Y.-T. Hu, M. Hasegawa-Johnson, and A. Schwing, "Equivariance discovery by learned parameter-sharing," in *International Conference on Artificial Intelligence and Statistics*. PMLR, 2022, pp. 1527–1545.

[205] S. Basu, A. Magesh, H. Yadav, and L. R. Varshney, "Autoequivariant network search via group decomposition," *arXiv preprint arXiv:2104.04848*, 2021.

[206] E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, J. Tan, Q. V. Le, and A. Kurakin, "Large-scale evolution of image classifiers," in *International Conference on Machine Learning*. PMLR, 2017, pp. 2902–2911.
[207] Z. Lu, I. Whalen, V. Boddeti, Y. Dhebar, K. Deb, E. Goodman, and W. Banzhaf, "NSGA-Net: neural architecture search using multi-objective genetic algorithm," in *Proceedings of the Genetic and Evolutionary Computation Conference*, 2019, pp. 419-427.

[208] D. W. Romero and S. Lohit, "Learning partial equivariances from data," *Advances in Neural Information Processing Systems*, vol. 35, 2022.

[209] T. F. van der Ouderaa, D. W. Romero, and M. van der Wilk, "Relaxing equivariance constraints with non-stationary continuous filters," *Advances in Neural Information Processing Systems*, vol. 35, 2022.

[210] D. Agrawal and J. Ostrowski, "A classification of $G$-invariant shallow neural networks," *Advances in Neural Information Processing Systems*, vol. 35, 2022.

[211] I. N. Herstein, *Topics in algebra*. John Wiley & Sons, 2006.

[212] A. Falanti, E. Lomurno, S. Samele, D. Ardagna, M. Matteucci et al., "POPNASv2: An efficient multi-objective neural architecture search technique," in *International Joint Conference on Neural Networks*, 2022.

[213] G. Elsayed, P. Ramachandran, J. Shlens, and S. Kornblith, "Revisiting spatial invariance with low-rank local connectivity," in *International Conference on Machine Learning*. PMLR, 2020, pp. 2868-2879.

[214] T. Salimans and D. P. Kingma, "Weight normalization: A simple reparameterization to accelerate training of deep neural networks," *Advances in Neural Information Processing Systems*, vol. 29, 2016.

[215] H. Larochelle, D. Erhan, A. Courville, J. Bergstra, and Y. Bengio, "An empirical evaluation of deep architectures on problems with many factors of variation," in *International Conference on Machine Learning*. PMLR, 2007, pp. 473-480.

[216] H. W. Leung and J. Bovy, "Deep learning of multi-element abundances from high-resolution spectroscopic data," *Monthly Notices of the Royal Astronomical Society*, vol. 483, no. 3, pp. 3255-3277, 2019.

[217] N. C. Codella, D. Gutman, M. E. Celebi, B. Helba, M. A. Marchetti, S. W. Dusza, A. Kalloo, K. Liopyris, N. Mishra, H. Kittler et al., "Skin lesion analysis toward melanoma detection: A challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), hosted by the International Skin Imaging Collaboration (ISIC)," in *International Symposium on Biomedical Imaging*. IEEE, 2018, pp. 168-172.

[218] P. Tschandl, C. Rosendahl, and H. Kittler, "The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions," *Scientific Data*, vol. 5, no. 1, pp. 1-9, 2018.

[219] M. Combalia, N. C. Codella, V. Rotemberg, B. Helba, V. Vilaplana, O. Reiter, C. Carrera, A. Barreiro, A. C. Halpern, S. Puig et al., "BCN20000: Dermoscopic lesions in the wild," *arXiv preprint arXiv:1908.02288*, 2019.

[220] D. Maclaurin, D. Duvenaud, and R. Adams, "Gradient-based hyperparameter optimization through reversible learning," in *International Conference on Machine Learning*. PMLR, 2015, pp. 2113-2122.

[221] J. Lichtarge, C. Alberti, and S. Kumar, "Simple and effective gradient-based tuning of sequence-to-sequence models," *International Conference on Automated Machine Learning*, 2022.
[222] L. Xie, X. Chen, K. Bi, L. Wei, Y. Xu, L. Wang, Z. Chen, A. Xiao, J. Chang, X. Zhang et al., "Weight-sharing neural architecture search: A battle to shrink the optimization gap," *ACM Computing Surveys (CSUR)*, vol. 54, no. 9, pp. 1-37, 2021.

[223] Y. Li, P. Zhao, G. Yuan, X. Lin, Y. Wang, and X. Chen, "Pruning-as-search: Efficient neural architecture search via channel pruning and structural reparameterization," in *International Joint Conference on Artificial Intelligence*, 2022.

[224] C. White, M. Khodak, R. Tu, S. Shah, S. Bubeck, and D. Dey, "A deeper look at zero-cost proxies for lightweight NAS," in *ICLR Blog Track*, 2022. [Online]. Available: https://iclr-blog-track.github.io/2022/03/25/zero-cost-proxies/

[225] E. Real, A. Aggarwal, Y. Huang, and Q. V. Le, "Regularized evolution for image classifier architecture search," in *Proceedings of the AAAI conference on artificial intelligence*, vol. 33, no. 01, 2019, pp. 4780-4789.

[226] Y. Li, Y. Zhou, Y. Wang, and Z. Tang, "PD-DARTS: Progressive discretization differentiable architecture search," in *International Conference on Pattern Recognition and Artificial Intelligence*. Springer, 2020, pp. 306-311.

[227] X. Chen, L. Xie, J. Wu, and Q. Tian, "Progressive differentiable architecture search: Bridging the depth gap between search and evaluation," in *Proceedings of the IEEE/CVF International Conference on Computer Vision*, 2019, pp. 1294-1303.

[228] R. Wang, M. Cheng, X. Chen, X. Tang, and C.-J. Hsieh, "Rethinking architecture selection in differentiable NAS," in *International Conference on Learning Representations*, 2020.

[229] X. Chu, X. Wang, B. Zhang, S. Lu, X. Wei, and J. Yan, "DARTS-: robustly stepping out of performance collapse without indicators," *arXiv preprint arXiv:2009.01027*, 2020.

[230] P. Zhou, C. Xiong, R. Socher, and S. C. Hoi, "Theory-inspired path-regularized differential network architecture search," *arXiv preprint arXiv:2006.16537*, 2020.

[231] T. E. Arber Zela, T. Saikia, Y. Marrakchi, T. Brox, and F. Hutter, "Understanding and robustifying differentiable architecture search," in *International Conference on Learning Representations*, vol. 3, 2020, p. 7.

[232] H. Liang, S. Zhang, J. Sun, X. He, W. Huang, K. Zhuang, and Z. Li, "DARTS+: Improved differentiable architecture search with early stopping," *arXiv preprint arXiv:1909.06035*, 2019.

[233] N. Parikh and S. Boyd, "Proximal algorithms," *Foundations and Trends in optimization*, vol. 1, no. 3, pp. 127-239, 2014.

[234] J. Martens, *Second-order optimization for neural networks*. University of Toronto (Canada), 2016.

[235] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," *arXiv preprint arXiv:1412.6980*, 2014.

[236] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska et al., "Overcoming catastrophic forgetting in neural networks," *Proceedings of the national academy of sciences*, vol. 114, no. 13, pp. 3521-3526, 2017.

[237] M. Tu, V. Berisha, Y. Cao, and J.-s. Seo, "Reducing the model order of deep neural networks using information theory," in *2016 IEEE Computer Society Annual Symposium on VLSI (ISVLSI)*. IEEE, 2016, pp. 93-98.
[238] L. Theis, I. Korshunova, A. Tejani, and F. Huszár, "Faster gaze prediction with dense networks and fisher pruning," *arXiv preprint arXiv:1801.05787*, 2018.

[239] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, "Distributed optimization and statistical learning via the alternating direction method of multipliers," *Foundations and Trends R in Machine Learning*, vol. 3, pp. 1–122, 2010.

[240] F. Kiaee, C. Gagné, and M. Abbasi, "Alternating direction method of multipliers for sparse convolutional neural networks," *arXiv preprint arXiv:1611.01590*, 2016.

[241] T. Zhang, S. Ye, K. Zhang, X. Ma, N. Liu, L. Zhang, J. Tang, K. Ma, X. Lin, M. Fardad et al., "Structadmm: A systematic, high-efficiency framework of structured weight pruning for dnns," *arXiv preprint arXiv:1807.11091*, 2018.

[242] S. Ye, T. Zhang, K. Zhang, J. Li, K. Xu, Y. Yang, F. Yu, J. Tang, M. Fardad, S. Liu et al., "Progressive weight pruning of deep neural networks using admm," *arXiv preprint arXiv:1810.07378*, 2018.

[243] X. Ma, G. Yuan, S. Lin, Z. Li, H. Sun, and Y. Wang, "Resnet can be pruned 60×: Introducing network purification and unused path removal (p-rm) after weight pruning," in *2019 IEEE/ACM International Symposium on Nanoscale Architectures (NANOARCH)*. IEEE, 2019, pp. 1–2.

[244] C. Liu, B. Zoph, M. Neumann, J. Shlens, W. Hua, L.-J. Li, L. Fei-Fei, A. Yuille, J. Huang, and K. Murphy, "Progressive neural architecture search," in *Proceedings of the European conference on computer vision (ECCV)*, 2018, pp. 19–34.

[245] H. Pham, M. Guan, B. Zoph, Q. Le, and J. Dean, "Efficient neural architecture search via parameters sharing," in *International Conference on Machine Learning*. PMLR, 2018, pp. 4095–4104.

[246] X. Dong and Y. Yang, "Searching for a robust neural architecture in four gpu hours," in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2019, pp. 1761–1770.

[247] A. Krizhevsky, V. Nair, and G. Hinton, "Cifar-10 and cifar-100 datasets," *URL: https://www.cs.toronto.edu/kriz/cifar.html*, vol. 6, no. 1, p. 1, 2009.

[248] M. P. Marcus, M. A. Marcinkiewicz, and B. Santorini, "Building a large annotated corpus of english: the penn treebank," *Computational Linguistics*, vol. 19, no. 2, pp. 313–330, 1993.

[249] Y. Bengio, N. Roux, P. Vincent, O. Delalleau, and P. Marcotte, "Convex neural networks," in *Advances in Neural Information Processing Systems*, vol. 18, 2005.

[250] L. Wu, D. Wang, and Q. Liu, "Splitting steepest descent for growing neural architectures," in *Advances in Neural Information Processing Systems*, vol. 32, 2019.

[251] A. M. Saxe, J. L. McClelland, and S. Ganguli, "Exact solutions to the nonlinear dynamics of learning in deep linear neural networks," in *2nd International Conference on Learning Representations*, 2014.

[252] H. Daneshmand, A. Joudaki, and F. Bach, "Batch normalization orthogonalizes representations in deep random networks," in *Advances in Neural Information Processing Systems*, vol. 34, 2021, pp. 4896–4906.

[253] C. Lyle, M. Rowland, and W. Dabney, "Understanding and preventing capacity loss in reinforcement learning," in *10th International Conference on Learning Representations*, 2022.

[254] L. Jing, P. Vincent, Y. LeCun, and Y. Tian, "Understanding dimensional collapse in contrastive self-supervised learning," in *10th International Conference on Learning Representations*, 2022.
[255] T. Chen, I. Goodfellow, and J. Shlens, "Net2Net: Accelerating learning via knowledge transfer," in *4th International Conference on Learning Representations*, 2016.

[256] Y. Kilcher, G. Bécigneul, and T. Hofmann, "Escaping flat areas via function-preserving structural network modifications," *openreview.net*, 2018.

[257] A. Kumar, R. Agarwal, D. Ghosh, and S. Levine, "Implicit under-parameterization inhibits data-efficient deep reinforcement learning," in *9th International Conference on Learning Representations*, 2021.

[258] D. P. Bertsekas, "Nonlinear programming," *Journal of the Operational Research Society*, vol. 48, no. 3, pp. 334-334, 1997.

[259] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors," *Nature*, vol. 323, no. 6088, pp. 533-536, 1986.

[260] L. Deng, "The MNIST database of handwritten digit images for machine learning research [best of the web]," *IEEE signal processing magazine*, vol. 29, no. 6, pp. 141-142, 2012.

[261] K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," in *3rd International Conference on Learning Representations*, 2015.

[262] S. Zagoruyko and N. Komodakis, "Wide residual networks," *arXiv preprint arXiv:1605.07146*, 2016.

[263] D. P. Kingma and J. Ba, "ADAM: A method for stochastic optimization," in *3rd International Conference on Learning Representations*, 2015.

[264] X. Glorot and Y. Bengio, "Understanding the difficulty of training deep feedforward neural networks," in *Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics*, vol. 9. PMLR, 2010, pp. 249-256.

[265] D. Bertoin, J. Bolte, S. Gerchinovitz, and E. Pauwels, "Numerical influence of ReLU(0) on backpropagation," in *Advances in Neural Information Processing Systems*, vol. 34, 2021, pp. 468-479.

[266] A. Lacoste, A. Luccioni, V. Schmidt, and T. Dandres, "Quantifying the carbon emissions of machine learning," *arXiv preprint arXiv:1910.09700*, 2019.

[267] L. Lannelongue, J. Grealey, and M. Inouye, "Green algorithms: Quantifying the carbon footprint of computation," *Advanced Science*, vol. 8, no. 12, p. 2100707, 2021.

[268] M. Hein, M. Andriushchenko, and J. Bitterwolf, "Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem," in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2019, pp. 41-50.

[269] G. Sokar, R. Agarwal, P. S. Castro, and U. Evci, "The dormant neuron phenomenon in deep reinforcement learning," in *International Conference on Machine Learning*. PMLR, 2023.

[270] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, "ImageNet Large Scale Visual Recognition Challenge," *International Journal of Computer Vision (IJCV)*, vol. 115, no. 3, pp. 211-252, 2015.

[271] J. Howard, "Imagenette," https://github.com/fastai/imenette.

[272] Y. Lee, A. S. Chen, F. Tajwar, A. Kumar, H. Yao, P. Liang, and C. Finn, "Surgical fine-tuning improves adaptation to distribution shifts," *11th International Conference on Learning Representations*, 2023.