Venue: AutoML 2022 Track Main

Reviews:

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: n/a
overall_review: n/a
review_summary: n/a
review_title: null

final_score: rating: 4: Marginally above the acceptance threshold (use sparsely)
review_content: summary_of_contributions: In this work, the authors present a framework to ensure the systematic use of neurogenesis. Neurogenesis refers to the process of integrating new neurons in the neural network during the learning process. They decomposed the process into two stages – triggers and initializations. Triggers are like heuristics that determine when, where, and how many neurons to add while initializations decide how to set their initial weights. They present a collection of such triggering and initialization strategies based on the orthogonality of activations or weights. They evaluate their approaches against other neurogenesis strategies on three datasets – a generated toy dataset, MNIST dataset, and CIFAR-10 - using two types of networks – MLP and CNN. Their approach generates models that perform competitively.
overall_review: Positive: 

Generally, neural network models are hand-designed and fixed in size. This work presents a framework that includes a collection of neurogenesis strategies. Therefore, the framework can be used for any task to dynamically grow the network to an efficient size while maintaining “reasonable” performance. 

The paper poses the key questions for neurogenesis - when to add neurons, where to add them in the network, and how to initialize the parameters of new neurons. I liked the way they attempted to answer these questions. To me decomposing the process into two distinct stages - triggers and initializations - is a proper direction to achieve modularity in the framework. This will also allow the users to independently choose the best strategies for both stages. 
 

Negative: 

In my opinion, the novelty of this work is limited. The presented strategies are mostly based on previous work. They mainly unified the neurogenesis strategies under a framework and analyzed their individual components.  

Based on the results presented in Figures 2 and 4, I do not see a clear advantage of the presented strategies compared to the performance of the big static models. The only benefit is the reduced number of parameters. Still, in Figure 2, the CPU Hours vs Total Hidden Neuron plot shows that the CPU Hour for the big model is still lower than the NORTH* strategies. Then what is the benefit of a reduced network size if that takes longer to train? 
review_summary: The work deals with an interesting issue of automatically adding neurons to a network based on utility. They empirically demonstrate that orthogonality-based neurogenesis strategies can be used to design a competitive architecture. However, in my opinion, the novelty and the performance improvement of the work are not significant, rather it is acceptable.
review_title: null

final_score: rating: 4: Marginally above the acceptance threshold (use sparsely)
review_content: summary_of_contributions: The paper addresses a framework for neural archtecture search (NAS), called neurogenesis: A (small) network can be extended on-the-fly (i.e. while learning) if so-called triggers deliver positive (non-zero) values. The triggers are mainly threshold-driven, measuring the orthogonality of e.g. weights. The ideas (including the initialization of the new weights/neurons) are based on the papers by Lyle et al. (2022) and Evci et al. (2022). It is not fully clear how the present work distinguishes from these papers. Experiments show that almost the same performance (of static large networks) can be achieved with considerably smaller size.  
overall_review: PRO:
The framework presented demonstrates on few data sets that this method leads to considerably smaller networks with on a small decay of performance.

CON:
The assessment of the method is based on two data sets only. Therefore I have only little confidence on the claims.
I am missing a comparison with the concept of pruning: Can we get also a similarly small network from a given big one by repeated pruning?

review_summary: The contributions of the paper are in a preliminary state. Therefore I have mixed feelings about a positive recommendation.
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: The concentration of the paper is on "neurogenesis" a subfield of structural learning concentrating on improving performance by dynamically growing the neural network via adding neurons to the architecture. 
The paper's contributions to the case of MLPs are as follows: 
   •	Proposing a framework for decomposing the neurogenesis problem into two steps;
        o	Layer-wised triggers; to identify when, where, and how to add neurons, and
        o	Layer-wised initialization; to initialize the newly added neurons' weight.
   •	Implementing triggers and initializations methods based on orthogonality of post-activation, and weights. Also, a gradient-based trigger is presented for the existing gradient-based initialization methods in the literature. 
   •	Then, the orthogonality methods are compared with the existing gradient-based methods. 
The paper aims to grow the neural architecture during training without having a predefined final size, and the objective function is to choose the neural network with the minimum loss function value.
overall_review: The paper certainly covers an interesting and potentially high-impact topic, and it is well written and easy to follow. The results should be reproducible due to the availability of code, data, and documentation of the code. However, it seems like this is a preliminary paper with a new idea but not fully developed yet, and it can be improved and investigated. With reference to the title, it does not seem to me that the authors answered the mentioned question. The evaluation should be more thorough since there is no comparison of the method to the state-of-art methods in the literature that people cite often. So, it is difficult to determine how well it works relative to those methods. Also, it would be more valuable if the authors mentioned how much more expensive their approach is compared to others since this is a major concern in NAS papers. If the comparison between the NORTH* methods and the gradient-based method exist in the CIFAR10 result, we could conclude that this works for the deeper convolutional neural networks as well. Also, the conclusion can be stronger if the mentioned points are addressed since, currently, it is not strong enough compared to the goal set out earlier in the paper. 
review_summary: Overall, the paper needs some improvements in terms of clarity, evaluation, and conclusion as mentioned in the above sections.
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: >>>
Motivation:

The focus of this paper is on neuro-genesis -- the study of automatically expanding neural networks for hosting more knowledge during training. 

>>>
Background:

This paper can be seen as an extension of  Lyle et al.'s 2021 work on understanding network capacity. In their paper, Lyle et al. proposed a metric based on singular value decomposition (SVD) to quantify the efficient usage of network capacity during reinforcement learning (RL). This helped Lyle et al. to open the black box and understand how the RL agent was interacting with the environment.

In this neuro-genesis paper, the authors showed that the SVD could be used in a different way. They showed that the SVD metric could serve as an indicator of the capacity of learning; and when all eigenvalues are large, this is a signal that more neurons should be added into the network in order to host more knowledge.
overall_review: >>>
Pros (+):

1) Nicely written, well motivated, and good discussion.
2) Adapted an existing technique from RL and gave it a fresh interpretation for neuro-genesis.
3) An lot of experiments covering most of the important aspects.
4) Demonstrated the flexibility of the network.

>>>
Cons (-):

1) A bit lower on novelty (the RL technique was adapted from Lyle et al.)
2) The experiements were slightly simpler and it would be good to see some results of NORTH applied to an architecture including residual connections.
review_summary: >>>
Final Review:

This paper is nicely written with many nice experimental comparisons and results. The novelty is slightly low because the authors took the existing technique from Lyle et al. However, they gave the existing method a fresh interpretation and demonstrated its usefulness beyond the original purpose. 

The experiements could be conducted on more difficult datasets -- for instance on CUB200 with ResNeXt50. However, most of the important aspects are covered.
review_title: null

final_score: rating: 2: Reject, not good enough
review_content: summary_of_contributions: The presented paper is situated in the NAS literature, particularly in dynamically learned architectures; i.e. the network is grown during the training phase alongside the optimization of weights. The authors try to answer the question of where, when and how to add neurons to layers to increase the capacity of the network midst training. They do this by introducing layer-wise triggers and initialization strategies. Crucially, they use three sources of information for their triggers; neural activations, weights and gradients, which help identify where and how many neurons should be added. Essentially, their heuristics all build on the orthogonality measure of Lyle et al. 22's layerwise heuristic, i.e. counting the nonsingular values of some weight-related matrix, where some threshold decides whether the singular value is counted or not. The first two approaches, namely activation-based and weight-based, both define the number of added neurons in reference to the ANN at its initialization (which, from my understanding, supposedly is a running baseline over the training process). The gradient-based flavor does not require this heuristic baseline but instead uses the in- & output weights respective loss gradient norms.

Regarding how to add the new neurons, the authors decided to add them in a function-preserving manner. The subsequent gradient steps inform the new neurons' contribution. Again, the authors present three initialization techniques: activation-, weight- and gradient-based. The activation-based entails two flavors, both of which try to find initialization for the neurons towards the post-activations, where no analytical solution exists. The weight-based version initializes via the orthogonal kernel of the current layers' weights. Lastly, the gradient-based approach simply re-implements existing work.

overall_review: General topic:
The overall idea of neurogenesis is timely and a highly relevant area of research to arrive at smaller and or sparser models that require less time to train and are sufficiently capable. From a theoretical point of view, growing architectures on their own without the ability of a contraction is troubling by the authors' own argument - since the addition of neurons might be dependent on the location in the loss surface. In particular, this could imply that later evolutions might find vast amounts of redundancy, which could suggest that smaller models are favorable again. The nonlinear dependencies between layers may aggravate this fact.

The theory's equations can easily be compressed into 4 "major" equations (2, 3, 6, 7), where eq. 2 & 6 are insufficiently motivated adaptations of Lyle et al.'s orthogonality metric. Regarding novelty, apart from the subsequent hyperparameter analysis, these are their core theoretical contributions, which they should emphasize and explain more intricately.

Regarding the hyperparameters: 
The sensitivity to the selection thresholds & hyperparameters epsilon and gamma are apparent based on the equations already - and not surprisingly, the authors suggest optimizing them (explicitly gamma) for each new task. Depending on the invested computation, this might partially defeat the purpose of a more lightweight training procedure. On their own, the hyperparameter's sensitivity suggests that this method is primarily a heuristic - just as the models they compare against. However, the dynamic perspective inherent in their evidence-driven heuristic significantly improves over static methods conceptually.

Initialization: 
Finding initializations that are not function-altering in the context of nonlinearities is a tedious task except for zeroing their outputs. The pre-activation procedure is interesting but should be motivated better.

The figures will definitely require some clarification. 

review_summary: Overall, the research area in which this paper is situated is an interesting and promising one. 
The paper's novelty is relatively limited and requires more elaborate motivation or even experiments supporting their intuition. Summary statistics on the layer's orthogonality metric over the course of optimization would be very much appreciated to support their argument.
The experimental results exhibit apparent issues and the proposed heuristics seem to foster an almost immediate increase in the number of neurons which is constant thereafter. This is only weakly aligned with the method's claim of dynamicity.
review_title: null

Meta-review: metareview: There is a lot of enthusiasm for this paper, but also one dissenting voice. The discussion was productive and the paper improved as a result. Reviewer 4ixA further suggests an experiment that would validate the results, and the authors should consider it. However, even without it, the paper makes a worthwhile contribution to the conference.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept