Venue: AutoML 2023 Conference

Reviews:

final_score: rating: 5: Borderline Leaning Reject: Technically sound paper where reasons to reject nonetheless outweigh reasons to accept. Please use sparingly.
review_content: summary_of_contributions: This work investigates how the architecture of a neural network impacts three metrics: The “expressivity” meaning how well the next work can approximate the training data, the “convergence” meaning how fast the network learns, and the “generalization” meaning how well the network performs on unseen data.  Given a fixed budget of parameters, a “no free lunch” behavior can be observed, meaning that no model from a NAS space can achieve optimal performance in all three metrics, given a fixed amount of parameters. The authors derive the claims from a theoretical point of view and perform experimental evaluations on Nas-bench-102 to verify the claims. 

overall_review: Positives:

- The results are interesting from a deep learning theory perspective. 
- The formal results section as well as the experiments are extensive.
- The three metrics are an interesting way to characterize the performance of a neural network.

Negatives:

- The authors claim in the abstract that the default measure of performance in AutoML (the test accuracy) is closely related to the three aspects (expressivity/convergence/generalization). However, this is only explained for the aspect of ‘generalization’ in Section 4.4, but not for the concepts of expressivity and convergence. As far as I can see, this is a major flaw, as it limits the conclusion that insights can be derived for the AutoML community. 
- The “Architecture bias in AutoML” section (lines 192-196) is shallow, given that here the “no free lunch” behavior is put in context to applications in AutoML. A more detailed explanation is necessary. 
review_summary: Overall, the paper presents interesting results and supports them with theoretical derivations and particle experiments. My main concern is that the impact on the field of AutoML is limited. 

actions_required_to_increase_overall_recommendation: More explanations on the impact in the field of AutoML.
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: The paper main contributions are as follows:
• Theoretical analysis of the dependence of a deep network’s manifold complexity, convergence rate, and generalization gap on its graph topology.
• Discovery of the “no free lunch” behaviour: given a fixed budget on the number of parameters pre-defined in an architecture space, no such a network can achieve optimal expressivity, convergence and generalization at the same time. 
• Analysis explaining a wide range of observations in AutoML and NAS practices. Experiments on popular datasets confirm the theoretical analysis. 
overall_review: 
Pros
1. sheds light on the limitations and challenges  of optimizing neural networks in an architecture space
2.  The paper is well-written and clearly explains complex technical concepts in a way that is accessible to a broad audience.
3. The paper demonstrates a comprehensive understanding of the technical literature in the field, and builds on it in a novel and innovative way
3. 

Cons
1. To generalise the results obtained from this study, different network and datasets should be included.
2. different networks may have different optimal learning rate. Therefore, it may be deemed unfair to compare them under identical training protocols



review_summary: 
The paper facilitates the explanation of the architecture bias in AutoML and NAS applications through jointly analysing a network’s expressivity, trainability, and generalization, and how they are influenced by the architecture’s graph topology. Given a fixed budget of the number of parameters, it has been shown that the expressivity favors networks of deep and narrow graph topologies, whereas both
the trainability and generalization prefer wide and shallow ones. The paper is the first to discover that these inductive biases lead to a “no free lunch” behavior in deep network architectures. The paper is well-written and easy to follow. 

However To generliaze the results obtained from this study, different network should be studied. Additionally, different networks may have different optimal learning rate. Therefore, it may be deemed unfair to compare them under identical training protocols. 

actions_required_to_increase_overall_recommendation: To generliaze the results obtained from this study, different network should be studied. Additionally, different networks may have different optimal learning rate. Therefore, it may be deemed unfair to compare them under identical training protocols.
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: The paper formalizes expressivity, convergence rate and generalization gap of a neural network based on its graph topology, and discover that given such representation and a fixed parameter budget, it is not possible to achieve optimal expressivity, convergence and generalization. This is because given the graph formulation of architectures, where Ph stands for number of end-to-end unique paths, and dp being the number of linear transformation operations on the p-th path, expressivity requies smaller Ph and larger dp, whereas convergence rate and generalization require larger Ph and lower dp. This is further empirically justified in a fixed setting in Figure 3 and Figure 4.
overall_review: This is a well written paper, with limited but effective empirical testing. Their formalization of expressivity, generalization and convergence is well justified and supported with conclusions verified by Figure 3 and Figure 4. The limitations are discussed in the conclusion, which would motivate future work. 

I would like the authors views on including discussions regarding the implications of the structure of the linear transformations e.g. instead of an MLP, would convolutions/transformer based architectures and other aspects of the operations themselves (stride/pooling etc.) change the conclusions in the paper?

Discussion on whether their toy search space is sufficient to support their conclusions would also help.


review_summary: I am somewhat confident that this paper should be accepted, it is well written with proofs and empirical evidence on a small search space of 729 neural networks to justify their observations derived from their formalization of expressivity, generalization and convergence. It is possible that I did not understand some parts of the submission (generalizability theorems) due to lack of familiarity with prior work.
actions_required_to_increase_overall_recommendation: Comments on whether their toy search space is sufficiently large/diverse for verifying their claims would help.
review_title: null

final_score: rating: 4: Weak Reject: For instance, a paper with minor technical flaws, limited impact, and/or weak evaluation.
review_content: summary_of_contributions: This paper discusses the factors that define the "goodness" of neural architectures. It demonstrates that there there is no single architecture that best optimizes all 3 key properties, namely $\textbf{expressivity}$ (the complexity of functions a network can approximate), $\textbf{convergence}$ (training speed under gradient descent), and $\textbf{generalization}$ (performance on unseen data).
overall_review: The paper gives a clear introduction to the topic and provides a clear structure of all findings. The experiments are limited in the sense that the width and depth of a neural architecture might be misinterpreted and the the relatively small operation set might influence the expressiveness of the architecture space. Also, some architectural topologies have been evaluated more intensively than others and there might be an unequal advantage from the pre-defined hyperparameter setting. The formal results offer valuable insights; however, it would be beneficial to present them as concisely as possible, emphasizing the essential conclusions only.
review_summary: Considering the technical inconsistencies and the limited impact, I am inclined to reject the paper in its present form. Nonetheless, if the identified shortcomings are adequately addressed and resolved, the work has the potential to provide valuable insights for the community.
actions_required_to_increase_overall_recommendation: (already addressed in depth above)
review_title: null

Decision: Accept(Main Track)