Principled Architecture-aware Scaling of Hyperparameters
Authors: Wuyang Chen
Junru Wu
Zhangyang Wang
Boris Hanin
Venue: ICLR 2024 poster (2024)
Keywords: Hyperparameter Transfer
Neural Network Architecture
Neural Network Initialization
Learning Rate
Primary area: general machine learning (i.e., none of the above)
TL;DR: We propose a principled and architecture-aware scaling rule for learning rate and initializations across a wide range of deep networks.

Abstract: Training a high-quality deep neural network requires choosing suitable hyperparameters, which is a non-trivial and expensive process. Current works try to automatically optimize or design principles of hyperparameters, such that they can generalize to diverse unseen scenarios. However, most designs of principles or optimization methods are agnostic to the choice of network structures, and thus largely ignore the impact of neural architectures on hyperparameters. In this work, we precisely characterize the dependence of initializations and maximal learning rates on the network architecture, which includes the network depth, width, convolutional kernel size, and connectivity patterns. By pursuing every parameter to be maximally updated with the same mean squared change in pre-activations, we can generalize our initialization and learning rates across MLPs (multi-layer perception) and CNNs (convolutional neural network) with sophisticated graph topologies. We verify our principles with comprehensive experiments. More importantly, our strategy further sheds light on advancing current benchmarks for architecture design. A fair comparison of AutoML algorithms requires accurate network rankings. However, we demonstrate that network rankings can be easily changed by better training networks in benchmarks with our architecture-aware learning rates and initialization.

Published as a conference paper at ICLR 2024
PRINCIPLED ARCHITECTURE-AWARE SCALING OF HY- PERPARAMETERS
Wuyang Chen' Junru W Zhangyang Wang's Boris Hanin
University of Texas, Austin Google Research (wwyang chen, at laswang) @utexas. Pjunruggoogle.cm com "bhaninprinnceton edu
3Princeton University
ABSTRACT
Training a high-quality deep neural network requires choosing suitable hyper- parameters, which is non-trivial and expensive process. Current works try to automatically optimize or design principles of hyperparameters, such that they can generalize to diverse unseen scenarios. However, most designs op- timization methods are agnostic to the choice of network structures, and thus largely ignore the impact of neural architectures work, we precisely characterize the dependence of initializations and maximal learning rates on the network architecture, which includes the network depth, width, convolutional kernel size, and connectivity patterns. By pursuing every parameter be maximally updated with the same mean squared change in pre- activations, we can generalize our initialization and learning rates across MLPs (multi-layer perception) and CNNs (convolutional neural network) with sophis- ticated graph topologies. We verify our principles with comprehensive experi- ments. More importantly, our strategy further sheds light on advancing current benchmarks for architecture design. A fair comparison of AutoMI. algorithms requires accurate network rankings. However, we demonstrate that network rank- ings can be easily changed by better training networks in benchmarks with our architecture-awar learning rates and initialization. Our code is available at hptss wwt.:huw...cc/////Aaa.a //cehhaaaccccm///TTccooooooooood
hyperparameters. In this
scaling ir init
INTRODUCTION
An important theme in the success of modern deep learning is the development of novel neural network architectures: ConvNets (Simonyan & Zisserman, 2014), ResNets (He al. 2016). Trans- formers (Dosovitskiy et al. 2020) to name few. Whether given architecture works well in practice, however, depends crucially on having reliable principles for selecting training hyperparameters. Indeed, even using more or less standard architectures at scale often requires considerable hyper- parameter tuning. Key examples of such hyperparameters are the learning rate and the network initialization, which play a crucial role in determining the quality of trained models. How to choose them in practice depends non- -riviiiially on the interplay between data, architecture, and optimizer.
Principles for matching deep networks to initialization and optimization schemes have been devel- oped in several vibrant lines work. This includes approaches - typically based on analysis in the infinite width limit developing initialization schemes suited to specific architectures, including fully connected architectures (Jacot et al.) 2018 Perrone et al., 2018). ConvNets (llievski & Feng 2016. Stoll et al.! 2020). ResNets (Horváth al. 2021; Li al. 2022b a). and Transformers (Zhu et al. 2021 Dinan et al. 2023). It also includes several approaches to zero-shot hyperparameter transfer, which seek to establish the functional relationships between hyperparameter (such as good lcarning rate) and the architecture characteristics, such as depth and width. This relation then allows for hyperparameter tuning on small, inexpensive models, to be transferred reliably to larger
architectures YYang et al. 2022 Iyer et al. 2022 Yaida 2022).
Architectures of deep networks, with complicated connectivity patterns and heterogeneous opera- tions, are common in practice and especially important in neural architectures search (Liu et al. 2018
Work done while Texas A&M University

---

Published as a conference paper at ICLR 2024
Dong & Yang 2020). This is important because long- rang-range connections and heterogeneous layer types are common in advanced networks (He et al. [016] Huang et al. 2017 Xie et al. 2019). How- ever, very little prior work develops initialization and learning rate schemes adapted complicated network architectures. Further, while automated methods for hyperparameter optimization (Bergstra et al.] 2013) are largely model agnostic, they cannot discover generalizable principles. As partial remedy, recent works (Zela et al., 2018; Klein & Hutter, 2019; Dong et al. 2020b) jointly design networks and hyperparameters, but still ignore the dependence of learning rates on the initialization scheme. Moreover, ad-hoc hyperparameters that are not tuned in an intelligent way will further jeopardize the design of novel architectures. Without a fair comparison and evaluation of different networks, benchmarks of network architectures may be misleading as they could be biased toward architectures on which "standard" initializations and learning rates we use happen to work out well.
Xie
This article seeks to develop principles for both initialization schemes and zero-shot learning rate selection for general neural network architectures, specified by a directed acyelic graph (DAG). These architectures can be highly irregular (Figure significantly complicating the development of simple intuitions for training hyperparameters based on experience with regular models (e.g. feedforward networks). At the beginning of gradient descent training, first derive simple initialization scheme that preserves the variance of pre-activations during the forward propagation of irregular architectures, and then derive architecture-aware learning rates by following the maximal update (μP) prescription from (Yang et al., 2022). We generalize our initialization and learning rate principles across MLPs (multi-layer perception) and CNNs (convolutional neural network) with arbitrary graph-based connectivity patterns. We experimentally verify our principles on a wide range of architecture configurations. More importantly, our strategy further sheds light on advancing current benchmarks for architecture design. We test our principles on benchmarks for neural architecture search, and we can immediately see the difference that our architecture- initializations and architecture-awan learning rates could make. Accurate network rankings are required such that different AutoML algorithms can be fairly compared by ordering their searched architectures. However, demonstrate that network rankings can be easily changed by better training networks in benchmarks with our architecture-ave learning rates and initialization. Our main contributions are:
We derive a simple modified fan-in initialization scheme that is architecture-aware and can provably preserve the flow of information through any architecture's 's graph topology (S ($1.2).
Using this modified fan-in initialization scheme, we analytically compute the dependence on the graph topology for how to scale learning rates in order to achieve the maximal update (μP) heuristic (Yang et al., [022), which asks that the first step of optimization neuron pre-activations change as much as possible without diverging at large width. For example, we find that ($ (.3) for a ReL.U network of a graph topology trained with MSE loss under gradient descent, the pP 1/2
learning rate scales (ΣP-
where P is the total number of end-to-end paths from the
input to the output, Ipp is the depth of each path, = 1,.,..,, P.
In experiments, we not only verify the superior performance of our prescriptions, but also further re-evaluate the quality of standard architecture benchmarks. By unleashing the potential of architectures (higher accuracy than the benchmark), we achieve largely different rankings of networks, which may lead to different evaluations of AutoML algorithms for neural architecture search (NAS).
2
RELATED WORKS
2.1 HYPERPARAMETER OPTIMIZATION
The problem of hyperparameter optimization (HPO) has become increasingly relevant as machine learning models have become more ubiquitous (Li et al., 2020; Chen et al.] [022 Yang et al. 2022). Poorly chosen hyperparameters can result in suboptimal performance and training instability. Beyond the basic grid or random search, many works tried to speed up HPO. (Snoek et al. 2012) optimize HPO via Bayesian optimization by jointly formulating the performance of each hyperparameter configuration a Gaussian process (GP), and further improved the time cost of HPO by replacing the GP with a neural network. Mcanwhile, other efforts tried to leverage large-scale parallel infrastructure
to speed up hyperparameter tuning (Jamieson & Talwalkar 2016: Li et al. 2020).
2

---

Published as a conference paper at ICLR 2024
Our approach is different from the above since we do not directly work hyperparameter opti- mization. Instead, we analyze the influence of network depth and architecture on the choice of hyperparameters. Mcanwhile, our method is complementary to any HPO method. Given multiple models, typical methods optimize hyperparameters for each model separately. In contrast, with our method, one only needs apply HPO once (to find the optimal learning rate for small network), and then directly scale up to larger models with our principle ($ 4
2.2 HYPERPARAMETER TRANSFER
Recent works also tried to propose principles of hyperparameter scaling and targeted on better stability for training deep neural networks (Glorot & Bengio 2010, Schoennolzz et al. 2016;; Yang &
Schoenholz, 2017 Zhang et al. 2019 Bachlechner et al. 202 Huang et al. al.|| |2020]20000 Liu et al.) 2020).
Some even ex explored transfer learning of hyperparameters (Yogatama & Mann, 2aann|200141 20014 Perrone et al.
2018, Stoll et al., 2020; Horváth et al. (022). (Smith et al. [017] Hoffer et al. 2017) proposed to
scale the learning rate with batch size while fixing the total epochs of training. (Shallue al. 2018) demonstrated the failure of finding a universal scaling law of learning rate and batch size across a range of datasets and models. Assuming that the optimal learning rate should scale with batch size, (Park et al. 2019) empirically studied how the optimal ratio of learning rate over batch size scales for MLP and CNNs trained with SGD.
More recently, Yang Hu 2020) found that standard parameterization (SP) and NTK parameteriza- tion (NTP) lead to bad infinite- infinite-width width limits and hence are suboptimal for wide neural networks. YYngg et al. 2022) proposed yuP initialization, which enabled the tuning of hyperparameters indirectly on a smaller model, and zero-shot transfer them the full-sized model without any further tuning. In ([yer et al. 2022). an empirical observation was found that, for fully-connected deep ReLU networks, the maximal initial learning rate follows a power law of the product of width and depth. JJlassi et al. (2023) recently studied the depth dependence of μP learming rates vanilla ReLU-based MLP networks. Two core differences between μP and our method: 1) pP mainly gives tedth-deeendent learning rates in the first and last layer, and is width-indenn width-indeepndent in other layers (see Table 3 in (Yang et al. 2022)). However, there is a non-trivial dependence on both network depth and topologies: 2) Architectures studied in μP mainly employ sequential structures without complicated graph topologies.
2.3 BECHMARKING NEURAL ARCHITECTURE SEARCH
Neural architecture search (NAS), one research area under the broad topic of automated machine learning (AutoML), targets the automated design of network architecture without incurring too much human inductive bias. It is proven to be principled in optimizing architectures with superior accuracy and balanced computation budgets (Zoph & Le 2016 Tan et al. 2019 Tan & Le 2019). To fairly and efficiently compare different NAS methods, many benchmarks have been developed (Ying et al.. 2019; Dong & Yang 2020 Dong al.| 2020aa. Meanwhile, many works explained the difficulty of evaluating NAS. For example, (Yang et al. 2019) emphasized the importance of tricks in the evaluation protocol and the inherently narrow accuracy range from the search space. However, rare works point out one obvious but largely ignored defect in each of these benchmarks and evaluations: diverse networks are blindly and equally trained under the same training protocol. It is questionable if the diverse architectures in these benchmarks would share the same set of hyperparameters. As result, to create a credible benchmark, it's necessary to find optimal hyperparameters for each network accurately and efficiently, which our method aims to solve. This also implied, NAS benchmarks that were previously regarded as "gold standard", could be no longer reliable and pose questions on existing NAS algorithms that evaluated upon it.
3 METHODS
For general network architectures of directed acyclic computational graphs (DAGs), we propose topology- aware initialization and learning rate scaling. Our core advantage lies in that we precisely characterize the non-trivial dependence of learning rate scales on both networks' depth, DAG topologies and CNN kernel sizes, beyond previous heuristics, with practical implications ($ 3.5
1. In a DAG architecture ($ 3.I. an initialization scheme that depends on a layer's in-degree is required to normalize inputs that flow into this layer 3.2
3

---

Published as a conference paper at ICLR 2024
2. Analyzing the initial step of SGD allows us to detail how changes in pre-activations depend on the network's ' depth and learning rate [SB
3. With our in-degree initialization, by ensuring the magnitude of pre-activation changes remains (1),, we deduce how the learning rate depends on the network topology ($ 3.3 and the CNN kernel size ($ 3.4p
DAG space = (MLP, ResNet, :"
W(o3)
3.1 DEFINITION OF DAG NETWORKS
Hend to endeedddttt paths
W(02)
Wish+1) puRi
"Widet IPI:
W(.,1)
W
We first define the graph topology of compli- cated neural architectures. This formulation is inspired by network structures designed for practical applications (Xie et al. 2019)) You al. 2020: Dong & Yang 2020).
2
200
2
(Input)
W
(Output)
W
"Depth" (dy): #"LinearrRelU"" on path
By definition, a directed acyclic graph is a directed graph 9 (V, E) in which the edge
W: operator (layer)
MIP
Linear RELU
set has no directed cycles (a sequence of edges that starts and ends in the same vertex). We will denote by 4 2 == [V the number of vertices (the input x and pre-activations 2) in V and write:
Silp connection lidentity)
Idisabiedd edge,
ResNet
forward/baccwwerl
Figure 1: neural network's architecture can be rep- resented by a direct acyclic graph (DAG). x is the in- put, =(1), 20)4) ,.... ....,," ..() are pre-activations (vertices), and g(++1 is the output. W is the layer operation (edge). Our DAG space includes architectures of different connections and layer types ("Linear ReLU", "Skip-connect"" and "Zero"). For example, by disabling some edges (gray) or replacing with identity (skip connection, dashed arrow), DAG can represent practical networks such as MLP and ResNet (He et al. 2016).
V [0, T. 1] == (0. 1......., 7. 1}.
We will adopt the convention that edges ter- minating in vertex é originate only in ver- tices f < &. We define DAG neural net- work (with computational graph G
E))
the number of vertices VV = L + 2, hid- den layer width n, output dimension 1, and
ReLU activations) to be function & E R + 2((+1) (x)
E R given by
f= 1,...,, L+1
Eee.EE OEE W i (o() (x)),
(I)
a() = ReLUU() = 10, W((,)) are matrices of weights initialized as follows:
o [N(O Cl /n),
f=
(2)
(=L+1
where C(t',)) ,9) is scaling variable that we will determine based on the network topology (equation We choose 1/n2 as the variance for the output layer mainly for aligning with jPP' desiderata.
3.2 TOPOLOGY-AWARE INTIIALIZTION SCHEME
common way to combine features from multiple layers to directly sum them up, often without normalization (He et al.) [016]] Dong & Yang 2020) as we do in equation [ This summation implies that the weight magnitude
each pre-activation layer should scale proportionally regarding the number of inputs (in-degree). However, most commonly used weight initialization schemes, such as He initialization (He et al. 2015) and Xavier initialization (Glorot Bengio 2010). only consider forward or backward propagation between consecutive layers, ignoring any high-level structures, i.e. how layers are connected with each other (see also (Defazio & Bottou 2022)). This strategy can be problematic in the presence of a bottleneck layer that has large number of inputs.
Motivated by this issue, we propose an architecture-awant initialization strategy. We target deriving the normalizing factor CTee' .F) in the initialization (equation 2 adapted to potentially very different in-degrees of the target layer. Like many theoretical approaches to neural network initialization, we do this by seeking to preserve the "flow of information" through the network at initialization. While this dictum can take various guises, we interpret it by asking that the expected norm of changes in pre-activations between consecutive layer have the same typical scale:
E [0]
B
for all e,
(3)
4

---

Published as a conference paper at ICLR 2024
where E [-] denotes the average over initialization. This condition essentially corresponds to asking for equal scales in any hidden layer representations. Our first result, which is closely related to the ideas developed for MLPs (Theorem 2 in (Defazio & Bottou 2022),, gives simple criterion for ensuring that equation holds for a DAG-structured neural network with ReLU layers:
Initialization scaling for DAG. For neural network with directed acyclic computational graph G (V, E), V = [0, L+1], width n, and ReLU activations, the information flow condition equation is satisfied on average over a centred random input if
C
2 d (P)
(4)
where for cach vertex é [0, ++ 1], we'v ve written d for its in- in G. This implies that, when we initialize a layer, we should consider how many features flow into We include our derivations in Appendix A
3.3 TOPOLOGY-AWARE LEARNING RATES
The purpose of this section is to describe specific method for choosing learning rates for DAG networks. Our approach follows the maximal update (μP) heuristic proposed in (Yang et al. 2022). Specifically, given DAG network recall from equation that
( (z) 21 (z),
denotes the pre-activations at layer corresponding to a network input æ. The key idea Yang et al. 2022 is to consider the change
Az, (e) of 2!{ from one gradient descent step training loss C:
A 1
Σ P
Ap
(5)
HSE
where y denotes network weight, /4 < l means that is a weight in ((',0) for some l' < - 1, Ap for the change in Jr after one step of GD, 17 the learning rate. By symmetry, all weights in layer should have the same learning rate at initialization and we will take as an objective function the MSE loss:
(())
Σ (X.9)EB
((++) (x; w|(
(6)
over batch B of training datapoints. The μP heuristic posited in (Yang al.) 2022) states that the best learning rate is the largest one for which Az(") remains O(1) for all f = 1..... ++ 1 independent of
the network width n. To make this precise, we will mainly focus analyzing the average squared change (second moment) of Az and ask to find learning rates such that
E. (A=!") O(1) for all & [1
(7)
with mean-field initialization, which coincides with equation 2 except that the final layer weights are initialized with variance 1/n² instead of 1/n. Our next result provides way estimate how the μP learning rate (i.e. the one determined by equation] 7) depends on the graph topology. To state it, we need two definitions:
DAG Width. Given DAG (V, E) with V {O..... L 1} we define its width be the number of unique directed paths from the input vertex 0 to output vertex L + 1.
Path Depth. Given a DAG g = (V, E), the number of ReLU activations in a directed path from the input vertex 0 to the output vertex L is called its depth, denoted as Ln (p 1,... ,
We now ready state the main result of this section:
5

---

Published as a conference paper at ICLR 2024
Learning rate scaling in DAG. Consider ReLU network with associated DAG G EV. hidden layer width n. hidden layer weights initialized as in $ $3.22 (3.2andd and the output weights initialized with mean-field scaling. Consider batch with one training datapoint (2, v) IR22 sampled independently of network weights and satisfying
|
=1,
E [ = O, Var y 1.
The pPP learning rate (17") for hidden layer weights is independent of n but has non-trivial dependence of depth, scaling as
1/2
n" =
Σ 1
1p
(8)
where the sum is over unique directed paths through G, and the implicit constant is independent of the graph topology and of the hidden layer width n.
This result indicates that, we should use smaller learning rates for networks with large depth and width (of its graph topology); and vice versa. We include our derivations in Appendix
3.4 LEARNING RATES IN DAG NETWORK WITH CONVOLUTIONAL LAYERS
In this section, we further expand our scaling principles to convolutional neural networks, again, with arbitrary graph topologies as we discussed in 3.1
Setting. Let Raxm be the input, where is the number of input channels and m is the number of pixels. Given 2(() E m for l E [1, L. 1], we first use operator ((-) to divide 200) into m patches. Each patch has size gn and this implies mapping of (x=(0)) E IRaxx. For example, when the stride is 1 and q 3, we have:
(E) T
21,0:2
(L.m 1om+1
)(=())=
(4)
(4) fn.m-1mmt1
2n.0:2
where we let
2,0 mm
(
(2)
0, i.e. zero-padding, Let (( E Bnxqn we have
2, (x) Eie wil' 4 (()))
f=0
{= L+1
Learning rate scaling in CNNs. Consider ReLU-CNN network with associated DAG G (V, E). kernel size q, hidden layer width n, hidden layer weights initialized as in $3.22 and the output weights initialized with mean-field scaling. The uP learning rate 17" for hidden layer weights is independent of n but has a non-trivial dependence depth, scaling as
1/2
n =
Σ
3 P
1
(9)
This result implies that in addition network depth and width, CNNs with larger kernel size should use smaller learning rate. We include our derivations in Appendix C
3.5 IMPLICATIONS ON ARCHITECTURE DESIGN
We would like to emphasize that, beyond training deep networks to better performance, our work will imply much broader impact in questioning the eredibility of benchmarks and algorithms for automated machine learning (AutoML).
Designing novel networks is vital the development of deep learning and artificial intelligence. Specifically, Neural Architecture Search (NAS) dramatically speeds up the discovery novel
architectures in principled way (Zoph Le 2016 Pham et al. 2018 Liu al. 2018) and

---

Published as a conference paper at ICLR 2024
meanwhile many of them archive state-of- of-the-art performance. The validation of NAS algorithms largely hinges on the quality of architecture benchmarks. However, in most literature, each NAS benchmark adapts the same set of training protocols when comparing diverse neural architectures, largely due to the prohibitive cost of searching the optimal hyperparameter for each architecture.
To best knowledge, there exist very few in-depth studies of "how train" architectures sampled from the NAS search space, thus we are motivated to examine the unexplored "hidden gem" question: how big a difference can be made we specifically focus on improving the training hyperparameterss? We will answer this question in $
4 EXPERIMENTS
In our experiments, we will apply our learning rate scaling to weights and bias. We study our principles MIPs, CNNs, and networks with advanced architectures from NAS-Bench-201 (Dong & Yang (020). All our experiments are repeated for three random runs. We also include more results in the Appendix ($ E. E. for ImageNet, $E.2 for the GeLU activation).
We adapt our principles as follows:
1. Find the base maximal learning rate of the base architecture with the smallest depth (L = 1 in our case: sea--hidden--oupput conduct grid search over a range of learning rates, and find the
maximal learning rate which achieves the smallest training loss at the end of one epoch"" !T]
2. Initialize the target network (of deeper layers or different topologies) according to equation
Train the target network by scaling the learning rate based on equation for MLPs or equation for CNNs.
The above steps are much cheaper than directly tuning hyperparameters for heavy networks, since the base network is much smaller. We verify this principle for MIPs with different depths in Appendix D
4.1 MLPs WITH TOPOLOGY SCALING
MLPs (Topology Scaling)
Maximal LR (Our Estimation)
3.00
0.838
We study MLP networks by scaling the learning rate to different graph structures based on equation g In addition to the learning rate, variances of initializations are also adjusted based on the in-degree of each layer in network.
km
To verify our scaling rule of both learning rates and ini- tializations, first find the "ground truth" maximal learning rates via grid search on MLPs with differ- graph structures, and then compare them with our scaled learning rates. As shown in Figure our estima- tion strongly correlates with the "ground truth" max- imal learning rates (r = 0.838). This result demon- strates that our learning rate scaling principle can be generalized to complicated network architectures.
2011
0002
Maximal LR (Experiments)
Figure 2: MLPs with different topologies (graph structures). X-axis: "ground truth" maximal learning rates found by grid search. Y-axis: esti- mated lcarning rates by our principle (cquation The red line indicates the identity. Based the "ground truth" maximal learning rate of the basic MLP with L = 1, we scale up both learning rates and initialization to diverse architectures. The radius of dot indicates the variance over three random runs. Data: CIFAR-II 10.
4.2 CNNS WITH TOPOLOGY SCALING
We further extend our study to CNNs. Our base CNN is of L 1 and kernel size 3, and we scale the learning rate to CNNs of both different graph structures and kernel sizes, based on equation 9 Both the learning rate and the variances of initializations are adjusted for each network.
Rationales for finding maximal learning rates for the 1st epoch: 1) Finding hyperparameters early training both practically effective and widely adopted (Tab. and of (Bohdal et al. (2022). Fig. and 3a of (Egele et al. 2023). Analyzing subsequent training epochs will require more computations and careful design of HPO algorithms. 2) /PP Yang et al. (2022) finds optimal learning rates in early training (Fig. 4 and 19). Exploiting training epochs (for HPO purposes) is orthogonal our method and is heyond the scope of our work.
7

---

Published as a conference paper at ICLR 2024
CNNs (Topology & Kernel Scaling)
Again, we first collect the "ground truth" maximal learning rates by grid search on CNNs with different graph structures and kemell sizes, and compare them
Maximal LR (Our Estimation)
6,556
Acrued
Fersel
Acruel 2
with our estimated ones. As shown in Figure 3 our esti- mation strongly correlates with the "ground truth" maxi- mal learning rates of heterogeneous CNNs (r 0.856). This result demonstrates that our initialization and learn- ing rate scaling principle can be further generalized to CNNs with sophisticated network architectures and dif- ferent kernel sizes.
0100
0.14
0.08
Maximal LR (Experiments)
Figure 3: CNNs with different graph structures and kernel sizes. The x-axis shows the "ground truth" maximal learning rates found by grid search. The y-axis shows the estimated leam- ing rates by our principle (cquation 9). The red line indicates the identity. Based on the "ground truth" maximal learning rate of the CNN with 1 and kernel size as 3, we scale up both learning rates and initialization to diverse archi- tectures. The radius of a dot indicates the vari- ance over three random runs. Data: CIFAR-10.
4.3 BREAKING NETWORK RANKINGS NAS
Networks' performance ranking is vital to the compar- ison of NAS algorithms (Guo et al. 2020 Mellor et al. 202T, Chu et al. 202T Chen et al.)
2021 If the rank- ing cannot faithfully reflect the true performance of networks, we cannot trust the quality of architecture benchmarks. Here we will show that: by simply better train networks, we can easily improve accuracies and break rankings of architectures (those in benchmarks or searched by AutoMI. algorithms).
We will use NAS-Bench-201 (Dong & Yang, 2020) as the test bed. NAS-Bench-200 provides a cell-based search space NAS benchmark, the network's accuracy is directly available by querying the database, benefiting the study of NAS methods without network evaluation. contains five heterogeneous operator types: none (zero), skip connection, conv1 x 1, comv3 x 3 convolution, and average pooling3 x 3. Accuracies are provided for three datasets: CIFAR-10, CIFAR-100, ImageNet16-1200 However, due to the prohibitive cost of finding the optimal hyperparameter set for each architecture, the performance of all 15,625 architectures in NAS-Bench-20) is obtained with the same protocol (learnnng rate, initialization, etc.). Sub-optimal hyperparameters may setback the maximally achievable performance of each architecture, and may lead to unreliable comparison between NAS methods.
We randomly sample architectures from NAS-Bench-201, and adopt the same training protocol as Dong & Yang (2020) (batch size, warm-up, learning rate decay, etc.). The only difference is that we use our principle ($ 3.4) to re-scale learning rates and initializations for each architecture, and train networks till converge. Results are shown in Figure 4 We derive three interesting findings from our plots to contribute to the NAS community from novel perspectives:
1. Compared with the default training protocol on NAS-Bench-201, neural architectures can benefit from our scaled learning rates and initialization with much better performance improvements (left column in Figure This is also a step further towards an improved "gold standard" NAS benchmark. Specifically, on average we outperform the test accuracy in NAS-Bench-201 by 0.44 on CIFAR-10, 1.24 on CIFAR- 100, and 2.09 on ImageNet16-120. We also show our superior performance over μPP in Appendix E.3
2. Most importantly, compared with the default training protocol on NAS-Bench-201, architectures exhibit largely different performance rankings, especially for top-performing ones (middle column in Figure 4) This implies: with every architecture better--raineed state-of-the-an arr NAS algorithms evaluated on benchmarks may also be re-ranked. This is because: many NAS methods (e.g. works summarized in (Dong et al.],2021) are agnostic to hyperparameters (i.e. search methods do not condition on training hyperparameters). Changes in the benchmark's training settings will not affect architectures they searched, but instead will lead to different training performances and rankings of searched architectures.
3. By adopting our scaling method, the performance gap between architectures is mitigated (right column in Figure In other words, our method enables all networks to be converged similarly well, thus less distinguishable in performance.
8

---

Published as a conference paper at ICLR 2024
CIFAR- 10
CIFAR-10
CIFAR-10 10
Painwise Distance of Architecurrss
Test Accuracy (Ouns
90-
Correaaionn vs. NAS Bench
[faakiing of Archibreure)
NGG Beah
0.8
Accuracy)
3.0
60)
2.5
60-
0.-
00-
100
40
20
1051
100
00
40
1051
Test Accuracy (05--Benl--211)
Top Architectures (%)
Top Architectures (%)
CIFAR-100
CIFAR-100 100
CIFAR-100 100
Test Accuracy (Own)
Correlatonn vs NAS Bench
(Tanking of Alchitectures)
0.5
Painvise Disamoo Architectures
Cco
N45-8aoo-211
60-
0.T-
(Test Accuracy)
0.0-
60
20
0.5
0.4
-00
100
00
40
20
1051
100
105
Test Accuracy (NAS-Bench-201)
Top Architectures (%)
Top Architectures (%)
ImageNet16-1 1- 120
ImageNet16-125
ImageNet16 120
Correlatoo: Our vs. NAS B..cee 201
0.3
Pairwise Dicance of Architectures
NAS DoS-Bexch222
Test Accuracy (Ours)
(Rarkings Architecluee)
0.8
(Test Acauras)
0.0
0.5
0.3
70 30
100
50
105
100
Test Accuracy (NAS-Bench-201)
Top Architectures (%)
Top Architectures (%)
Figure 4: We adopt our scaling principles to existing architecture benchmarks. Left column: better accu-
racy. For different network architectures, we plot the accuracy trained with different scaling principles (y-axis "Re-scaled") against the accuracy trained with fixed training recipe (x-axis represents a unique architecture with different layer types and topologies (Figure in (Dong & Yang) (020)). Our principle (blue dots) achieves better accuracy compared to the benchmark (red line y = 2)) Middle column: network rankings in benchmarks are fragile. We compare networks' performance rankings at different top K% percentiles (K = 100, 90, 10. 5, I; bottom right dots represent networks the top-right in the left column), trained by our method benchmarks, and find better networks are ranked more differently from the benchmark. This indicates current network rankings in benchmarks (widely used to compare NAS algorithms) can be easily broken by simply better train networks. Right column: less distinguishable architectures. We plot the pairwise performance gaps between different architectures, and find our principle makes networks similar and less distinguishable in terms of their accuracies.
Each dot
CONCLUSION
Training a high-quality deep neural network requires choosing suitable hyperparameters, which non-trivial and expensive. However, most scaling or optimization methods are agnostic the choice of networks, and thus largely ignore the impact of neural architectures on hyperparameters. In this work, by analyzing the dependence of pre-activations on network architectures. we propose a scaling principle for both the learning rate and initialization that can generalize across MLPs and CNNs, with different depths, connectivity patterns, and kernel sizes. Comprehensive experiments verify our principles. More importantly, our strategy further sheds light on potential improvements in current benchmarks for architecture design. We point out that by appropriately adjusting the learning rate and initializations per network, current rankings in these benchmarks can be easily broken.Our work contributes to both network training and model design for the Machine Learning community. Limitations of our work: 1) Principle of width-dependent scaling of the learning rate. 2) Characterization of seeth-dependence of learning rates beyond the first step / early training of networks. 3) Depth-dependence of learning rates with the existence of normalization layers.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
B. Hanin and Z. Wang are supported by NSF Scale-MoDL (award numbers: 2133806, 2133861).
REFERENCES
Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley, Rezero is all you need: Fast convergence at large depth. In Uncertainty in Artificial Intelligence, pp. 13611 PMLR, 2021.
James Bergstra, Daniel Yamins, and David Cox. Making a science of model search: Hyperparameter optimization in hundreds dimensions for vision architectures. In International conference on machine learning, pp. 115-123. PMLR, 2013.
Ondrej Bohdal, Lukas Balles, Beyza Ermis, Cédric Archambeau, and Giovanni Zappella. Pasha: Efficient hpo with progressive resource allocation. arXiv preprint arXiv:2207.06940, 2022.
Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. arXiv preprint arXiv:2102.11535, 2021.
Yutian Chen, Xingyou Song. Chansoo Lee, Zi Wang, Qiuyi Zhang, David Dohan, Kazuya Kawakami, Greg Kochanski, Arnaud Doucet, Marc' aureloo Ranzato, et al. Towards learning universal hyperpa- rameter optimizers with transformers. arXiv preprint arXiv:2205. 13320, 2022.
Xiangxiang Chu, Bo Zhang, and Ruijun Xu. Fairnas: Rethinking evaluation fairness of weight sharing neural architecture search. In Proceedings of the IEEE/CVF International Conference on computer vision, pp. 12239-12248, 2021.
Aaron Defazio and Léon Bottou. A scaling calculus for the design and initialization of relu networks. Neural Computing and Applications, (1117)7:4400714444 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database, In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Emily Dinan, Sho Yaida, and Susan Zhang. Effective theory transformers at initialization. arXiv preprint arXiv:2304.0200 2023.
Xuanyi Dong and Yang. Nas-bench- 102: Extending the scope of reproducible neural architecture search. arXiv preprint arXiv:2001.003266 2020.
Xuanyi Dong, Lu Liu, Katarzyna Musial, and Bogdan Gabrys. Nats-bench: Benchmarking nas
algorithms for architecture topology and size. arXiv preprint 2000:00000900443,, 2020a.
Xuanyi Dong, Mingxing Tan, Adams Wei Yu, Daiyi Peng, Bogdan Gabrys, and Quoc V Le. Autohas: Efficient hyperparameter and architecture search. arXiv preprint 1.xXi:220060036566 2020b.
Xuanyi Dong, David Jacob Kedziora, Katarzyna Musial, and Bogdan Gabrys. Automated deep learning: Neural architecture search is not the end. arXiv preprint arXiv:211 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Romain Egele, Isabelle Guyon, Yixuan Sun, and Prasanna Balaprakash. Is one epoch all you need for multi-fidelity hyperparameter optimization? arXiv preprint 1rXiv:2307.15422, 2023.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
10

---

Published as a conference paper at ICLR 2024
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng. Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XVI 16, pp. 544-560. Springer, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEFE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generaliza- tion gap in large batch training of neural networks. Advances in neural information processing systems, 30, 2017.
Samuel Horváth, Aaron Klein, Peter Richtárik, and Cédric Archambeau. Hyperparameter transfer learning with adaptive complexity. In International Conference on Artificial Intelligence and Statistics, pp. 1378-1386. PMLR, 2021.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700 4708, 2017.
Xiao Shi Huang, Felipe Perez, Jimmy Ba, and Maksims Volkovs. Improving transformer optimization through better initialization. In International Conference on Machine Learning, pp. 4475 44833 4483. PMLR, 2020.
Ilija llievski and Jiashi Feng. Hyperparameter transfer learning through surrogate alignment for efficient deep neural network training. arXiv preprint
2016.
Gaurav lyer, Boris Hanin, and David Rolnick. Maximal initial learning rates deep relu networks.
arXiv preprint arXiv:22/2.07225. 2022.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in neural information processing systems, pp. 8571-8580, 2018
Kevin Jamieson and Ameet Talwalkar. Non-stochastic best arm identification and hyperparameter optimization. In Artificial intelligence and statistics, pp. 240-248. PMLR, 2016.
Samy Jelassi, Boris Hanin, Ziwei Ji, Sashank J Reddi, Srinadh Bhojanapalli, and Sanjiv Kumar. Depth dependence of up learning rates in relu mips. arXiv preprint arXiv:2305.07810, 2023.
Aaron Klein and Frank Hutter. Tabular benchmarks for joint architecture and hyperparameter optimization. arXiv preprint urXiv:1905.04499 2019.
Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Jonathan Ben-Tzur, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. system for massively parallel hyperparameter tuning. Proceedings of Machine Learning and Systems, 2:230-246 220022466622 2020.
Yang Li, Yu Shen, Huaijun Jiang, Tianyi Bai, Wentao Zhang, Ce Zhang, and Bin Cui. Transfer learning based search space design for hyperparameter tuning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 967- 2022a.
Yang Li, Yu Shen, Huaijun Jiang, Wentao Zhang, Zhi Yang, Ce Zhang, and Bin Cui. Transbo: Hyperparameter optimization via two-phase transfer learning. In Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 956-966-666 2022b.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv: 1806 090500005, 2018.
11

---

Published as a conference paper at ICLR 2024
Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the difficulty of training transformers. arXiv preprint arXiv:2004.08244 2020.
Joe Mellor, Jack Turner, Amos Storkey, and Elliot J Crowley. Neural architecture search without training. In International Conference on Machine Learning, pp. 7588-7598. PMLR, 2021.
Daniel Park, Jascha Sohl-Dickstein, Quoe Le, and Samuel Smith. The effect of network width stochastic gradient descent and generalization: an empirical study. In International Conference on Machine Learning, pp. 5042-5051. PMLR, 2019.
Valerio Perrone, Rodolphe Jenatton, Mathias W Seeger, and Cédric Archambeau. Scalable hyperpa- rameter transfer learning. Advances in neural information processing systems, 31, 2018.
Hieu Pham, Melody Y Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv:1802.000 002.0000.00.2... 2018.
Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. arXiv preprint lrXiv:1611.0123. 2016.
Christopher Shallue, Jachoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George Dahl. Measuring the effects of data parallelism on neural network training. arXiv preprint arXiv: 1.03600, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Samuel L Smith, Pieter-Jan Kindermans, Chris Ying, and Quoe Le. Don't decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00489 2017.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012.
Danny Stoll, Jörg KH Franke, Diane Wagner, Simon Selg, and Frank Hutter. Hyperparameter transfer across developer adjustments. arXiv preprint arXiv:2010.1311 13 2020.
Mingxing Tan and Quoc V Le. Efficientne: Rethinking model scaling for convolutional neural
networks. arXiv preprint arXiv: 1905. 1946, 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2820-2828, 2019.
Saining Xie, Alexander Kirillov, Ross Girshick, and Kaiming He. Exploring randomly wired neural networks for image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2019.
arXiv preprint Sho Yaida. Meta-principled family of hyperparameter scaling strategies. arXiv:2210.04909, 2022.
Antoine Yang, Pedro M Esperança, and Fabio M Carlucci. Nas evaluation frustratingly hard. arXiv preprint arXiv: 1912. 12522, 2019.
Ge Yang and Samuel Schoenholz. Mean field residual networks: On the edge of chaos. In Advances in neural information processing systems, pp. 7103-7114, 2017.
Greg Yang and Edward J Hu. Feature learning in infinite-wide neural networks. arXiv preprint arXiv:2011.1452.. 2020.
Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.033: 0034660 2022.
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas- bench-101 101: Towards reproducible neural architecture search. In International Conference on Machine Learning, pp. 7105-7114 14, PMLR, 2019.
12

---

Published as a conference paper at ICLR 2024
Dani Yogatama and Gideon Mann. Efficient transfer learning method for automatic hyperparameter tuning. In Artificial intelligence and statistics, pp. 1077-1085. PMLR, 2014.
Jiaxuan You, Jure Leskovec, Kaiming He, and Saining Xie. Graph structure of neural networks.
arXiv preprint arXiv:2007.06559 2020.
Arber Zela, Aaron Klein, Stefan Falkner, and Frank Hutter. Towards automated deep learning: Efficient joint neural architecture and hyperparameter search. arXiv preprint arXiv:1807.066 006006 2018.
Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Residual learning without normalization via better initialization. In International Conference on Learning Representations, volume 3, pp. 2, 2019.
Chen Zhu, Renkun Ni, Zheng Xu, Kezhi Kong, W Ronny Huang, and Tom Goldstein. Gradinit: Learning to initialize neural networks for stable and efficient training. Advances in Neural Information Processing Systems, 34: 16l116141-642,, 2021.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning, arXiv preprint 1rX:5:161..01578, 2016.
A ARCHITECTURE-AWAR INITIALIZATION
Derivations for $3.22 Consider a random network input x sampled from a distribution for which and -x have the same distribution. Let us first check that the joint distribution of the pre-activation vectors (z) with L' O, ..... + 1 is symmetric around We proceed by induction on & this statement is true by assumption. Next, suppose we have proved the statement for O,..., l. Then, since z(f+1)(z)) (Ix) is linear function of (x)) (x)). we conclude that it is also symmetric around 0. A direct consequence of this statement is that, by symmetry.
When L
E
(
(4
- = (3))
lj= l= 0, ..... ++1
(10)
Let us fix h = 1,..., and vertex l we calculate Ow in equation 2
19]
27
E
20
=
Σ
wle j8
((e))
(E 1'=1
((e)) 27
E
n
(EEEEEE y'=1
Σ
(1I)
(E,')
We ask || = ||=te>| l?]
which yields
2
(12)
d(
We now seek to show
|
E
E
ve,
(13)
To do so, let us define for each = O. O...,,,++7
d(e, L 1) = (length of longest directed path in from to I. + 1)
13

---

Published as a conference paper at ICLR 2024
The relation equation 13 now follows from simple argument by induction show that by induction
on d(e, L + 1). starting with d(L, L + ) O. Indeed, the case d(e, L 1) simply corresponds
to 8 = L + 1. Next, note that if d( L + 1) > 0 and if (E", by the maximality of the path length in the definition of d(e, + 1). Hence, substituting equation 12
E, then d(e, L + + < d(L' L
into equation [1 completes the proof of the inductive step.
B ARCHITECTURE-DEPENDDER LEARNING RATES (FOR 3.3
We start by setting some notation. Specifically, we fix network input E R at which we study both the forward and backward pass. We denote by
(2) (z),
200 1= (x)
the corresponding pre-activations vertex E. We assume our network has uniform width over layers (ng n.), and
learning rates:
nl = learning rate of μ..
We will restrict to the case at the end. Further, recall from equation 6 that we consider the empirical MSE over a batch with single element (x,, y) and hence our loss
(LL4))
L 2
The starting point for derivations in 3.3 is the following Lemma
Lemma B.1 (Adapted from Lemma 2.1 in Jelassi et al. (2023)) For l = 1, ,.... L have
E8
A 2 )
A + B(0)
where
A =
Σ
Mas Mug Ou, 10
=1 2000pp 01
(0)
(14)
n
Ms 122
Σ
1
(15)
n²
(.IL L+I)EE
(11)
(61)
(12)
200000 (1)))
(6) (11 (())(()
(122
Fi Opz
2/2
0p3 231
022212 22
in ii
B = E
Σ ME
Th Tu
Mpp 8p 21
0yz
(1+1]
E (ELL+DEEE
Σ
0
2/
(c")
Ouc 200
(16)
di
Proof.
The proof of this result follows very closely the derivation of Lemma 2.1 Jelassi al. (2033) which considered only the case of MLPs. We first expand A Az by applying the chain rule:
Σ ASE
20p2 An,
(17)
where the sum is weights that belong some weight matrix W("", .f) for which there is directed path from in G and we' SGD update satisfies:
we've 'e denoted by A the change in after one step of GD. The
Au
Tu 0, (2(4+1)- -1) w)²
n/, 0,, (((tt))y).
-1)
(18)
2
14

---

Published as a conference paper at ICLR 2024
we've where we' ve denoted by (z. y) the training datapoint in the first batch. We now combine equation 17 and equation 18 to obtain:
Σ
mu (0) 0 0uz((t1) (v- 2((t11).
(19)
MSE.
Using equation 19
we obtain
27
E
A
(1)
27
E
Σ
The Opar 100 Opiz (1ttl (2(Lt3) -2)
uSe
E
Σ
na Mu2 Ouz 21 Ou
200000 (0) Ons (1 (1+ 2 Byy
(2((th)) 3)
PI-PaS
(20)
Here, we've used that by symmetry the answer is independent of i. Taking into account the distribution of 2(++1) and y, we have
(xLtt)) -v) 3)² (z(l-1))2 +
By
(21)
We plug equation 21 in equation 20 and obtain
B B[(A2 = A(O) + B(O,
(22)
where
Tu Mug Op, 2000 100 2 1(++1) 2,9 2(1-11 (alltt))
A E
Σ
(23)
$1
B
Σ
The Mug 2u (00 0mg 20, 2((++)) 2pp ((1-1)
(24)
P1-
By definition, have
wielttD~ ~ N (1((
((+11)
Σ
W Wil
)
(1,L+))EE==1
Therefore, integrating out weights of the form ( in equation 24 yields
(
E
1
Mu Tuo Bn 21 1 (0)
1
Σ
Σ
(2)0)) Op2 (2fe7)
(1+1)
in
(e
(+IEEE
j=l
=
E
1
Σ PE
Toe Ou 21
Opp
(1+1)
Σ
Σ
(8)) Ous
Oun 2
(F,L+)EEE
1=1
where in the last equality we used equation computation yields the expression for A()
10 This yields the desired formula for B A similar
As in the proof of Theorem 1.1 Jelassi et al.
2023 we have
A
O(n
due the presence of extra pre-factor of 1/n. Our derivation in $ 3.3 therefore comes down finding how (()) is influenced by the topology of the graph G.
15

---

Published as a conference paper at ICLR 2024
We then re- werte the high-level change of pre rreaaitivaiions (equation 5) in arbitrary graph topology. We need to accumulate all changes of pre- arevactivations that flow into a layer (summation over in in-degrees L' L 1):
A
E E Opzi Ap
(25)
E-+++1 MSE
Therefore, for the change of pre-activation of layer I + 1, we can simplify the analysis to each individual layer E" that connects L + 1 (i.e. edge (e" + L flow into layer l.
E), and then sum up all layers that
Next, we focus on deriving B(d) of each individual path in the case of the DAG structure of network architectures. It is important that here we adopt our architecture-awwat initialization ($ 3.2)
B =
Σ St
Mue Buz Sp 2/ 0pp
(1)
Σ
(e') 2j pur Σ 1=1
1000pp 0p2
(0)
((L+1) (1,+1)EE
Σ PI.MaSE
2u 10 10
2j 2p (1)
Tue Bu
2/
(26)
=E
Σ PE
10 The Tuu Dp2 2i Opzi 0n
(0
0pz (8)
Thus, we can see that with our architecture-wware initialization, ( reduce back to the basic sequential MLP case in Theorem 1.1 in Jelassi et al. (003) Therefore, we have:
P B(++1) ~ Σ
O(1² L.)
p=1
Thus
-1/2
Σ
1
where P is the total number of end-to-end paths that flow from the input the final output 24+1, and Lp is the number of ReLU layers each end-to-end path.
C LEARNING RATE SCALING FOR CNNS
Derivations for $3.4 In addition B(0 in equation 24 we further refer to the contributing term of
(()) in Jelassi et al. (2023 when #1 S and #2 E (or vice versa), denoted (());
( := E
1
7p USE
(1) 100
(27)
J1.12=1
We start from deriving the recursion of ( of each individual end-to-end path for convolutional layers of a kernel size as Again, we assume our network has uniform width over layers (nz = n).
(1,, U2 l then the contribution equation 24
17 (2)
E
Σ
(
(8-1)
(6-))
- (101))
9²
|0|| 1R
When μn < El - and μl E ll (or vice versa) the contribution to equation
24
2n (1) qE
1
Σ 0
((-1))
(0p 2/ (0))
n7 19 9(
Mat
k=1
1=1
16

---

Published as a conference paper at ICLR 2024
Finally, when M1, U2 S e-11 1 we find the contribution to equation 24 becomes
E
Mei Mpp { 20 0pg
) -m
1 (1-1) 0n 21 One 2
(1
1
0pp =2
(0)
n
n
M1.
1 1
B(c-1) + B(t-1).
n
Therefore, we have
_ (1/10) 2
q² n²
||l|| c + + 11 qc(t-1)
+ (1+ 1 Blt-1) + B(c-1)
1 B (1))
2
||||||
e s & + n q(t-)1)
1 B(e- + n
((1-1)
n²
n
We further derive (*), When E the contribution to equation 27 is
9
(
|
c
E
n
n
J1.J2
When μ 56- 1 the contribution to equation 27 is
Σ (
(2) 2
+ 11 (1-11 n
(0r=1)) (2(0))
n
1 c(e-1) n
c(e-1)
Therefore,
n
| n²
e - + c(e-1)
(1+ 1 Ō(--1)
+
n
Finally, for each end-to-end path, we have
B = 0(12
Therefore, together with
3.3 want
P
-1/2
13
1 ~
D MLPs WITH DEPTH SCALING
We verify the depth-wise scaling rule in Jelassi et al. (023). We scale a vanilla feedforward network by increasing its depth (adding more Ainear-ReLU layers). Starting from the most basic feedforward network with of learning rates and find the maximal learning rate. We then scale the learning rate to feedforward networks of different depths according to equation X Different feedforward networks share the same initialization since both the out-degree and in-degree of all layers are 1.
(an input layer, a hidden layer, plus an output layer), we first scan wide range
To verify the scaling results, we also conduct the grid search of learning rates for feedforward networks of different depths, and compare them with the scaled learning rates. As shown in Figure 5 on CIFAR-10 the estimation strongly correlates with the "ground truth" maximal learning rates (r 0.962), and the plotted dots are very close to the identity line. This result demonstrates that the wist learning rate scaling principle in Jelassii et al, (2023) is highly accurate across feedforward neural networks of different depths and across different datasets.
17

---

Published as a conference paper at ICLR 2024
MLPs (Depth L Scaling)
Maximal (Our Estimation)
0.005
0,902
0.000
Maximal LR (Experiments)
Figure 5: MLP networks of different depths on CIFAR-10. The x-axis shows the "ground truth" maximal learning rates found by grid search experiments. The y-axis shows the estimated learning rates by principle in equation [ The red line indicates identity. Based on the true maximal learinng rate of feedforward networks of 3, we scale up to L = 10. The radius of dot indicates the variance over three random runs.
E MORE EXPERIMENTS
E. IMAGENET
MLPs (Depth L Scaling) (ImageNet)
MLPs (Topology Scaling) (ImageNet)
Maximal UR (Our Estimation)
8.866
Maximal LR (Our Estimation)
0.729
8008
0.60
0.055
0.03
0.00
10
6:06
11..00
00000
E1200
2.00
0.133
400h
0.00
0.003
01:01
00506
Maximal LR (Experiments)
Maximal LR (Experiments)
Figure MLP networks (RcLU activation) of different depths (left) and graph topologies (right) on Ima- geNet[Dong et al. (2009). The x-axis shows the "ground truth" maximal learning rates found by our grid search experiments. The y-axis shows the estimated learning rates by our principle in equation The red line indicates the identity. The radius of dot indicates the variance over three random runs.
Similar to Figure and Figure 2] further verify the learning rate scaling rule ImageNet Deng et al. (2009). We scale up a vanilla feedforward network by increasing its depth (adding more Linear-Rel.U layers) changing its graph topology. As shown in Figure our estimations achieve strong correlations with the "ground truth" maximal learning rates for both depth-wise scaling 0.856) and topology-wise scaling (r 0.729). This result demonstrates that our learning rate scaling principle is highly accurate across feedforward neural networks of different depths and across different datasets.
E.2 THE GELU ACTIVATION
To demonstrate that our learning rate scaling principle generalize different activation functions, we further empirically verify MLP networks with GELU layers (on CIFAR-10). We scale up vanilla feedforward network by increasing its depth (adding more Linear-ReLU layers) changing its graph topology. Again, our estimations achieve strong correlations with the "ground truth" maximal learning rates for both depth-wise wise scaling (r 0.920) and topology-wise scaling (r 0.680). Moreover, for CNNs, our estimation can also achieve 0.949.
E.3 THE /P HRURISTICS
We further compare the μP heuristics (Yang et al. 2022) with our architecture-awee aware initial- ization and leaming rate scaling by training architectures defined in NAS-Bench-20 (Dong &
18

---

Published as a conference paper at ICLR 2024
MLPs (GELU) (Depth Scaling)
MLPs (GELU) (Topology Scaling)
CNNs (GELU) (Topology & Kernel Scaling)
Moximal UR (Our Esimation)
Maximel LR (Dur Estimution
LR (Our Eslmnntoo))
Esnwe
6.30
0.0
1:0.0
10
6005 10r0 cat0 onts setee
e3t0 oene
1000 E305 100 o
6.0
Maximal (Experiments]
Maxmal LR (Experiments)
Maximal LR (Experiments)
Figure 7: MLP networks with GELU activations of different depths (left) and graph topologies (middle), and CNNs (right), on CIFAR-IO. The x-axis shows the "ground truth" maximal learning rates found by our grid search experiments. The y-axis shows the estimated learning rates by our principle in equation [ The red line indicates the identity. The radius of a dot indicates the variance over three random runs.
Yang. 2020 We follow the usage from https: httpt//ttttt.u.eeo com /m1 m crosoft/mup?tant readie ov fi le #basic
to set the base model, and train with MuSGD. The learn-
ing rate as 0.1 following the original setting NAS-Bench-201, From Figure g we can see that uP initialization and scaling strategy yields inferior results.
CIFAR-10
CIFAR-100
ImageNet16-122
70-
Test Acauracy (Ours
Test Accuracy (Ours)
60
Test Accuracy [Ours)
00-
40-
20
20-
20
70
30
Test Accuracy (cP)
Test, Accuracy (jP)
Test Accuracy (jP)
Figure 8: Comparison between the μP heuristics Yang al. (2022) (x-axis) and our architecture-wware initial-
ization and lcarning scaling (y-axis) on NAS-Hench-20) Dong & Yang (0200. Lefft: CIFAR-10. Middle:
CIFAR-100. Right: ImageNet-1661222
E.4 NETWORK RANKINGS INFLUECCD BY RANDOM SEEDS
To delve deeper into Figure analyze how the random seed affects network rankings. Specifically, MAS-Bencht 201 reports network performance trained with three different random seeds (seeds [777, 888, 999]). For CIFAR-I00, we created plot similar Figure 4 middle column, but shows pairwise ranking correlations among those three random seeds. As shown in Figure although we observe different ranking correlations between seeds, they are consistently higher (i.e., their rankings more consistent) than those produced by method. This confirms that changes in network rankings by our architecture-aware hyperparameters are meaningful, which can train networks to better performance.
19

---

Published as a conference paper at ICLR 2024
CIFAR-100
Uoor: Ours vs, NAS AAS-Bench-225 NAS-Bench-20 sencc-221
Architectuures
0.7
0.5
0.4
vs. bench-20 201 (all seeds)
tench-201
988)
bench-201 Iseed 777 vs. 9991
888
0.2
20 105
Top Architectures (%)
Figure 9: Comparison of network rankings influenced by our method versus random seeds. NAS-Beneh 201 Dong & Yang (2020) provides training results with three different random seeds (777, 888, 999). Bluc dots represent the Kendall-Tar Tau correlations between networks trained by our method and the accuracy from NAS- Bench-201 (averaged over three seeds). We also plat correlations between random seeds in pairwise manner. We compare networks' performance rankings at different top 5% percentiles (K" = 100, 90,,90.- .... 10, 1; bottom right dots represent networks on the top-right in the left column). This indicates that, although network rankings be influenced by randomness during training, method leads significant changes in their rankings while enabling these networks to achieve better accuracy.
20


Reviews:
[{"review_id": "mhEZWbl0Km", "reviewer": "Reviewer_BP74", "summary": "This work provides an extension of $\\mu$transfer to arbitrary network structures and an application of the resulting method to neural architecture search (NAS), specifically in order to make NAS benchmarks more comparable by ensuring all competing networks are given a better chance to train. The major contributions:\n\n* An improved way to set initializations and layer specific learning rates to stabilise training over network scales\n* Experimental results demonstrating that NAS benchmarks are flawed: making the training process more stable massively changes the results\n\nWeights are initialized to be normally distributed with variance:\n$$\n\\sigma^2 = \\frac{C^{(l', l)}}{n} = \\frac{2}{d_{in}^{(l')}}\n$$\nfor an $n \\times n$ weight matrix with $d_{in}^{(l')}$ in-degree (the number of connections to the layer from the previous layer).\nAlso, the final layer is scaled by $\\propto \\frac{1}{n^2}$.\n\nScale the layerwise learning rates according to\n$$\n\\eta^{*} \\simeq c \\left( \\sum_{p=1}^{P} L_p^3 \\right)^{-1/2}\n$$\nwhere $c$ is a layer-independent constant and $L_{p}$ is the number of relu layers on path $p$.", "questions": "For each network architecture sampled in the NAS-Bench experiments, the network learning rate is first tuned at small scale. Was the learning rate also tuned on the networks being trained without this hyperparameter transfer method? This may already be in the experiments section or the Appendices and I may have missed it.\n\nWhy are the maximal LRs found by experiment in Section 4.1 and 4.2 not the same as those computed by the theory? Is there any way they could be brought closer together?\n\nHow was the experiment in Sections 4.1 and 4.2 performed? I guess you find the optimal learning rate at a small scale on a set of networks, then scale the network up using the method described in the paper to scale the initalization and learning rates, then perform a grid search to find the optimal learning rates to compare against?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Neural Architecture Search needs a way to verify that the networks being searched over are actually being trained sufficiently. The goal of this paper to standardise that process around a framework that should allow much more reliable results in this area of study.\n\nI have not checked all the derivations but the theory appears to be correct. I am confident that I could verify the results if I had more time.\n\nThe contributions as presented are met:\n\n* The initialization is provided, justified and works in the experiments\n* The extension to $\\mu$P is introduced clearly and also works in experiments\n* Experiments demonstrate a key failing of NAS benchmarks and present a solution\n\nThe two limitations in the literature that this paper addresses are:\n\n1. $\\mu$P is not defined for arbitrary DAGs, only feedforward networks\n2. Neural Architecture Search benchmarks are useless if we can't trust that the networks have been trained well\n\nThe presentation is good, explaining the key points of the $\\mu$P paper in a brief way, better than the original paper.\nThe authors use an idiosyncratic system of emphasis, using both underlines and bold fonts on key points. This actually works quite well and I found the points being emphasised generally did earn more attention. The authors also use § in place of \"Section\", which I guess saved space, and works just as well.", "weaknesses": "The initialization, learning rate tuning and layer specific learning rates are introduced to maintain the update scale at $O(1)$ but the update scale during training doesn't appear to have been measured. It would be nice to see empirical verification that the method is working as intended. Although, I understand that the results in 4.1 and 4.2 both indicate that it is.\n\nGiven the equations above it is not easy for the reader to replicate the exact initialization and learning rate scaling required. Some pseudocode or a reference implementation would help a lot. It's been a problem for $\\mu$P adoption as well, that practitioners find that there were ambiguities in the description that make it hard to implement in practice. For example, I don't know where the $c$ parameter is set and I'm not fully confident how to count the number of paths to produce $L_p$.\n\nIt is stated \"the final layer weights are initialized with variance $1/n^2$ instead of $1/n$\" on page 5, and I know this is from $\\mu$P but it is never stated in this paper why.\n\nMinor presentation issues:\n\n1. \"most designs of principles\" in abstract doesn't make sense\n2. Why is the MSE loss introduced in equation 6? The experiments are mostly classification results using a cross-entropy loss\n3. Equation 8 contains $L_p$ but $L_p$ is only defined in the Appendix, it should be defined near the equation\n4. Section 3.5 \"speed-up\" -> \"speeds up\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ip5sliTNuy", "reviewer": "Reviewer_drLU", "summary": "The paper aims to establish principles for initializing and selecting learning rates in neural network architectures characterized by directed acyclic graphs (DAGs), which can be highly irregular in structure. The authors propose an initialization method to maintain pre-activation variance during forward propagation and derive architecture-specific learning rates using a maximal update prescription. Experimental validation demonstrates their effectiveness and potential for improving neural architecture search benchmarks by enhancing network training and rankings.", "questions": "See previous section of weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tThe paper builds on previous research in hyperparameter and architecture search, aiming to establish a principled connection between weight initialization and learning rate choices with both MLP and CNN architectures.\n-\tIt introduces an architecture-aware modified fan-in initialization scheme that preserves information flow through various graph topologies.\n-\tThe paper analytically derives formulas for scaling learning rates based on architecture topology, specifically using the maximal update (μP) heuristic.\n-\tExperimental results demonstrate the superior performance of the proposed methods and highlight their potential to reshape network rankings in standard NAS benchmarks.\n-\tThese findings suggest that implementing the proposed principles may lead to improved evaluations of NAS algorithms.", "weaknesses": "-\tThe clarity of the main body could be improved a lot, via providing a concise summary of main derivations, potentially by reducing the repetitive criticism of architecture search.\n-\tThe experimental section raises several concerns. The strategy of finding a base maximal learning rate for one epoch may not be meaningful for practical training cycles, where learning rates follow complex schedules. The criticism of NAS for using \"the same hyperparameters\" overlooks the intricate learning rate strategies commonly employed.\n-\tThe empirical results are also not entirely convincing. Figure 2 exhibits a weak correlation with questionable linearity and Figure 3 has a limited range of learning rates. The proposed improvements in Figure 4 seem to be primarily in lower accuracy regimes, raising doubts about absolute improvement on the top-performer architectures (which are of the most interest)\n-\tThe discussion of prior art is limited, particularly regarding weight initialization and learning rates, and it's unclear how the proposed method advances over existing insights in the literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6Tx3JxxiS6", "reviewer": "Reviewer_nJvL", "summary": "This paper proposes a method for optimizing hyperparameters in deep neural networks that takes into account the impact of neural architectures on hyperparameters. The authors characterize the dependence of initializations and maximal learning rates on the network architecture, including depth, width, conv kernel size, and connectivity patterns. They generalize their initialization and learning rates across MLPs and CNNs with sophisticated graph topologies. The authors verify their principles with comprehensive experiments and demonstrate that network rankings can be easily changed by better training networks in benchmarks with their architecture-aware learning rates and initialization.", "questions": "Same as Weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tThe paper begins with a compelling motivation, shedding light on prevalent trends in the current literature landscape. It questions the evaluation of architectural choices and the often overlooked dependency on hyperparameters.\n-\tThe paper provides thorough and well-supported proofs for its derivations related to initialization and maximal learning rates. It introduces a straightforward adjustment to weight initialization by scaling with the layers' in-degree, leading to notable performance enhancements downstream.\n-\tWhile the derivation draws substantial inspiration from previous works, it seems technically correct (and original to certain extent), and diligently cites these sources. Although maybe not the most groundbreaking, gaining a deeper understanding of how to determine initial learning rates for specific architectures holds significant importance. \n-\tThe paper critiques the evaluation methodology employed in NAS benchmarks, uncovering significant shortcomings in the process in their experiments. The evaluation of NAS encompasses a wide range of comprehensive datasets, adding to the paper's robustness and relevance. There is a strong likelihood that the rules derived from this work could prove relevant and valuable for practitioners.\n-\tThe paper exhibits excellent writing and organization, ensuring a smooth reading experience.", "weaknesses": "A few questions seem to hinder my understanding of this paper’s contributions, particularly with regard to the root causes behind the perceived limitations of the μP method. \n\nFirst, the paper argue that their proposed method outperforms the μP initialization and scaling method introduced by Yang et al. in 2022, often achieving significantly better results. Noting that μP results were not included in the main paper and were instead instead deferred to a much later section E.3. \n\nFrom a theoretical standpoint, Yang et al. (2022) primarily derived their scaling strategy for multi-layer perceptron (MLP) architectures in their main text, with additional derivations provided in Appendix L. On the empirical side, Yang et al. (2022) also presented experimental results on hyperparameter transfer for ResNet and Transformer architectures.\n\nThe present paper introduces a novel Directed Acyclic Graph (DAG) tool to derive \"architecture-aware\" hyperparameters. However, a fundamental question remains: What are the specific failure modes of the μP when it encounters intricate network topologies? Is the limitation attributed to a violation of Desiderata L.1, or are there other contributing factors at play?\n\nAdditionally, further clarification is needed regarding the assumptions made and the empirical comparisons drawn, especially across various architectural configurations. For example, it was not clearly mentioned until late, that the authors’ theory cannot apply to normalization layers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DVMBTMLoSJ", "reviewer": "Reviewer_X2JZ", "summary": "This paper provides valuable insights into the dynamic relationship among network architecture, initialization methods, and learning rates. It combines theoretical advancements with compelling empirical evidence, offering a precise characterization of how network architecture influences the interdependence of initialization strategies and the determination of optimal learning rates. The empirical evaluation is carried out on the NAS-Bench dataset, and the findings hold relevance not only for MLPs but also for CNNs with complex architectural designs. As a result, the implications of this research extend beyond the confines of deep learning theory, sparking meaningful reflections within the practical neural network design and NAS.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper delivers a compelling contribution that encompasses both theoretical and empirical dimensions. The clarity of its exposition, coupled with effective visual aids, greatly enhances comprehension of the concepts presented.\n\nIn terms of theoretical advancements, the authors expand upon the maximal-update (μP) scaling strategy initially introduced by Yang et al. in 2022. The authors delve into an investigation of how hyperparameters depend on various aspects of network architecture, including depth, connectivity patterns such as residual skips and long-range connections, kernel sizes, layer types, and convolution extensions. \n\nThe authors establish a link between learning rate scaling and layer depth for the first time, demonstrating its effectiveness over approaches that do not account for depth. Section E2 shows that the method also extends to GeLU neurons.\n\nThe authors showcase that the proposed scaled learning rates and initializations bring improvements in achievable accuracies across a broad spectrum of neural network architectures within the NAS-Bench search space. This outcome bears significant implications for the practical design of neural networks and the field of NAS, particularly considering the widespread adoption of NAS-Bench as a guiding benchmark.\n\nThe introduction of scaled learning rates and initializations yields a noteworthy consequence by narrowing the performance gap between \"good\" and \"bad\" architectures within NAS-Bench (Figure 4). They prompts questions about the ranking of architectures within NAS-Bench, prompting a critical reassessment of progress within the NAS field.", "weaknesses": "While this paper, on the whole, demonstrates a commendable effort, I encountered two noteworthy concerns while reading the experimental section:\n\n-\tThe rankings of architectures may be influenced by various factors during training, including random seeds. It raises the question of how the authors determined that their revised learning rates and initializations would have the most significant impact on their observation of updated rankings.\n-\tIt's recognized that NAS-Bench typically selects learning rates closer to those utilized by \"good architectures\" that are in practical use, reflecting an intentional bias towards favoring such architectures over \"bad architectures.\" However, in Section 4.3, I could not discern evidence that the new learning rate/initialization approach improves upon the performance of the top-performing architectures, while it appears to only boost many of the \"bad architectures.\" Given this, it raises the question of why this new learning rate/initialization strategy is relevant for practical NAS if it cannot enhance the performance of the best-found architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)