Response Modeling of Hyper-Parameters for Deep Convolutional Neural Networks
Authors: Mathieu Tuli
Mahdi S. Hosseini
Konstantinos N Plataniotis
Venue:  (2021)
Keywords: Hyper-Parameter Optimization
Response Surface Modeling
Convolution Neural Network
Low-Rank Factorization

Abstract: Hyper-parameter optimization (HPO) is critical in training high performing Deep Neural Networks (DNN). Current methodologies fail to define an analytical response surface and remain a training bottleneck due to their use of additional internal hyper-parameters and lengthy evaluation cycles. We demonstrate that the low-rank factorization of the convolution weights of intermediate layers of a CNN can define an analytical response surface. We quantify how this surface acts as an auxiliary to optimizing training metrics. We introduce a dynamic tracking algorithm -- autoHyper -- that performs HPO on the order of hours for various datasets including ImageNet and requires no manual tuning. Our method -- using a single RTX2080Ti -- is able to select a learning rate within 59 hours for AdaM on ResNet34 applied to ImageNet and improves in testing accuracy by 4.93% over the default learning rate. In contrast to previous methods, we empirically prove that our algorithm and response surface generalize well across model, optimizer, and dataset selection removing the need for extensive domain knowledge to achieve high levels of performance.

Under review as a conference paper at ICLR 2021
RESPONSE MODELING OF HYPER-PARAMETERS FOR DEEP CONVOLUTIONAL NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Hyper-parameter optimization (HPO) is critical in training high performing Deep Neural Networks (DNN). Current methodologies fail to define an analytical re- sponse surface (Bergstra Bengio 2012) and remain a training bottleneck due to their use of additional internal hyper-parameters and lengthy manual evaluation cycles. We demonstrate that the low-rank factorization of the convolution weights of intermediate layers of CNN can define an analytical response surface. We quantify how this surface acts as an auxiliary to optimizing training metrics. We introduce fully autonomous dynamic tracking algorithm autoHyper- that per- forms HPO on the order of hours for various datasets including ImageNet and re- quires no manual intervention RTX2080Ti is able to select a learning rate within 59 hours for AdaM (Kingma Ba 2014) on ResNet34 applied to ImageNet and improves in top-1 test accu- racy by 4.93 % over the default learning rate. In contrast to previous methods, we empirically prove that our algorithm and response surface generalize well across model, optimizer, and dataset selection removing the need for extensive domain knowledge to achieve high levels of performance.
priori knowledge. Our method using single
INTRODUCTION
The choice of Hyper-Parameters (HP) - such as initial learning rate, batch size, and weight decay has shown greatly impact the generalization performance of Deep Neural Network (DNN) training (Keskar et al.) 2017] Wilson et al., 2017 Li et al., 2019 Yu & Zhu 2020). By increasing the com- plexity of network architectures (from high to low parameterized models) and training datasets (class number and samples), the manual intervention to tune these parameters for optimization becomes a practically expensive and highly challenging task. Therefore, the problem of Hyper- Parrmmear Optimization (HPO) becomes central to developing highly efficient training workflows.
Recent studies shift the gear toward development of meaningful metric measure explain effective HP tuning for DNN training. This is done in several behavioural studies. including changes in loss surfaces (Keskar et al. 2017). input perturbation analysis (Novak et al. 2018), and the energy norm of the covariance of gradients (Jastrzebski et al. (020)). just to name few. In fact, the abstract formulation of the HPO problem, as highlighted by Bergstra & Bengio (2012), can be modelled by
X arg min EZ [[(;; (x(((ina))) }}} XEA
where, (itain) and x are random variables, modelled by some natural distribution M, that represent the train and validation data, respectively, L(-) is some expected loss, and Ax (main) is learning algorithm that maps (tmin) to some learned function, conditioned on the hyper-parametts set X. Note that this learned function, denoted as f(0; X; ((rain),, involves its own inner optimization problem. The HPO in TT) highlights two optimization problems which optimization over À cannot occur until optimization over (0; X; X (train)) is complete. This fact applies heavy computational burden for HPO. Bergstra & Bengio (2012) reduce this burden by attempting solve the following
arg min aigmin T()), xEA
where 7 is called the hyper-parameter response function or response surface, and is some set of choices for X (i.e. the search space). The goal of the response surface is introduce auxiliary

---

Under review as a conference paper at ICLR 2021
function parameterized by À of which its minimization is directly correlated to minimization of the objective function f().. Little advancements in an analytical model of the response surface has led to estimating it by (a) running multiple trials of different HP configurations (e.g. grid searching), using evaluation against validation sets as an estimate to r; or (b) characterizing the distribution model of configuration's performance metric (e.g. cross-validation performances) to numerically define a relationship between T and X.
An important shift occurred when Bergstra & Bengio (202)) showed that random searching is more efficient to grid searching, particularly when optimizing nigh-dimensional HP sets. To mitigate the time complexity and increase overall performance, subsequent methods attempted character- ize the distribution model for such random configurations (Snock et al.] 2012: Eggensperger et al.
2013 Feurer et al. 2015a bJ Klein et al. 2017 Falkner et al., 2018) or employed population control Young et al.] 2015 Jaderberg et al.j 2017) 7) or early-stopping (Karnin et al. 2013 Li et al.) 2017
2018]. However, these methods suffer from (a) additional internal HPs that require manual tuning
facilitated by extensive domain knowledge; (b) heavy computational overhead whereby the opti-
mization process takes days to weeks most cases (Li al. 2017 Falkner et al. 2018 Yu & Zhu
2020); (c) poor generalization across model selection, datasets, and general experimental configura- tions (e.g. optimizers); and (d) strong dependence on a manually defined search ranges that heavily influences results (Choi al. 2020 Siiaaraaadd et al. 2020 Importantly, these ranges are generally chosen based on intuition, expert domain knowledge, or some form of a priori knowledge.
In this paper, employ the notion of knowledge gain (Hosseini & Plataniotis 2020) model response surface - solvable with low computational overhead - and that does not require any priori knowledge while still achieving competitive performance against baselines and existing state of the art (SOTA) methods. Our goal is therefore to develop an algo- rithm that is fully autonomous and domain independent that can achieve competitive performance (not necessarily superior performance). We restrict our response surface to consider single HP, namely the initial learning rate '/, and support this choice by noting that the initial learning rate is the most sensitive and important HP towards final model performance (Goodfellow et al., 2016] Bergstra& & Bengio, 2012) Yu & Zhu 2020) (see also Figure I0 in Appendix Cj. We demonstrate how our method's optimization directly correlates to optimizing model performance. Finally, we provide empirical measures of the computational requirements of our algorithm and present thor- ough experiments on diverse set of Convolutional Neural Network (CNN) and Computer Vision
and use it to perform automatic HPO
dataset that demonstrate the generalization of our response surface.
The main contributions of this work
follows:
1. Inspired by knowledge gain, we introduce a well-defined, analytical response surface using the low-rank factorizatii of convolution weights (Equation 5).
2. We propose a dynamic tracking algorithm of low computational overhead on the order of minutes and hours, dubbed autoHyper, to optimize our response surface and conduct HPO.
3. This algorithm requires no domain knowledge, human intuition, or manual intervention, and is not bound by a manually set searching space, allowing for completely automatic setting of the initial learning rate; novelty for deep learning practitioners.
1.1 RELATED WORKS
We leave extensive analysis of the related works to established surveys (Luo) 2016 He et al. 2019 Yu & Zhu 2020) but present general overview here. Grid searching and manual tuning techniques that require extensive domain knowledge trial various configurations and retain the best. Random search (Bergstra Bengio 2012) was proven be more efficient, particularly in high-ddmensional cases, but these methods suffer from redundancy and high computational overhead. Bayesian opti- mization (Snoek et al. 2012, Eggensperger et al.) 2013 Feurer et al. 2015a by Klein et al., (017) techniques attempt to characterize the distribution model of the random HP configurations. They fail to properly define the response surface T and resolve to estimating it by rationalizing a Gaussian process over sampling points. The use of neural networks over Gaussian to model the generalization performance was shown to have better computational performance (Snoek et al. 2015] Springen- berg et al.) 2016). Furthermore, the early stopping methods (Karnin et al. 2013 Li et al. 2017; 2018) spawn various configurations with equal resource distributions, successively stopping poor- performing configurations and reassigning resources dynamically. Population- based training (PBT)
2

---

Under review as a conference paper at ICLR 2021
methods (Young al. 2015 Jaderberg et al., 2017) follow an evolutionary approach by spawn- ing various experimental configurations and adapting poor-performing trials to warm restart with inherited learnable parameters and HPs. In addition, other methods such orthogonal array tun- ing (Zhang et al. 2019), box-constrained derivative-free optimization (Diaz et al. 2017). reverse dynamics algorithm for SGD optimization (Maclaurin et al.] 2013), and hybrid methods (Swersky et al., [013] 2014; Domhan et al. 2015; Falkner et al. 2018 Kandasamy et al. 2016) exist but demonstrate no significant benefits over the previous techniques. Generally, each of these methods suffer from high computational overheads on the order of days to weeks to converge as well as additional internal HPs that heavily influence performance and generalization. In recent years, many Python libraries have also been developed that include these optimization methods (Bergstra et al. 2013 Kotthoff et al., 2017 Akiba et 2019).
A NEW RESPONSE SURFACE MODEL
In this section, we motivate and develop a new response surface model T()) based on the low-rank factorization of convolutional weights in CNN. Unlike the common approach of cross-validation performance measures, we define a new measure on the well-posedness of the intermediate layers of
CNN and relate this measure to the general performance of the network. We first start by adopting the low-rank measure convolution weights.
2.1 KNOWLEDGE GAIN VIA LOW-RANK FACTORIZATION
Consider a four-way array (4-D tensor) W RMXxNxNNNNNN the convolution weights of an intermediate layer of CNN (N1 and N2 being the height and width of kernel size, and N3 and N4 the input and output channel size, respectively). Under the convolution operation, the input feature maps F1 E IR Hx are mapped to an arbitrary output feature map F E IR W www.X.. N: by
Na
FO SA
FL SiAg
* W..ikssk
W (4-D Tensor)
unfold
Wa (2-D Matrix)
factorize then decompose
ÚLE.VI +Ea
We note the importance of factorizing the unfolded matrix Wa using a low-rank factorization (we use the Variational Bayesian Matrix Factorization (VBMF) Nakajima et al. 2013 Without this factorization, the presence of noise will inhibit proper analysis. This noise Ea will "capture" the randomness of initialization and ignoring it will allow us to better analyze our unfolded matrices and make our response surface robust to initialization method.
Following the definition of Knowledge Gain (KG) from Hosseini & Plataniotis 2020 one can now define metric for each network layer using the norm energy of the low-rank factorization
Ga Wa)
o d)
Na . 01 (Wa)
where, 01 2 02 ... N. are the associated low-rank singular values in descending order. Here Na = rank ( Wa} and the unfolding can be done in either input or output channels i.e. d E (3, 4). For more information on KG as well as its efficient algorithmic computation, we refer the reader to Hosseini & Plataniotis 2020)).
The metric defined in 3 is normalized such that Ga [0, 1] and can be used to probe CNN layers to monitor their efficiency in the carriage information from input to output feature maps. We can further parameterize the KG by the HP set X, epoch t, and network layer as (A). A perfect network and set of HPs would yield (A) = VÊ E [L] where L is the number of layers in the network and T is the last epoch. In this case, network layer functions as a better autoencoder through iterative training and the carriage of information throughout the network is maximized. Conversely, (AA).e(A) = 0 indicates that the information flow is very weak such that the mapping is effectively random (|) Ex is maximized.).
3

---

Under review as a conference paper at ICLR 2021
2.2 DEFINITION OF NEW RESPONSE FUNCTION
Interestingly, if Ga...(A) 0 in early stages of training, it is evidence that no learning has occurred, indicative of initial learning rate that is small (no progress has been made to reduce the randomization). It then becomes useful to track the zero-valued KGs within a network's s intermediate layers' input and output channels, which effectively becomes measure of channel rank. We denote this rank per epoch as follows:
1 Σ Σ 1 (a))
2(1)
2L
[E[L] (E(3,4)
where (A) E [0, Finally, we define the average rank across I epochs as
Σ (()). 2(A) + T LE(T]
Note that (()) [0, 1). The average rank measure in is therefore normalized summation of the vero-valued singular values of the low-rank factorization across all layers' input and output unfolded tensor arrays.
Relating to the notion of HPO and the response surface, we return to and ( Where previously the nature of these two optimization problems was poorly understood practically unsolvable, we propose a new problem that is well understood and practically solvable (on the computational order of hours). To solve for the optimal HP set A, we look at the following optimization problem
f arg min - (A),, subject to
VA) 11 S
(5)
where c E [0, 1) some small conditioning error. Returning equation 2 our response surface therefore defined our problem to only consider À y). Also, we do not explicitly calculate the gradient 2 but rather use this constraint to guide our dynamic tracking algorithm (see (section 3). To explain this somewhat counterintuitive formulation, we analyze Figures I(a] & which demonstrate that as learning rates increase, (1) plateaus to zero. Specifically, we notice that optimal learning rates lie towards the inception of the plateau of (1)), before ((n) = 0. This can also be seen Figures wish to design response surface such that the solution
1 2(A) subject to V^ 1|35 e. Note that we now simplify
& in Appendix A] Therefore, lies at the inception of the plateau-ing region (see the red dot in Figure
((a) In this case, our constraint || (()(A) || ك e promotes learning rates that lie along this plateau-ing region, while arg min 1 - (()) promotes, of those learning rates in the plateau-ing region, learning rate that
X
lies towards the inception of this plateau-ing region.
Top-1 Accuracy
1.0
Probability bibiiitity Density
2.8
Training Loss
1.4
0.06
0.7
1.2
0.04
0.4
0.8
0.2
C.02
0.6
4
0.0
0.2
0.4
D.6
0.8
Epoch (t)
Epoch - (0)
Learning Rate (n)
Z(n)
(a) Auxiliary Behaviour
(b) Histogram of (
Figure 1: (a) Auxiliary representation of 2(7) to training loss and training accuracy. The red dot indicates the learning rate chosen by method with corresponding metrics drawn in red lines. Lines and scatter points in grayscale show various trialled learning rates. (b) Distribution of 2(7) taken from the searching phase of autoHyper over numerous experimental configurations applied on CIFAR10, 100, TinylmageNet, and ImageNet (see subsection 4.1)
More generally, high Z()) indicates learning rate that is too small, intermediate layers do not make sufficient progress in early stages of learning and therefore their KGs remain very low. This observation follows that of Li et al. (2019) in which larger initial learning rates result in better
4

---

Under review as a conference paper at ICLR 2021
generalization performance. Promotion of these larger learning rates is therefore achieved by gradient constraint in Equation Conversely, too large of learning rate can over-regulate network and, therefore, wish not to minimize (nn completely but tune to be sufficiently small to arrive at the inception of its plateau, creating sort of kick-start if you will. This is achieved by the balance between our minimization and constraint in [Equation
16
2,0
8
Z4ni
1099 comn c309 gost ooet cama Learning (n)
Epoch
Learning Rate (n)
suen ot ose
Epech
Learning Rate (n)
Epoch
(a) AdaBound
(b) AdaGrad
(c) AdaM
Figure 2: E(n) for various learning rates using AdaBound, AdaGrad, and AdAM on ResNet34 applied to CIFARIO. The suggeted initial learning rate is indicated by the red markers, and the autoHyper suggested learning rate is indicated by the green markers.
Finally, we choose T = 5 in our experiment. The early phase of training has been shown to be an important criterion in optimal model performance (Jastrzebski et al.) 2020 Li et al. 2019) and we therefore wish to only consider 5-epochs to ensure optimization within this phase. Additionally, Figure and Figures & Appendix tell that (1)) stabilizes after 5 epochs.
2.3 EMPIRICAL EVALUATION OF NEW RESPONSE MODEL.
Figure [(a) visualizes results of our method on ResNet34 on CIFAR10 optimized with AdaM. The learning rate selected by our method results in lowest training loss and highest top-1 training accu- racy over the 5-epoch range we consider. We note the importance of stopping the inception of this plateau region as even though higher learning rates, highlighted by the gray-scale lines/markers, result in potentially lower 2(1), they do not guarantee lower training losses or higher training accu- racies. We conclude that response surface is strong auxiliary to training loss, and optimizing HPs relative to our response surface will fact optimize towards training loss and accuracy.
various experimental configurations (see subsection 4.1). Note the presence of a multimodal distribution that peaks at low 2 but impor- tantly not zero. This visualizes our method's tendency to converge to a consistent range of values for irrespective of experimental configuration, showing the generalization of our response surface.
Figure ((b) displays the histogram of 2(1) values
3 AUTOHYPER: AUTOMATIC TUNING OF INITIAL LEARNING RATE
The pseudo-code for autoHyper is presented in Algorithm Analyzing equation 5 we state that the optimal solution lies within the inception of the plateauing region of ( To find this region, autoHyper first initializes logarithmic grid space, from Mmin = x 10 to Mmar 0.1 of S = 20 step sizes, denoted by plateau is reached. Once plateau is reached, Ω is reset such that Mmin and Maar "zoom" towards the learning rates at the plateau. This process is repeated recursively until no significant difference between Mmin and Jimaz exists. On average, this recursion occurs 3 to 4 times and as shown in Figure 3. the number of trialled learning rates remains very low (between 10-30 on average). Importantly, algorithm is not constrained by its initial grid space. As it tracks 2(n) over learning rates, it may grow and shrink its search bounds dynamically. This permits our method to be fully autonomous and require human intuition in setting of the initial grid space.
iterates through each 7) E Ω E {0..... 19), and computes 2(n). until
5

---

Under review as a conference paper at ICLR 2021
Algorithm autoHyper
Require: grid space function v, learning significant difference 10 and change I: procedure RESTART()
learning rate index
3: 42 (immen Tmaas S)
4: end procedure
RESTART()
6: while Truc de
i [S2] then
search space plateau found yet increase Tomu and RESTART()
8:
9:
end
10:
Tmaa Mmin then // limits of search space sigmificanlly different
11:
return Mmaa
12:
13:
with R. train for 5 spochs
compute (rpa) per equation
15:
((n))= then If aal KG acro-valued, increuse Tmiin and RESTART() end
small
16:
17:
18:
if O and Z(m) < 0.5 and this is the first un then // initial Yhuaee is too large
19:
roduce Mmon RESTART()
20:
else
[(mr)
zero, search further, perform "zoom" fh Yimaz fr and RESTART()
22:
23:
end if
24:
compute change of Z(m)): (((Z(n)).... 2(n)))
25:
if rate of change platemus then // perform "zoom"
Mmm f,-1, Mmaa
RESTART()
27:
end if
29:
+
30:
end while
AdaBoound
AdaGGrd
Adalt
AddS
Aded
P.8.55
Densekee12 21
Reehhee B
Bedhwar10
Dereehie12 121
Recker'
Becker3
Figure 3: Computational analysis of autoHyper over various setups (number of learning rates au- toHyper trialled before converging). ResNet34 trials take 3 minutes, 3 minutes, 18 minutes, and 220 minutes for CIFARIO, CIFAR100, TinyImageNet and, ImageNet, respectively. ResNet18, ResNeXt50, and DenseNet121 trials take 2 minutes, minutes, and 3 minutes respectively for both CIFARIO and CIFAR100.
We note here that the choice of & and Ç mentioned in Algorithm 3 (i.e. grid space and rate of change functions, respectively) will have significant affect the final generated learning rate. We make use of numpy's geomspace function for the logarithmic grid spacing, and calculate the rate of change in 2(r) by taking the cumulative product of the sequence of Z(n).. to the power of 0.8. A logarithmic grid space is used as our response surface more sensitive to smaller learning rates (see Figure 2 Note that initial grid bounds are not important as our algorithm can shift those bounds dynamically, however the successive increments between learning rates in the grid must be sufficiently small (on the order of 1x 10 as in our initialization). Since our response surface itself not guaranteed to monotonically decrease, as shown Figure 4bb we employ the cumulative product of 2(nk) (as rate of change function), which is monotonically decreasing function
2(m)
z(n)
umw
5300 wwwraaaaae car caranann
55 19 MN ure nis wis
Learning Rate - (n)
Learning Rate (n)
(a) ResNet34/Adam
(b) EANeeBB/Adaa 0.8
Figure 4: 2(r) (blue) vs. cumprod( (n)) (or- ange) for (a) a stable and (b) an unstable architec- ture on CIFAR10.

---

Under review as a conference paper at ICLR 2021
since 2(n) E [0.1) and is therefore always guaranteed to converge. The cumulative product (to the power of 0.8) is a good choice because it (a) is always guaranteed to plateau (since 0 S Z(m:) < 1), which removes the need for some manually tuned threshold and (b) because it dampens noise well. Because the cumulative product on its own degrades to zero rather quickly in many scenarios, raising it the power of 0.8 regulates this effect. This power is technically tune-able, however we show empirically in Figure 4(() and [4(b] that 0.8 behaves well for both stable and unstable architectures. Refer Figure 9 in Appendix C for the performance results of EfficientNetBO.
4 EXPERIMENTS
In this section, we conduct an ablative study of our algorithm autoHyper and response surface on various network architectures trained using various optimizers and applied to image classification datasets. We also compare autoHyper against existing SOTA; Random Search.
4.1 EXPERIMENTAL SETUPS
Ablative study. All experiments are run using an RTX2080Ti, 3 cores of an Intel Xeon Gold 6246 processor, and 64 gigabytes of RAM. our ablative study, we run experiments on CIFARIO
Krizhevsky et al. 2009). CIFAR100 (Krizhevsky et al.) 2009), TinyImageNet (Li et al.). and Im- ageNet (Russakovsky et al. 2015). On CIFARIO and CIFARI00, we apply ResNet18 (He et al. 2015), ResNer34 (He et al.) 2015). ResNeXt50 (Xie et al. (016)) and DenseNet121 (Huang et al.
2017 J. On TinylmageNee and ImageNet, we apply ResNet34. For architectures applied to CIFARIO
and CIFAR100, we train using AdaM (Kingma& & Ba 2014), AdaBound (Luo et al. 2019), AdaGrad
Duchi et al. 2011). RMSProp (Tieleman & Hinton 2012). AdaS = (0.8, 0.96 0.975)) (Hosseini
& Plataniotis 2020) (with early-stop), and SLS (Vaswani et al. 2019). For ResNet34 applied to TinylmageNet, we train using AdaM, AdaBound, AdaGrad, and AdaS (0.8, 0.9, 0.95, 0.975)), For
train using AdaM and Ada {0.9, 0.95, 0.975)), Note that the in AdaS- is known as the gain factor and trades between performance and convergence rate: a low 3 converges faster but the cost of performance and vice-versa. For each experimental setup we ran one training sequence using suggested learning rates (baseline) and one training sequence using learning rates generated by autoHyper (see Tables I 4 in (Appendix B). Refer (Apendix for additional details on the ablative study.
ResNet34 applied to ImageNet,
Comparison to Random Search. Because Random Search generally requires iterative manual refinement and is highly sensitive to the manually set search space (Choi et al. 2020 Sivaprasad
al, 2020). we attempt a fair comparison by providing the same initial search space that autoHyper starts with, and allow for the same number of trials that autoHyper takes (see Figure 3. We note however that this does provide the Random Search with a slight advantage since a priori knowledge of how many trials consider is not provided to autoHyper. See Appendix D for additional detail.
4.2 RESULTS
Consistent performance across architectures, datasets, and optimizers. We visualize the pri- mary results of each experiment in Figure 5(a) (additional results are shown in Figure [] in Ap- pendix C.. From these figures see how well our method generalizes experimental configura- tions by noting the consistency in top-1 test accuracies when training using the autoHyper generated initial learning rate vs. the baseline. Further, we note that if there is loss of performance when using an initial learning rate generated by autoHyper, we identify that this loss is < 1% in all exper- iments except three: On CIFAR 100, the baselines of Rt50 trained using AdaM, ResNext50 trained using RMSProp, and DenseNet121 trained using AdaBound achieve 1.2 1.2%, 2.28% and 1.9% better top-1 test accuracy, respectively. We note importantly however that when accounting for the standard deviation of each of these results, only the DenseNet121 12 experiement maintains its > 1% improvement. Refer to Apppeddix (Tables) 5 g to see the tabulated results of each experiment. We also importantly highlight how auto is able to generalize across experimental setup whereas Random Search cannot (see Figure 5(b).. Because Random Search (and other SOTA methods) de- pend heavily on their manually defined parameters such epoch budget or initial search space,
7

---

Under review as a conference paper at ICLR 2021
Adelouun Ada/ounnd
Adwlrred Adwrrrdd
Adwlt Ada/e:
Aded 0---p---22
Aded 6--0.-
Aced P- 1..5
204.5--0..75
SL5
Ad15 0
AG05 8 0.3"
Ac0S f 0.95*
AdaS p 9-0.9755
RMESPro"
S5
0.45
0.05
0:34
0:44
Trp Test ACCUreSY
Ro. Tss Accuracy
0.31
0:43
0.53
0032
Text
0.3
0.53
0.40
0.00
0.93
0.09
1.00
0.85
0.55
107
Taining Lo35
10
beiting Loxe
Taining Loss
Tennn
20m4
18
30
100
130
200
250
20
100
130
270
250
300
150
200
250
32
100
130
200
230
Epacch
Epoch
Spoch
0.14
100 Test ACCU Accurscy
5.76
0.74
0.72
00.2
Bo- Test
0,72
0.:8
0.00
0.64
RosMet18 CIFAR100
RosNo034 CIFARICO
ResheHt50 on CIRAR100
DonsetNet123 CIRARLOC
0.52
0.725
Top-1 Train Accuracy
Tap-1 Irain Accuracy
0.722
ltp-1 lest Accurscy
0.70
0.635
0.51
0.555
0.32
0.625
0.66
0.5
0.02
30
300
350
202
250
30
100
150
210
253
100
325
359
25
30
15
100
150
Epoch
Epoch
Epoth
Eguch
Restiet34 Tinylmagetiet
ResNet34 Tinylmagetiet
ResNet34 ImageNet
ResNet34 ImageNet
(a) Ablative Study
0.02
8.25
0.76
300 1e5t Accurac
0:30
kpp1 Accuracc
0.53
0.200
00:07
0054
0:51
0.70
aggehet
CIFAR100
(b) Random Search Comparison
Figure 5: Results of the (a) ablative study and (b) Random Search comparison experiments. Titles below plots indicate what experiment the above plots refers to. Legend labels marked by *** (solid lines) show results for autoHyper generated learning rates and dotted lines are the (a) basclines and (b) Random Search results.
8

---

Under review as a conference paper at ICLR 2021
generalization to experimental setup is not feasible, demonstrated here. In contrast, we have shown that autoHyper is perfectly capable of performing well no matter the experimental setting without need for manual intervnnion/reeinmmenn of any kind; a novelty.
Fully autonomous discovery of optimal learning rates. Importantly, we highlight how our method is able to fully autonomously tune the initial learning and achieve very competitive performance. Whereas traditional HPO methods (like Random Search) are extremely sensitive to initialization of the search space, which would normally require extensive domain or a priori knowledge to set, our method is not: given a new dataset, model, and/or other hyper-parameter configurations, a practi- tioner could use simply call our algorithm to automatically set very competitive initial learning rate. If truly superior performance is required, one could perform more extensive HPO around the autoHyyersuggested learning rate, removing the need to perform iterative manual refinement.
Superior performance over existing SOTA, As visualized in Figure 13 although Random Search proves competitive for Adabound and AdaM applied on CIFARIO and CIFARI00, it cannot find competitive learning rate for AdaS = 0.9 or AdaGrad and performs worse for AdaM applied on TinyImageNet AdaGrad applied on TinyImageNer loses as much as 4% top-1 test accuracy. This highlights how ayto can automatically find more competitive learning rates to a Random Search given the same computational budget, and with significantly less manual intervention. These results additionally highlight why validation loss (or accuracy) cannot be used as a substitute to our metric (see Figure 14 in subsection D.2 for additional discussion).
Drastic improvements in AdaM applied to TinyImageNet and ImageNet. ResNet34 trained using AdaM and applied to TinylmageNet and ImageNet achieves final improvements of 3.14% and 4.93% in top-1 test accuracy, respectively (see Table 5 in Appendix Cj. Such improvements come at a minimal cost using our method, requiring 13 trials (4 hours) and 16 trials (59 hours) for TinylmageNet and ImageNet, respectively (see Figure B)
Extremely fast and consistent convergence rates. We visualize the convergence rates of our method in Figure 3 Importantly, we identify the consistency of required trials per optimizer across architecture and dataset selection as well as the low convergence times. We identify that the longest convergence time for our method is on ResNet34 trained using AdaS 0.95 applied to ImageNet, which took 31 trials and total of 114 hours. We note that our method exhibits less consistent results when optimizing using SLS as SLS tends to result high (1) over multiple epochs and different learning rates. Despite this, our model still converges and results in competitive performance.
Performance improvement over increased epoch budgets. In reference to Table in Appendix Cj we highlight how, of the 29 experimental configurations, when trained using the initial learning rate suggested by autoHyper, only 12 of them outperform the baseline. However, as training progresses, note that by the end of the fixed epoch budget, 18 of the 29 experiments trained using the initial learning rate suggested by autoHyper outperform the baselines. Further, in many of the cases where baselines perform better, they remain within the standard deviation of trials, and are therefore not significantly better. These results are surprising as our goal with this method was to achieve competitive results in tuning the initial learning rate however, in more than half the cases, our method results in increased performance at a significantly smaller computational cost.
CONCLUSION
In this work we proposed an analytical response surface that acts as auxiliary to training metrics and generalizes well. We proposed an algorithm, autoHyper, that solves this surface and quickly gen- erates learning rates that are competitive to author-suggested and Random Search-suggested values- in some cases, even drasti- cally superior. We therefore have introduced an algorithm that can perform HPO fully autonomously and extremely efficiently, and resolves many of the drawbacks of current SOTA. Figure O vi- sualizes our response surface over a multi-dimension HP set and highlights how our response surface remains solvable. We iden- tify that autoHyper could be adapted to simultaneously optimize multiple HPs by tracking tangents across this surface towards the minimum, but leave this future work.
Figure 6: 2 7) for learning rate (7)) and mini-batch size (7).
9

---

Under review as a conference paper at ICLR 2021
REFERENCES
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Pp. 2623-2631, 2019.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optmmiiiaaiiiiiiiiiiiiii The Journal of Machine Learning Research, 13(1):281-305, 2012.
James Bergstra, Dan Yamins, and David D Cox. Hyperopt: A python library for optimizing the hyperparameters of machine learning algorithms. In Proceedings of the 12th Python in science conference, volume 13, pp. 13-19. Citeseer, 2013.
Dami Choi, Christopher J. Shallue, Zachary Nado, Jachoon Lee, Chris J. Maddison, and George E. Dahl. On empirical comparisons of optimizers for deep learning, 2020.
Gonzalo Diaz, Achille Fokoue-Nkoutche, Giacomo Nannicini, and Horst Samulowitz. An effective algorithm for hyperparameter optimization of neural networks. IBM Journal of Research and Development, 61(4))):--, 2017.
Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. Speeding up automatic hyperparam- eter optimization of deep neural networks by extrapolation of learning curves. In twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.
John Duchi, Flad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
Katharina Eggesseererr Matthiss Feurer, Frank Hutter, James Bergstra, Jasper Snock, Holger Hoos, and Kevin Leyton-Brown Towards an empirical foundation for assessing bayesian optimization of hyperparameters. In NIPS workshop on Bayesian Optimization in Theory and Practice, vol- ume 10, pp. 2013.
Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter opti- mization at scale, 2018.
Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenherg, Manuel Blum, and Frank Hutter. Efficient and robust automated machine learning. In Advances in neural information processing systems, pp. 2962-2970, 2015a.
Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. Initializing bayesian hyperparame- ter optimization via meta-learning In Twenty-Nnth AAAI Conference on Artificial Intelligence, 2015b.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, Cambridge, MA, USA, 2016. httpp://wwy http::/wwwwweppeeaanngbboooooo org
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition, 2015.
Xin He, Kaiyong Zhao, and Xiaowen Chu. Automl: A survey of the state-of-the-an arXiv preprint arXiv:1908.00700 2019.
Mahdi S. Hosseini and Konstantinos N. Plataniotis. Adas: Adaptive scheduling of stochastic gradi- ents, 2020.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojeiech M Czamecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning. Karen Simonyan, et al. Population based train- ing of neural networks. arXiv preprint arXiv: 171 1.0984,, 2017.
10

---

Under review as a conference paper at ICLR 2021
Stanislaw Jastrzebski, Maciej Szymezak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho and Krzysztof Geras The break-even point on optimization trajectories of deep neural networks. In International Conference Learning Representations, 2020. URL https:// epenreview.nee wwwee/eo/e/riiioeee wwwrrrmmidirlrr 7C4KWB
Kirthevasan Kandasamy, Gautam Dasarathy, Junier B Oliva, Jeff Schneider, and Barnabás Póczos. Gaussian process bandit optimisation with multi-fidelity evaluations. In Advances in Neural In- formation Processing Systems, pp. 992-1000, 2016.
Zohar Karnin, Tomer Koren, and Oren Somekh. Almost optimal exploration in multi-armed bandits. In International Conference on Machine Learning, pp. 1238-1222 1246, 2013.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations, 2017. URL
net/
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014.
Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, and Frank Hutter. Fast bayesian op- timization of machine learning hyperparameters on large datasets. In Artificial Intelligence and Statistics, pp. 528 536, 2017.
Lars Kotthoff, Chris Thornton, Holger H Hoos, Frank Hutter, and Kevin Leyton- Browon Auto- weka 2.0: Automatic model selection and hyperparameter optimization in weka. The Journal of Machine Learning Research, 18(1):826-830, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Fei-Fei Li, Andrej Karpathy, and Justin Johnson. herokuapp.com
URL https://ting -amoyyymmgenee...
Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. Massively parallel hyperparameter tuning. arXiv preprint 1XXi:18100.0934, 2018.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit--ased approach to hyperparameter optimization. The Journal of Machine Learning Research, 18(1):6765-68 6816,1666666666666611 2017.
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. In Advances in Neural Information Processing Systems, pp. 11674-1 1685, 2019.
Gang Luo. review of automatic selection methods for machine learning algorithms and hyper- parameter values. Network Modeling Analysis in Health Informatics and Bioinformatics, 5(1):18, 2016.
Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate, 2019.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient- hyperparameter optimiza- tion through reversible learning. In International Conference on Machine Learning, pp. 2113- 2122, 2015.
Shinichi Nakajima, Masashi Sugiyama, S Derin Babacan, and Ryota Tomioka. Global analytic solution of fully-observed variational bayesian matrix factorization. Journal of Machine Learning Research, 14(Jan):1-37, 2013.
Roman Novak, Yasaman Bahri, Daniel A. Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Sensitivity and generalization in neural networks: empirical study. In International Confer- ence on Learning Representations, 2018. URL httpp:://penzeeview net/ forum?id
HJC2SZZCC

---

Under review as a conference paper at ICLR 2021
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeey Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Viston (IJV), 1 11(())211-252, 2015. doi: 10. 1007// 1263-015-0816-y.
Prabhu Teja Sivaprasad, Florian Mai, Thijs Vogels, Martin Jaggi, and François Fleuret. Optimizer benchmarking needs to account for hyperparameter tuning, 2020.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In Advances in neural information processing systems, pp. 2951-2959, 2012.
Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable bayesian optimization using deep neural networks. International conference on machine learning, pp. 2171-2180, 2015.
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimization with robust bayesian neural networks. In Advances in neural information processing systems, pp. 4134 4142, 2016.
Kevin Swersky, Jasper Snoek, and Ryan Adams. Multi-task bayesian optimization. In Advances in neural information processing systems, pp. 2004-2012, 2013.
Kevin Swersky, Jasper Snoek, and Ryan Prescott Adams. Freeze-thaw bayesian optimization. arXiv preprint 4rXiv:140.33896 2014.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning. 4(2):26- 31, 2012.
Sharan Vaswani, Aaron Mishkin, Issam H. Laradji, Mark Schmidt, Gauthier Gidel, and Simon Lacoste-Julien. Painless stochastic gradient: Interpolation, line-search, and convergence rates. CoRR, hiss/105500997, 2019. URL
09997
Ashia C. Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. 2017.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks, 2016.
Steven R Young, Derek C Rose, Thomas P Karnowski, Seung- Hwan Lim, and Robert M Patton. Optimizing deep learning hyper-parameters through an evolutionary algorithm. In Proceedings of the Workshop on Machine Learning in fighh-Pefformacce Computing Environments, pp. 4. ACM, 2015.
Tong Yu and Hong Zhu. Hyper- parametr optimization: review algorithms and applications. arXiv preprint arXiv:2003.05689 2020.
Xiang Zhang, Xiaocong Chen, Lina Yao, Chang Ge, and Manqing Dong- Deep neural network hyperparameter optimization with orthogonal array tuning. In International Conference on Neural Information Processing, pp. 287-295. Springer, 2019.
12

---

Under review as a conference paper at ICLR 2021
A RANK BEHAVIOUR OVER MULTIPLE EPOCHS
(a) AdaM
(b) AdaGrad
(c) AdaS 8 0.8
(d) RMSProp
Figure 7: Rank (2(1)) for various learning rates VGG16 trained using AdaM, AdaGrad, AdaS 8 = 0.8, and RMSProp and applied to CIFARIO. A fixed epoch budget of 20 was used. We highlight how across these 20 epochs, very little progress is made beyond the first first epochs. It is from this analysis that we choose our epoch range of T 5.
(a) AdaM
(b) AdaGrad
(c) AdaS 3 0.8
(d) RMSProp
Figure 8: (Z(n)) for various learning rates ResNet34 trained using AdaM, AdaGrad, AdaS B 0.8, and RMSProp and applied to CIFAR1O. A fixed epoch budget of 20 was used. We highlight how across these 20 epochs, very little progress is made beyond the first first epochs. It is from this analysis that we choose our epoch range of T = 5.
B ADDITIONAL EXPERIMENTAL DETAILS FOR SUBSECTION 4.
We note the additional configurations for experimental setups.
Datasets: For CIFARIO and CIFAR100, we perform random cropping to 32 x 32 and random horizontal flipping the training images and make alterations the test set. For TinylmageNet,
13

---

Under review as a conference paper at ICLR 2021
perform random resized cropping to 64 x 64 and random horizontal flipping the training
images and center crop resizing to 64 x 64 on the test set. For ImageNet, we follow He et al. (2015) and perform random resized cropping to 224 244 and random horizontal flipping and 256 256 resizing with 224 x 224 center cropping on the test set.
Experiments CIFARIO, CIFARI00, and TinyImageNet used mini- batch sizes of 128 and ImageNet experiments used mini-batch sizes of 256. For weight decay, 5 x 10-4 was used for AdaS-variants on CIFARIO and CIFAR experiments and 1 x 10 for all optimizers on TinylmageNet and ImageNet experiments, with the exception of AdaM using a weight decay of 7.8125 10- For AdaS-variant, the momentum rate for momentum-S3 SGD was set to 0.9. All other hyper- hyper-parameee parameters for each respective optimizer remained default as reported in their original papers. For CIFARIO and CIFAR 100, we use the manually tuned suggested learning rates reported in Wilson et al. (2017) for AdaM, RMSProp. and AdaGrad. For TinyImageNet and ImageNet, we use the suggested learning rates as reported in each optimizer's respective paper. Refer to Tables I 4 to see exactly which learning rates were used, as well as the learning rates generated by aytoHyper. CIFARIO, CIFARIO0, and TinylmageNet experiments were trained for trials with maximum of 250 epochs and ImageNet experiments were trained for 3 trials with maximum of 150 epochs. Due to AdaS' stable test accuracy behaviour as demonstrated by Hosseini & Plataniotis 2020), an early-stop criteria, monitoring testing accuracy, was used for CIFARTO, CIFAR 100, and ImageNet experiments. For CIFAR10 10 and CIFAR 100, a threshold of x 10-3 for A = 0.8 and 1 10 for AdaSs f0.9, 0.95) and patience window of 10 epochs. For ImageNet, a threshold of x 10-4 for Adas = {u.s, 0.9, 0.95} and patience window of 20 epochs. No early stop is used for A 0.975
Additional Configurations:
Learning Rates: We report every learning rate in Tables
Table I: Learning rates for ResNet34 experiments. Left inner columns show suggested, right inner columns show autoHyper generated. Note that the superscript for AdaS-variants indicates their B gain factor.
Optimizer
CIFARIO
CIFARIO0
TinylmageNet
ImagetNet
AdaM
0.0003
0.000333
0.0003
0.000241
0.001
0.0001965
0.001
0.0001965
AdaBound
0.001
0.000347
0.001
0.000347
0.001
0,0000944
AdaGrad
0.01
0.002861
0.01
0.002236
0.01
0.0022359
AdaSu.s)
0.03
0.012374
0.03
0.010401
0.03
0.010185
00)
0.03
0.012374
0.03
0.010190
0.03
0.0085857
0.02
0.011479
Ada5tu.95)
0.03
0.015336
0.03
0.015025
0.03
0.0085857
0.02
0.011479
AdaS
0.03
0.012374
0.03
L.001190
0.03
0.0085857
0.02
0.01
RMSProp
0.0003
0.000168
0.0003
0.000197
SL.
1.0
0.034491
1.0
0.034191
Table 2: Learning rates for ResNet18 experiments. Left inner columns show suggested, right inner columns show autoHyper generated.
Optimizer
CIFARIO
COF
AdaM
0.0003
0.00D6756
0.0003
0.0006756
AdaBound
0.001
0.000360410
0.001
0.000241966
AdaGrad
0.01
0.0049724
0.01
0.0049724
AdaS 0.8)
0.03
0.0126594
0.03
0.0126594
AdaS
0.03
0.0104254
0.03
0.0126594
AdaS 0.95)
0.03
0.01042544
0.03
0.010425433
Ada 0.975)
0.03
0.010425-1
0.03
0.007071
RMSProp
0.0003
0.0004697
0.0003
0.0004697
SL.S
1.0
0.03419131
1.0
0.03/11913
14

---

Under review as a conference paper at ICLR 2021
Table Learning rates for ResNeXt50 experiments. Left inner columns show suggested, right inner
columns show autoHyper generated.
Optimizer
CIFARIO
CIFARI00
AdaM
0.0003
0.0006756
0.0003
0.0006756
AdaBound
0.001
0.00036040
0.001
0.00024896
AdaGrad
0.01
0.0049724
0.01
0.0049924
AdaS
0.03
0.0126594
0.03
0.0126594
AdaŞ 0.9)
0.03
0.0104254
0.03
0.01 26512
AdaŞ 0.300)
0.03
0.02313-1
0.03
0.022666
AdaS
0.03
0.0104254
0.03
0.007771
RMSProp
0.0003
0.0001697
0.0003
0.000-1697
SLS
1.0
0.03419134
1.0
0.0341913
Table 4: Learning rates for DenseNet121 experiments. Left inner columns show suggested, right
inner columns show autoHyper generated.
Optimizer
CIFARTO
CIFARTOO
AdaM
0.0003
0.0020109
0.0003
0.00093162
AdaBound
0.001
0.0030176
0.001
0.0031
AdaGrad
0.01
0.01537206
0.01
0.01537206
AdaS = 0.8)
0.03
0.06107176
0.03
0.03978104
AdaS =
0.03
0.0598363
0.03
0.050414
AdaSio 0.93)
0.03
0.0492772
0.03
0.0357048479
AdaS 0:975)
0.03
0.0598363
0.03
0.050-1145 43
RMSProp
0.0003
0.00201093
0.0003
0.00067564
SLS
1.0
0.0031 184
1.0
0.087943
C ADDITIONAL RESULTS FOR SUBSECTION 4.2
Large deviation from the suggested initial learning rates, Referring Tables 14 notice variation in autoHyper suggested learning rates as compared to the author-suggested and Random Search-selected ones. The learning rates generated by method reveal the "blind spots" that the authors originally overlooked in their HPO. Interestingly, however, we note the similarity in initial learning for ResNet34 trained using AdaM CIFAR10, and can confirm this an optimal learning rate. Importantly, our method is significantly quicker than the grid searching technique employed by Wilson et (017)) Observations on the generalization characteristics of optimizers. Fig- ure identifies the poor generalization characteristics of AdaM, AdaBound, AdaGrad, and SLS where they consistently achieve low training losses, but do not exhibit equivalently high top-1 test accuracies. We note that these observations are similar to those made by Wilson et al. (2017); Li (et al. (2019); Jastrzebski al. (2020). We additionally contribute that AdaS does generalize well. We also highlight SLS' multi-order-of-maaniie stability of the AdaS optimizer, particularly when applied on TinylmageNG
tolerance to initial learning rate well as the
15

---

Under review as a conference paper at ICLR 2021
AdaGrad
AdaM
AdaS B
AdaS B= RMSProp
AdaGrad
AdaS B = 0.8
AdaS f 0.9
AdaM
AdaS ß 0.8
Ada5 B 0.975
RMSProp*
10
Top-1 Test Accuracy
0.90
Training Loss
0.88
10
0.86
0.84
10
100
200
100
200
Epoch
Epoch
EfficientNetB0 CIFAR10
Figure 9: Test accuracy and trianing loss for EfficientNetBO applied to CIFAR100. Importantly, EfficientNetBO is unstable network architecture in relation our response surface and yet our method, autoHyper, is still able to converge and achieve competitive performance.
StepLR
StepLR*
10c
0.95
Top-1 Test Accuracy
0.94
Training Loss
10-1
0.93
0.92
0.91
10-2
10-1
100
150
200
250
100
150
200
250
Epoch
Epoch
Figure 10: Demonstration of the importance of initial learning rate in scheduled learning rate case, for ResNet18 applied on CIFARIO, using Step-Decay method with step-size = 25 epochs and decay rate 0.5. As before, the dotted line represents the baseline results, with initial learning rate 0.1, and the solid line represents the results using autoHyper's 's suggested learning rate of 0.008585. These results highlight the importance of initial learning rate, even when using scheduled learning rate heuristie, and demonstrates parameters. Despite better initial performance from the autoHyper suggest learning rate, the step- size and decay rate choice cause the performance to plateau too early.
importance of the additional step-size and decay rate hyper-
16

---

Under review as a conference paper at ICLR 2021
AdaSound
Adadrod Adedredd
AdaM
AdS5
9-05--09
AcO5-p 10055-00.5
Ads5
Ad65 0.8
Ad65 0.5
Aced p1: 0.95
AdW5 0.975
0.20
0.25
0.76
Accumscy
T0p- Text Accues 10
0.34
0.72
5.70
0083
0.20
0.70
5.60
0.65
0:90
2.00
0.00
0.06
0.2
00
1000
200
750
1.0
1.90
300
750
180
150
700
750
100
1550
200
7.0
Epo07
Epoch
Epoch
Epoch
Talsing lowe
Talring LoGs
bellinn Laoe
Talning Locs
20n1
20m0
10
10a
Epach
Epoch
Epech
Reshiet1 on CITARIBO
RedNett3 on CIFAR1OD
Destlext50 on CIAR100
Densetleet22 on CITARLOO
2.50
Accursyy
Top-5 Accuracy
0.02
2.77
0.20
0.52
2.50
0.25
Epoch
Epoch
Epoch
ResNet34 on TinylmageNet
ResNet34 on TinytmageNet
ResNet34 on TinylmageNee
ResNet34 on TinytmageNeet
Train Accuracy
Top-3 Train Accuracy
Tost Accurscy
8:90-
3.00
0:00
2.00
2.500
2.64
8:60
1.000
2.00
0:00
Epuch
ResNet34 Imageliet
Restiet34 on ImageNett
ResNet34 Imageliet
ResNeet34 on ImageNet
Full results of CIFAR100, TinylmageNet, and ImageNet experiments. Top-1 test accu- racy and training losses are reported for CIFAR 100 experiments and top-1 and top-5 test and training accuracies are reported for TinyImageNet and ImageNet, Titles below the figures indicate to which experiments the above figures belong to. As before, lines indicated by the *** (solid lines), are results using initial learning rate as suggested by autoHyper.
Figure 11:
17

---

Under review as a conference paper at ICLR 2021
AdaBound
AdeeBound
AdwGred
Acaht
AdeS p 6.0"
AdeSS 0.0.
Ades5 9 0.95
AdeS P- D.975
0.62
0.60
Top-1 Test Accuracy
0.58
0.56
Test Loss
0.54
0.52
0.50
0.48
2 10"
100
150
200
250
50
100
150
200
250
Epoch
Epoch
Figure 12: Top-1 Test Accuracy and Test Loss for ResNet34 Experiments applied on Tiny As before, lines indicated by the *** (solid lines), are results using initial learning rate suggested by autoHyper. These results visualize the inconsistency in tracking test loss as a metric to optimize final testing accuracy. This can seen, for example, when looking the loss and test accuracy plots for AdaM, where the test loss for the baseline is lower than that of the autoHyper suggested results but autoHyper achieves better test accuracy. These results also highlight the instability of tracking testing accuracy or less instead of the metric defined in Equation
18

---

Under review as conference paper at ICLR 2021
Table 5: List of ResNet34 top-1 test accuracy using various optimizers and applied to various datasets. Note that each cell reports the average accuracy over each experimental trial, sub-scripted by the standard deviation over the trials. Values on the left are from trials trained using the initial learning rate as generated by autoHyper. Values on the right are from trials trained using the suggested initial learning rate. Note a '*' indicates that early-stop has been activated.
Dataset
Optimizer
Epoch 50
Epoch 100
Epoch 150
Epoch 200
Epoch 250
AdaBound
91 91.6640..50 99..84
92.37 40.25 /92.03+
92.78 0..88 (9..67
92.79 42 2taeeo...222259/0 49.55
93.2440.17 092.
AdaGrad
00.300...3//91.1 19..9991.....
90.77 99..3333//9.....0..
AdaM
91.4420.10/915544
92.72.t0.1/922 10.27
92.9320.21/933.
#0.33
93.0620..4//33..
=0.30
93.39.20.319932222
CIFAR
Ad
98.35=to.44993.04k
93.40
15.400...1/93.00+uu..
93.401o.15 1
93.401n..1/9953
AdaS
94.00
94.251 /94.4.29/94448..... /94.48....1
13
94.25
22.254o..2944445.200 19..45.0..11
94.251..2
/94.47 1..15
AdaSu.55
91.90+0.25 99.003 29..00aa.../99..3::..
94.91 /9..c//99.144.4. to.16
95.081
15.004..1s/95.2000 99.220
95.08%..1z
/95.20
95.081.o.s. /95.00/
AdaS
87.6820.85/
18.9888.1111 +1.30
92.92+0.32/933 0..52
94.75+0.12 99..9944.9.....
95.01 19.1
15.13
RMSProp
90.6920.47
/ /9..5940.77
21.41
92.2220.65/
097222..6//9269.... 92.669
"2.94=o.as/92..8
10.30
2..03....3/99.990..
SLS
93.3040.1e0 10/...28///..
23.41
/93.4840.00
93.39 09..4944.00
25.34=0.19/93..112
93.33+0.06/
/9..4540.10
AdaBound
69.21 40.69/ 20..s//6........7.....
71.38
7
72.83.to.0/772
AdaGrad
65.35.0.46 /5. 05.257
66.72
(6. 46.58
67.03
66..91 +0.31
67.16+t0.50
0660.0//66.7777 /66.97 +.025
67.4340..8
0...o.9967.000 07.002...23
CIFAR 100
AdaM
68.31 06...6
69.71 (9.71:o.ss66..74.o... 69.77
70.43=0.29/ 70.432..2070.445.u 77.445......
70.98 43.8//4.//////0 20.33
71.43 21.1/////.111131
Ad
73.58 0..
74.22110.85
73.56......77.2222.... 74.222-0.35
73.5810.sh/
73.58-...5
74.19
73.5810.3n
74.2211..00
61
AdaS
75.5420.25/
10.64
75.781 /6..022.7762....100
75.78
25.781.../76.000..... 17.00
75.781
76.05
75.7810.1/76.9940.m
75.781
25..2//7..99..00
AdaSu:ss
71.25+0.90/ 77.57 1.00
77.48 (7.
77.48 77488..../77.664400 77.60++..22
77.4810.37. 77.60
77.481..37
Ada5
75.15
75.15 20.4888...3
78.44
+
2
78.26+0.33.
RMSProp
67.1723.0066.66 66..6
17.17
68.78+1.20 69..669.......
70.23
70.20
70.39 +0.50
70.57+0..4
70.25
SLS
66.8111.25564..885
23.48
15.7820.19/3..11
73.7410.32 13.24 73.7420.32/132222..11
73.77
10.17
73.22=0.11
AdaBound
$5.07+1.08/ 15007441.1//5.11111.111..
55.75-tn.3/55..
255.4/55.37 40.15
56.66
0at.oa..055500... /55.00
56.06
56.22
13.5222..17/55.488.1 4..6
Tiny ImageNet
AdaGrad
52.01 20.03 52.75
54.01 +0.39/ 54.27 +0.77
54.4640.0s 54.94
54.69 55. 12.12
55.044...4.
+0.84
AdaM
54.67 44.99
05.86
54.81 700.1oo../00........ /5..14 1..20
54.46
44.44/.1.11//2...
AdaS
58.0640..4/57.854
58.20+0.42 058.0620.45
o.
58. 58.18 43/ 16
58.02 /57.9820.44
Ad
91455...1/58..5..r
58.99 59.97 +0.88
60.08
39.97=0..0
61.38 60.00
61.56 /5999
20.45
Ad5
56.1841.34/ 57.
60.83 60.229
61.17 /6..72..00.2..
61. 61.10 /0...a//61..000 61.003..14
61.28
60. 20.20
Ada50:975
54.87 10..222/5..119 t0.74
58.91 59.199+0..99
60.62=0.60/
66.222
61. 61.70 63.13
61.81 61.81+0..45
40.27
ImageNet
AdaM
66.22+0.08
14.022
68.17 63. #0.211
668.68
Ada5
71.61
71. 77.48//7778888 48..5
71.87 72.11
71.871o.20/ 20774o..//2.1110.11
Ada50.90
67.52.0.47
68. 15..1....1111 20.21
72.88 72. 00.334
22.099...s/73..552
Ada50.975
58.25440.3
58.09.8.0.0 10.36
69.17 20.09 42.4222.22
T2.122..u5/7..522o..o

---

Under review as conference paper at ICLR 2021
Table 6: List of R top-1 test accuracies using various optimizers and applied to various datasets. Note that each cell reports the average accuracy over each experimental trial, sub-scripted by the standard deviation the trials. Values the left from trials trained using the initial learning rate generated by autoHyper. Values on the right are from trials trained using the suggested initial learning rate. Note a **' indicates that early-stop has been activated.
Dataset
Optimizer
Epoch 50
Epoch 100
Epoch 150
Epoch 200
Epoch 250
AdaBound
91. 91.24.to.21 190.39003....
92.25 99..64
9..69 92.00922.2
92.79 20t.:o.24/9222000 29/.2222200z
92.85 /92.35
AdaGrad
80.42 /89.67
90.3940.18/
90.55 10.55=a..1/91.16-0000 //..1/111.......
90.75.0
99..2......00
AdaM
22.17:o..3/992.1622
13
92.63....15/92..0 10.17
92.88.10..32992.633.
92.95.0.24/92.9332
CIFARIO
AdaSu.a
12.81+0.1/922.88
92.804
92.80
00f.4...992.92..// /92.92
92.80
AdaSu.9
93.71
93.754e.12
94.05 10.10
93.75 /94.05
2..10
93.75"
/9..05
93.754
/94.05-...00
91.96
94.6940.15/
/9...994.1111 40.16
94.74_..m 16/
190.eo..e9999.... 194.93
94.742..14/
/94.93 /4.742..1/944.93....
94.741o.1e.
Ada$0.376
87.99
93.3540.43.
43.
94.74 009..7777000
94.88=a.07
/95.07
94.94+0.04
10..o04995..11. /9..1440.20
RMSProp
90.35+1.21/
91.8540.10 10.4
92.26
92.66.o..1/92.333
92.6920..33
SLS
93.4110.00/
/93.4880.009
93.39 99..4920.00
03.312o.13/93.4142
93.33+0.00
93.45500.1
AdaBound
70.15 t0.31/ /28.//6..7.... 70..75....42
72.29+0.21177
72.91 7..2//
71.60
73.10:0..8
73.1640 40.25 77.004
AdaGrad
66.07 40.36/ 0t..o//6..122... 66.12
67.39 40.48 67.3 +0.39
67.624...0
40.50
67.85
66.77++.33
67.7540.36
67.76+0.50
CIFAR 100
AdaM
68.3220.65/ 68.51 +0.39
69.3340.42
69.56
69.60=o 18
70.07
69.72=o 22
70.49
70.09+0.35
70.34
AdaS
73.3220.30/ 2o..oo..//7.44......
73.38
70.59
73.38
73.59+0.00
73.38
73.59
73.38
20
AdaSu:9
25.2710.28/
75.27 1
7.......77...5:..... 75.15
75.27
75.15+0.17
70.271n.28)
Ada5u:ss
74.40+0.17/78.. 773.6...77...
0..3 76.44
76.44
76.47
76.49
76.53+0.300
76.49%
AdaS
63.62 64.44 10.444111/64444440000
+0.25
23.
76.59 19.19
77.05+0.10
76.62 77.1
76.68 77.23+
70.68:...s/7.2234...0
RMSProp
66.29 66.52
68 68.4840.31
68 40.49
20.88_0.20
69.
+
69. 69.29-0.55
69.78
69.28+0.50
70.08
SLS
66.81 64.88.1.22 1.23
73.48
73.0220.20
73.78 73..1..97/3314....
73.7 (3.7410.. 10.32
73.2440.11
73.77 73.7710.12
10.13/ 73.2220.11

---

Under review as conference paper at [CLR 2021
Table 7: List of ResNeXt50 top-1 test accuracies using various optimizers and applied to various datasets. Note that each cell reports the average accuracy over each experimental trial, sub-scripted by the standard deviation over the trials. Values the left from trials trained using the initial learning rate generated by autoHyper. Values on the right are from trials trained using the suggested initial learning rate. Note a '*" indicates that early-stop has been activated.
Dataset
Optimizer
Epoch 50
Epoch 100
Epoch 150
Epoch 200
Epoch 250
AdaBound
88.00 $9.01
90.90 990.75.0.99
90.76 65 /9..29
91.63
91.69 w/..////99..........
AdaGrad
87.80 88..85
89.2540.33/
/89.3799..0
89.79
117.//9.63.. 60.60
90.07 89.80
90.13
/90.070000
AdaM
90.11 10.1190.59/9000
11.............. 399.74_20.80
91.1540.10/
91.68 26/ 26/11.82.0..2
91.61 90.00
92.12+0.07/
/92 99.1182...1111
CIFARIO
AdaSu.a
91.51+0.22/
91.494..10
91.491...10
91.56=00.07
91.491.u.e/ 99..5620.07
91.494..
AdaSu.9
93.31
#3.11/93.33
/93.37 40.21
93.51
93.60
93.51
/9..69 L0.16
93.51 93.6
93.514
/9. 93.660...1e
a9.0
/09.:4.../999999000 /90.49 40,24
94.37+0.13/
/94.61 40.15
94.59
/94.9994..... 0.110
94.61 14.61...1/94.62...o
99.
94.611..11
/9..62
Ada$0.376
80.65 15....335/853311.0
92.07
94.93+0.07 000.0o///944...000 /94.8644..11
95.03 /9..9/999........
95.02+0.0n
/9.003
RMSProp
89.26.10.42
20.80
89.6540.88/
90.17
91.0220036 00t.:////99......0.
90.9620.45 /91. 491.=oo.s/91..00o...
91.34+0.50
99. 00.
SLS
93.05 0e..//931111. 29..1
93.52+0.12 12.62:...2//9..100 100
93.5110.20
03.544.0.23/33.11
93.5610.22 20.. 13.0112..4
AdaBound
66.66 (00.o64/66..700
/6..75
69.924a.40/
70.13 +0.40
70.63 /1.061111.... 11106++0.4 00.51
71.02=0.28
77.33
71.20 71.43
AdaGrad
40.77 62.97
40.04
65.27 4..46 +0.64
+0.64
65.36 60.30
65.75
66.03 20.50 4..3
CIFAR 100
AdaM
66.7410.48 /66. /66.4664 20.45
68. 68.4040.24
68.868..7700 20.50
20.57
69.59
69.04_..11/70.14420 70.1144...0..
69.04_o.18/
69.12+o.1 16 70.32.0.40
AdaS
72.06 /72.39+..22
72.00
72.00
72.4110.16
72.00 72.4140.16
72.0040.44/ 72.447777.441.01
22
AdaSu:9
7..23
74.30
74.41
74.43
74.41
o 0..0
74.43
74.41 0.26 74.443
74.411o.26. 74.402/777.43....
72.78 27wp:/22.99.....
75.47
46.03
75.74
75.95
75.63 75.95
75.63 77.9554..20
AdaS AdaS
03.1641.84/ +1.48
74.42 74.522
76.27
+0.23
76.45 76.47
76.58
/
RMSProp
63.69 1..96
66.2 66.24
67.82.0.82
66.35_0.26
11.16
66.83 30
69.31
67.17
69.4541.17
SLS
55.3414.77
58.80
70.10
70.75
71.51 20.1
71.777..7....
71.8920.0
72.03
71.82+0.23
72.0840.45

---

Under review as conference paper at [CLR 2021
Table 8: List of DenseNer121 top-1 test-accuracies using various optimizers and applied to various datasets. Note that each cell reports the average accuracy over each experimental trial, sub-scripted by the standard deviation over the trials. Values the left from trials trained using the initial learning rate generated by autoHyper. Values on the right are from trials trained using the suggested initial learning rate. Note a '*" indicates that early-stop has been activated.
Dataset
Optimizer
Epoch 50
Epoch 100
Epoch 150
Epoch 200
Epoch 250
AdaBound
80.47
88.773
90.094..23
90.6444..25
90.65 91.20
90.59 99.32
90.9640.4x 091.699......0
AdaGrad
$8.57=0.22
188.31 :0.45
80.20
00.
89.36.40.09 889 26
89.47
89.333
89.85.0.119
-9.5220.16
AdaM
90.75.0.22
02..1222.77
91.39 20..0....90.........
91.75.10.27
91.72 29..2999.........
91.86110.20 20..2////......o.
CIFARIO
AdaSu.a
$1.6110.a0
(8.228+a..... 0.226
91.5940.25 99.228
91.591...25
91 20.23
91.591...m
99.29 2824884.000 +0.23
91.591...0
991 28
AdaSu.9
92.91
t0.08/ /92.
-0 +0.22
92.97
93.06.0.11
92.97
/9.006
92.97 -..1
93.
92.97
99. 03.06.4..10 14
AdaSu.ss
91.59
91.59 4:0,2
93.33
993.4534444. 40.18
93.33 24 /93.51
0.220
93.331
983.1110.00
93.334
/93.5 +.0.20
AdaS0.375
86.96+1.15/
86. 36.1776.. 00...0
91.87
92.00
93.17=0.20 20T=2.00/99.355 /93.993.3...
0.11
93.40=0.22
/93.72
93.47
RMSProp
58.8040.993
188. 1.16
90.95.1..5 28..
91.5410.52
00022o..2/90.77730 /2/p//0.77000 10.64
91.65 20t.:///999.5.....
91.83.10.30
SLS
92.95 28/ /92. 92.78=20.0
93.35 20..2.09993....1..3
93.40+..177 17 13.1220.15
03.3840..1/93.110
93.3640.
AdaBound
63.87 /65 65.30+t..2 22
65.61
48.00
66.66
68.658.665
+0.21
66.69
68.4640.32
67.00
68 /6..9668..0000
AdaGrad
$1.62.t0.44 /6.008 0.28
4..31
62.40
00.50
40.41
62.09 +0.22
62.79 1....//..44444.44 62.1440.11
CIFAR 100
AdaM
66.53 20.50
67.64
65.69
68.32+..27
66.42.0.57
68.46440.28
66.8940.55
69.05 /t../67..411. 48
AdaS
70.90 11.00=...170.9-::
71.01 7 o.28 20.63
71.01 70.63
71.01
77.66340.3
71.01 700.63
AdaSu.
73. 43. 73.113....300
73.131..44/
73.25 25
73.13"..44
73.25
73.13"
73.131..44
20..2..4/333..4400 73.25
o..6 72.90
73.984 4n.09
73.98 10.33
74.22
73.98 74.0://4..2..0.
73.981..33. 74.22:2.22
AdaS0.355
61.85
63.44
72.1640.30/
72.65
73.81 44.44/ 73.92
73.88
73.9
73.97
77. 74.10+0.47
RMSProp
60.78
62.626. 20.54
66.22 26.00
64..5
67.57+0.00
/te.oo//66.7744 66.7744..00
67.35+t0.00
66.30
68. 68.13+0.00
/6.66 40.58
SLS
52.01 55.666...666..
65.96 62. 62.29
69.88 26..22//77777 11.11
70.12+0.45
68.63+1.61
70.2540.1. 69.444994444.

---

Under review as a conference paper at ICLR 2021
D COMPARISON AGAINST STATE OF THE ART (RANDOM SEARCH)
D.1 SETUP
The search space is set to [] x 10 0.1] and a loguniform (see SciPy) distribution is used for sampling. This is motivated by the fact that autoHyper also uses and logaribmnialll-spaeed grid space. We note that we ran initial tests against a uniform distribution for sampling was done and showed slightly worse results, as the favouring of smaller learning rates benefits the optimizers we considered. In keeping with autoHyper's 8 design, the learning rate that resulted in lowest training loss after 5 epochs was chosen. One could also track validation accuracy, however visualized in Figures ((a) & 13 validation loss is more stable for the datasets we are considering. This selection could be altered if the dataset being used exhibits different behaviour, however this would be manual alteration at the selection of the practitioner - one that does not need to be made using autoHyper.
D.2 ADDITIONAL DISCUSSION AND RESULTS
Can you replace 2(r) with validation loss? Replacing 2(1) with validation loss does not work because greedily taking validation loss (or accuracy) is not stable nor domain independent. Analyz- ing Figures (()) & validation loss/accuracy is unstable since either the network (EfficientNeiB0 in Figure or the dataset (TinylmageNet in Figure 5(a) results in unstable top-1 test accuracy/test
loss scores that are unreliable to track. See also Figure 14 which demonstrates the inbaility to track validation loss/accuracy for various learning rates. Further, validation accuracy/loss can vary greatly based on initialization, whereas our method does not vary due to its low-rank factorization. Finally, our metric. (1)), is always guaranteed to be zero with a sufficiently small learning rate and maxi- mized with large learning rates, therefore can always dynamically adapt our search range the proper range. This fact is not so true for tracking validation accuracy/loss.
Additionally, low validation loss does not correlate high validation accuracy (an additional figure, Figure 12 in Appendix Shows this). You might then suggest to take a k of the best performing learning rates based on validation accuracy/loss and focus on those, but this requires you to manually define & then attempt a manually defined Grid/Random Search refinements around those areas, with manual heuristics to indicate when to stop searching, whereas our method fully automatic and self-converges. Not to mention, this would take more time.
In summation, existing SOTA method like Random Search cannot compete with autoHyper when given similar budgets and minimizing the manual intervention/refieeeent This displays autoHy- per's prominent feature of being a low-cost, fully automatic algorithm to search for optimal hyper- parameter bounds (namely in this work, the initial learning rate). Future work could include using autoHyper to quickly discover this optimal hyper- syper-parameter range, and then further refine using more extensive HPO methods with greater budgets if truly superior performance is required, and this could further alleviate a lot of manual refinement that currently plagues existing SOTA meth- ods.
Table 9: Learning rates for ResNet34 Random Search comparison. Left inner columns show Ran- dom Search generated, right inner columns show autoHyper generated.
Optimizer
CIFARIÓ
CIFARI00
TinyImageNet
AdaM
0.000330
0.000333
0.000125
0.000241
0.000175
0.0001965
AdaBound
0.000392
0.000347
0.000353
0.000347
0.000124
0.0000944
AdaGrad
0.001598
0.002861
0.001828
0.002236
0.000715
0.0022359
0.006779
0.012374
0.009252
0.010190
0.039480
0.0085857
23

---

Under review as a conference paper at ICLR 2021
AdtBound
AdeBound
Ado0red
AS3M
AdAN
Ads5 0
6.95
0.75
Test Accuracy
Top-1 Test Accuracy
0:33
0912
0.72
0.34
0:91
8:00
0.9
0.40
0.00
107
107
Training Loss
Trairing Loze
Training Loos
32-
204
Epoch
Epoch
ResNict44 on TinylmageNet
ResNec34 on CIFAR10
ResNet34 on CIFAR10D
Figure 13: Top-1 test accuracy and train loss for ResNet34 applied to TinyImageNet, CIFARIO 10, and CIFARIOK 100, using learning rates as suggested by either Random Search (as described above) or autoHyper. Titles below plots indicate what experiment the above plots refers Legend labels marked by (solid lines) show results for autoHyper generated learning rates and dotted lines are the Random Search results.
AdaBound
AdaGrad
AdaM
AdaS-0.9
2.25
Validation Accuracy
Validation Accuracy
4.0
0.5
1.75
0.5
1.25
5.6
0.75
3.0
0.00
0.84
0.05
0.08
0.02
ResNet34 on TinylmageNet
ResNet34 on CIFAR10
ResNet34 on CIFAR100
(a) Validation Loss
AdaBound
AdaGrad
AdaM
AdaS 0.9
0.8
Accuracy
Validation Accuracy
Validation Accuracy
0.6
0.3
0.1
0.4
0.1
Epach ResNet34 on TinylmageNet
Epoch
Epoch
ResNet34 on CIFAR10
ResNet34 on CIFAR10D
(b) Validation Accuracy
Figure 14: Visualization of the (a) validation loss and (b) validation accuracy for various learning rates ResNet34 various datasets. These figures demonstrate the inability to propoerly track these metrics as we do ours (i.e. 2
24


Reviews:
[{"review_id": "yHNg9RpdCZo", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview:\nOverall I find this an interesting algorithm, but have several serious concerns about the (lack of) experimental baselines.\n\nMy primary concern is that, given this paper is introducing a new hyperparameter tuning algorithm, there are no comparisons to baseline hyperparameter tuning setups. For example, in Figure 3, what is the best final validation and/or test accuracies achieved by autoHyper and by random search, for the same number of trials (one could even run autoHyper first to see how many trials it takes, and then see if random search could beat it in that many trials). I understand that BayesOpt algorithms can require more involved engineering to set up, but there do exist open source codebases that could be used (such as https://github.com/HIPS/Spearmint), and for a very solid paper I believe that comparing to existing SOTA methods would also be useful. For all of these methods, one could take the validation loss at 5 epochs (as is done in the proposed method) as the metric to tune on for a fair comparison.\n\nAdditionally, the suggested initial LRs seem problematic to compare to. It would useful to highlight how the suggested initial LRs were tuned. Using the information in Appendix D from Wilson 2017, I see they were tuned using less trials than the proposed algorithm, and using a grid search algorithm that may be worse than random search (Bergstra & Bengio (2012)). Furthermore, these suggested initial LRs were for a different model (this VGG model referenced from Wilson 2017 http://torch.ch/blog/2015/07/30/cifar.html) than the models considered in the experiments here, which means they could serve as a poor baseline. This baseline weakness is also shown when one considers that the ranges of values tuned over in Wilson 2017 seem to be quite close to several of the values proposed by autoHyper, meaning that perhaps repeating the same grid search (even with the same early stopping as in autoHyper) could be competitive (for Adam, Wilson 2017 considered {0.005, 0.001, 0.0005, 0.0003, 0.0001, 0.00005} and autoHyper proposed 0.000333, and for AdaGrad Wilson 2017 considered {0.1, 0.05, 0.01, 0.0075, 0.005} and autoHyper proposed 0.0049724). In the cases where the autoHyper values are not close, they are sometimes (although not always) outside the range considered by Wilson 2017, which could bias the results towards the proposed algorithm because the suggested LRs may not have had the chance to encounter the more successful values proposed by autoHyper (in the case of AdaGrad, Wilson 2017 considered {0.1, 0.05, 0.01, 0.0075, 0.005} while the proposed method tunes within [1e-4., 0.1] and selects 0.002861 for CIFAR10 ResNet34). Several recent works have shown that when tuning optimizers one needs to be careful to report the ranges used, as changing the hyperparameter ranges can drastically affect experimental results (https://arxiv.org/abs/2007.01547, https://arxiv.org/abs/1910.11758, https://arxiv.org/abs/1910.05446), and while they mainly consider the effect on optimizer comparisons the point still stands for comparisons between hyperparameter tuning algorithms (when comparing to random search, the range of LRs should be the same for each tuning algorithm).\n\nPros:\n-It is useful to show that the tuning algorithm works across many optimizers, models, and datasets. The types of experiments seem sufficient, just missing baseline tuning algorithms.\n-It is important to call out the negative results presented in experiments, and the authors did a good job of that (when applicable) in section 4.2\n\nConcerns:\n-Figure 1a would be much more informative if you showed the entire training trajectory, including past the first 5 epochs, to see if the selected learning rates actually generalize noticeably better.\n-In your conclusion you discuss that you could extend your tuning algorithm to multiple hyperparameters, and I believe in order to truly demonstrate its capabilities this would be required, given that relatively simple baselines such as random search can continue to perform well in multi-dimensional tuning setups (although I would recommend against tuning the batch size given that it interacts so strongly with numerous other hyperparameters, and would instead suggest tuning momentum, weight decay, and/or label smoothing).\n\nWriting:\n-Is there an extra 2 in the denominator of Zt(λ)?\n-It was initially confusing that the gradient notation was used for the constraint on Eq. 5, it would be clearer if the authors stated earlier that they did not compute the literal gradients (which would require backpropagating through unrolled updates).\n-The “Observations on the generalization characteristics of optimizers.” subsection seems somewhat out of place\n\nAdditional feedback, comments, suggestions for improvement and questions for the authors:\n-Instead of calculating Z(λ) with G¯λ(t, l), one could imagine making another baseline out of using validation accuracy at each epoch (perhaps normalized by some SOTA number if being in [0, 1] is desirable.)\n-While very informative, Figures 5 and 10 seem cluttered, and in addition to them it may help for a future version to have the diff between the baseline and proposed method instead of two different lines", "extra_scores": {}, "extra_text": {}}, {"review_id": "qJQCtR11NR_", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "** Summary\n\nThe paper proposes an efficient framework to search for the optimal initial learning rate to train neural networks. The key idea is to introduce Knowledge Gain, a metric derived from the singular values of each layer, to indicate the convergency quality of training. Taking advantage of the metric, a logarithmic grid search algorithm (AutoHyper) is proposed to search for the optimal learning rate according to Eq 5 via short-time training (e.g. for 5 epochs), which is demonstrated to be very efficient and take effect to some extent. \n\n\n** Pros\n\n1)\tThe paper studies a very important problem in neural network HPO: how to efficiently model and approximate the response function. The usage of Knowledge Gains as the proxy metric looks interesting and somewhat intuitive (however, I still have some concerns, see next). \n2)\tI appreciate the authors present a lot of empirical results, on different models, datasets and optimizers. Detailed experimental settings and numerical data are also provided, which could be convenient to reproduce. \n\n** Cons\n\n1)\tThe writing and presentation of the paper is not good. \n2)\tFrom Table 5 to Table 8, it seems the automatically searched results do not outperform the suggested baselines significantly in many cases. \n\n** Technical concerns\n\nIn the definition of Knowledge Gain (Eq 3), though the distribution of singular values can be used as a metric to indicate the redundancy of a layer, to my knowledge it should be sensitive to the way of initialization; for example, an identity/orthogonal initialization may correspond to larger KG but random gaussian may result in smaller values. Furthermore, the evolving of KG during training may also vary from different layers, while in the formulation above Eq 4 it is just derived by averaging over all layers. So, I doubt whether the proposed metric is universally applicable for different situations. From the experiments, it seems the gain is not that significant. \n\n** Presentation\n\nAnother of my major concerns is that the paper is not well written. I can hardly understand what exactly the paper suggests, since many formulations and figures are not clear, for example:\n\n1)\tIn Eq 2, what is the exact definition of \\hat{W}_d? What does “low-rank factorized matrix” refer to? It seems the formulation here is slightly different from those in Hosseini & Plataniotis (2020).\n2)\tAt the beginning of Sec 2.2, when \\bar{\\mathcal{G}}_\\lambda equals to 0? (from Eq 3, it seems never to be zero). \n3)\tIn Eq 5, why to minimize (1 - \\mathcal{Z})? Doesn’t smaller \\mathcal{Z} correspond to better results? What is the meaning of the constraint in Eq 5?\n4)\tIn Fig 1(b), how to calculate the histogram of \\mathcal{Z} exactly?\n\n\n================\n\nThanks for the rebuttal. I appreciate the authors' efforts on polishing the presentation. It clears some of my concerns. So, I raise my rating from 4 to 5. However, after reading the rebuttal and other reviewers' comments, I still feel that the contribution of the paper is not that significant,  since the search space just includes a scalar and the empirical improvements are not strong (Table 5 to Table 8). I think the authors may consider further applying their method to multi-dimension HPO problems (e.g. joint searching initial learning rate, momentum and weight decay) to verity the effectiveness on more challenging configurations. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "UOw_Ik_lLeM", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a new hyper-parameter optimization method based on a new response function defined on the low-rank factorization of 4D convolution weights. The presented approach appears to mathematically solid and interesting. Although I am not an expert in this hyper-parameter optimization, I think the approach has the potential to speed up and improve the hyper-parameter search or neural network architecture search.\n\nIn my opinion, the demonstrated experiments are less interesting because the focus is only on selecting a single initial learning rate. This is not that interesting because we normally have a learning rate scheduler that will change over time. The experiment somehow ignores different types of learning rate schedulers for comparisons. In addition, as the learning rate is just a scalar, we can simply do some grid search such as [0.01, 0.001, ..] to coarsely find a good learning rate and then refine.\n\nAlso, the presented work would be more interesting if it can demonstrate improvement in other hyper-paramter optimization such as weight decay and convolution filter size and channels. \n\nThere are some questions about the presented approach:\n- Are all the convolution layers in a CNN used in Eqn (4)? The definition of mode-3 and mode-4 is not clear to me.\n- ", "extra_scores": {}, "extra_text": {}}, {"review_id": "5VX1yfhfr37", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "###############################################################\n\nSummary:\n\nThis paper used the knowledge gain to provide an algorithm to choose the initial learning rate. The paper explained the reason that choosing the specific response function and demonstrated the effectiveness of the algorithm by several experiments. \n\n##############################################################\n\nReason for Score:\n\nThe intuition is reasonable, but there are several points regarding the feasibility and correctness that need to be clarified.  \n\n##############################################################\n\npros:\n\n1, This paper proposed a dynamic tracking algorithm of low computational overhead on the order of minutes and hours to find a good initial learning rate.\n\n2, The paper explained in detail about the reason to choose the specific response surface function, and demonstrated the reasonability of the algorithm.\n\n##############################################################\n\ncons:\n\n1, The algorithm had an implicit assumption: the response function is monotonously decreasing (maybe under expectation). Otherwise, the algorithm can either not converge or go to a local minimum. Even though the author list some examples, I doubt about this. Imagine that we choose a very large learning rate (some unreasonable number that just for a counterexample), the training loss cannot be decreased, then I doubt that the response function cannot decrease either.  Did the author conduct some examples about this? If this is the case, then how to choose the range of the searching area matters a lot.\n\n2, The algorithm induces some \"hyperparameter\" again. Such as the alpha, and the condition \"rate of change plateaus\". How to choose the alpha, and what is the \"rate of change plateaus\"? Does these \"hyperparameters\" influence the result?\n\n3, The initial learning rate seems only influence the convergence rate for training loss, but why it influences the testing accuracy? I suggest the author at least give some reasonable explanation about this.\n\n4, The paper compared the algorithm only with the baseline. Is there any other work about tuning the initial learning rate? Or this paper is the first?\n\n5, minor problems:\n\n(1) In equation 1, x is not showed in expectation function (I can only see X(train)), and also, it should be the minimum of expectation, not the expectation of minimum.\n\n(2) In algorithm 1, there is nowhere that the index i is changed.\n\n(3) I suggest the author change the color of the lines in graph. It is very difficult to find which line corresponds to which experiment.\n\nThanks for your rebuttal. It solves some of my concerns. However, combining with rebuttal and other reviewers' comment, I think only choosing the initial learning rate is not that reasonable. It may be more convincing to me that the whole parameters are chosen together. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the important problem of efficiently identifying good hyperparameters for convolutional neural networks. The proposed approach is based on using an SVD of unfolded weight tensors to build a response surface that can be optimized with a dynamic tracking algorithm. Reviewers raised a number of concerns which were not fully addressed in rebuttals and lead me to recommend rejecting this work. In particular: focus on single hyperparameter (learning rate) made it unclear whether the proposed approach could actually be used for other hyperparameters or to jointly optimize combination of hyperparameters, empirical improvements even for learning rate are not strong and baselines are weak, and concern that initial success early in training (5 epochs) may not lead to generalization late in training. Additionally, there were several concerns around the clarity of the presentation, which I also found hard to follow: how is KG related to information-theoretic metrics, why is the particular form of averaging across layers reasonable, and how is it related to other generalization / performance metrics? With additional experiments on other hyperparameters (for example L2 regularization), I think the work would be greatly strengthened. 
Decision: Reject