EA-HAS-Bench: Energy-aware Hyperparameter and Architecture Search Benchmark
Authors: Shuguang Dou
XINYANG JIANG
Cai Rong Zhao
Dongsheng Li
Venue: ICLR 2023 notable top 25% (2023)

Abstract: The energy consumption for training deep learning models is increasing at an alarming rate due to the growth of training data and model scale, resulting in a negative impact on carbon neutrality. Energy consumption is an especially pressing issue for AutoML algorithms because it usually requires repeatedly training large numbers of computationally intensive deep models to search for optimal configurations. This paper takes one of the most essential steps in developing energy-aware (EA) NAS methods, by providing a benchmark that makes EA-NAS research more reproducible and accessible. Specifically, we present the first large-scale energy-aware benchmark that allows studying AutoML methods to achieve better trade-offs between performance and search energy consumption, named EA-HAS-Bench. EA-HAS-Bench provides a large-scale architecture/hyperparameter joint search space, covering diversified configurations related to energy consumption. Furthermore, we propose a novel surrogate model specially designed for large joint search space, which proposes a Bezier curve-based model to predict learning curves with unlimited shape and length. Based on the proposed dataset, we new energy-aware AutoML method that arms existing AutoML algorithms to consider the search energy consumption, and our experiments show that the modified energy-aware AutoML methods achieve a better trade-off between energy consumption and model performance.

Published as a conference paper at ICLR 2023
EA-HAS-BENCH: ENERGY-AWARE HYPERPARAME- TER AND ARCHITECTURE SEARCH BENCHMARK
Shuguang Dou' ;Xinyang Jiang" Cairong Zhao' Dongsheng Li
Tongji University, 2 Miirosooft Research Asia
ABSTRACT
The energy consumption for training deep learning models is increasing at an alarming rate due to the growth of training data and model scale, resulting in a negative impact carbon neutrality. Energy consumption is an especially press- ing issue for AutoML algorithms because it usually requires repeatedly training large numbers of computationally intensive deep models to search for optimal configurations. This paper takes one of the most essential steps in developing energy-aware (EA) NAS methods, by providing a benchmark that makes EA- NAS research more reproducible and accessible. Specifically, we present the first large-scale energy-aware benchmark that allows studying AutoML methods achieve better trade-offs between performance and search energy consumption, named H--H--S-Bechhh EA-HAS-Bench provides a large- scale architecture/hy- perparameter joint search space, covering diversified configurations related to en- ergy consumption. Furthermore, we propose a novel surrogate model specially designed for large joint search space, which proposes Bézier curve-based model to predict learning curves with unlimited shape and length. Based on the proposed dataset, we modify existing AutoML algorithms to consider the search energy consumption, and our experiments show that the modified energy-awan AutoML methods achieve better trade-off between energy consumption and model per- formance.
INTRODUCTION
As deep learning technology progresses rapidly, its alarming increased rate of energy consumption causes growing concerns (Schwartz et al. 20200 Li et al. [02 Strubell et al.) 2019)) Neural archi- tecture search (NAS) (Elsken et al. 2019). J. hyperparameter optimization (HPO) (Feurer & Hutter, 2019) lifted the manual effort of neural architecture and hyperparameter tuning but require repeat- edly training large numbers of computationally intensive deep models, leading to significant energy consumption and carbon emissions. For instance, training 10K models CIFAR-10 (Krizhevsky) jet al., 2009) for 100 epochs consume about 500,000 kWh of energy power, which equivalent to the annual electricity consumption of about 600 households in China.
As result, is essential to develop search energy cost aware (EA) AutoMI. methods, which are able to find models with good performance while minimizing the overall energy consumption throughout the search process. However, existing NAS studies mainly focus on the resource cost the searched deep model, such parameter size, the number of float-point operations (FLOPS), latency (Tan jet al. 2019: Wu et al.) 2019 He et al.) 2021). Exploiting the trade-off between model performance and energy during the searching process has been rarely studied (Elsken al. 2019). In this paper, we propose to take one of the most essential steps in developing energy-aware (EA) NAS methods that make EA-NAS research more reproducible and accessible. Specifically, we provide a benchmark for EA-NAS called Energy Aware Hyperparameter and Architecture Search Bench- mark HA--H--BBench,, where the researchers can easily obtain the training energy cost and model performance of certain architecture and hyperparameter configuration, without actually training the model. In order to support developing energy-aware HPO and NAS methods, the proposed EA-HAS-Bence should satisfy three requirements.
WWok done during an internship in Microsoft Research Asia. Email: dousg @tonggi.cdu.coo Corresponning authors. Email: zhaocairong wwwggi.ceuu.cnn eoiirsoinnmmmmrooo...oommmm

---

Published as a conference paper at ICLR 2023
Table 1: Overview of NAS benchmarks related to
Bench.
Arch.
Benchmark
Size
Type
Type
LCs
Query
NAS-Bench-101
423k
Tab.
Cell
Accuracy
MAS-Bench-201
6k
Tab.
Cell
Accuracy & Loss
NAS-Bench-300
10
Surr.
Cell
Accuracy
NATS-Bench
32K
Tab.
Macro
Accuracy Loss
NAS
HW-NAS-Bent Bench
15K/102t
Tab./Fsti.
Cell
Latency Inference Energy
423k
Surr.
Cell
Accuracy
6k
Surr.
Cell
Accuracy & Loss
10is
Surr.
Cell
Accuracy
NAS-HPO-Bende
62K
Tab.
MLP
Accuracy
NAS+
NAS-HPO-Bench-II
192K
Surr.
Cell
Accuracy
HPO
EA-HAS-Bench (Ours)
3 x 10
Surr.
Macro
Accuracy & Inference and Total Search Energy
Our dataset needs to provide the total search energy cost of rumning specific AutoML method. This can be obtained by measuring the energy cost of each particular configuration the method traverses and summing them up. As shown in Table most of the existing conventional
Search Energy Cost
benchmarks (Ying et al.] 2019 Dong & Yang 2020 [Siems et al. 2020) like NAS-Bench-100 do directly provide training energy cost but model training time the training resource budget, which as verified by our experiments, is an inaccurate estimation of energy cost. HW-NAS-benchs (Li al., 2021bb provides the inference latency and inference energy consumption of different model architectures but also does not provide the search energy cost.
Energy Related Joint Search Space The search space of EA-HAS-Bendh should include the configurations that affect both the model training energy cost and performance. Since both model architectures (e.g., scales, width, depth) and training hyperparameters (e.g., number of epochs) are correlated to the training energy cost, designing a joint hyperparameter and architecture search (HAS) search space is essential. Most NAS benchmarks use single fixed training hyperparam- eter configuration for all architectures. Existing HAS benchmarks (Klein & Hutter) [2019] Hirose et al. 2021) use small toy search space which does not cover enough critical factors affecting the search energy cost. As result, EA-HAS-Bence provides the first dataset with a ren-billion-leell archieeture/hyppppraneee joint space, covering diverse types of configurations related to search energy cost.
Surrogate Model for Joint Search Space Due to the enormous size of the proposed joint search space, surrogate model is needed to fill out the entire search space with subset of sampled configurations. Existing surrogate methods (Zela et al. 2022: Yan al. 202T) are not applicable to our proposed large-scale joint space. For instance, those methods do not consider the situation of maximum epoch variation and predict only fixed-length learning curve or final performance. Thus, propose the Bézier Curve-based Surrogate (BCS) model fit accuracy learning curves of unlimited shape and length.
We summarize the contribution of this paper as follows:
EA-HAS-Bench is the first HAS dataset aware of the overall search energy cost. Based on this dataset, we further propose a energy-aware AutoML method with search energy cost related penalties, showing energy-aware AutoMI. achieves better trade-off between model performance and search energy cost.
We provide the first large-scale benchmark containing an architettur/hyperppraeeerr joint search space with over 10 billion configurations, covering various configurations related to search energy cost.
We develop novel surrogate model suitable for more general and complex joint HAS search space, which outperforms NAS-Bench-Xi and other parametric learning curve- based methods.
"The dataset and codebase of EA-HAS-Benchl available http://gghaaaccommmmierffffffff Bench.
2

---

Published as a conference paper at ICLR 2023
Table 2: Overview of EA-HAS-Bench's search space
Type
Hyperpurameter
Range
Quantize
Space
Depth d'
[6.15]
10
[48, 128]
10
RegNet
[8,32]
0.1
241
[2.5. 31
0.001
501
Group Width
[1. 32]
5
Total of Network Architectures
6 10"
[0.001, 0.003, 0.005, 0.01, 0.03,
Learning rate
0.05, 0.1, 0.3, 0.5, 1.0)
10
Optima
Max epoch
(50, 100, 2003
3
Decay policy
"cos'. 'exp'. 'lin'}
3
Optimizer
'sdd'. 'adam', "aaamm,"'daen 'adammw
3
Training
Data augmentation
[onne Total of Hyperparameter Space
2
540
2 CREATING ENERGY AWARE BENCHMARKS
2.1 EA-HAS-BENCH SEARCH SPACE
Unlike the search space of existing mainstream NAS-Bench that focuses only network architec- tures, our EA-HAS- H--HBBBehchhh consists of a combination of two parts: the network architecture space- RegNet (Radosavovic al. 2020) and the hyperparameter space for optimization and training, in order to cover diversified configurations that affect both performance and energy consumption. The details of the search space are shown in Table
Neg Search Space. Our benchmark applies RegNet as the architecture search space because it contains several essential factors that control the scale and shape of the deep model, which corre- sponds to the training energy cost. Specifically, RegNet is a macro search space with the variation in width and depth explained by quantized linear function. Specifically, for widths of residual blocks: uy = Wy + wa x j. where 0 S j I d and denotes the depth. An additional parameter tmm is introduced to quantize uj, ie. uj = wi x w and the quantized per-block widths wy is computed by
wy
wi w
where [sy] denotes round sy. the original search space of RegNet is for ImageNet and is non-trivial to directly apply to other datasets. result, we shrink down the original model size range of the RegNet space and constraint the total parameters and FLOPss of model to a relatively small size, which achieves faster training and saves resources.
Training hyperparameter Search Space. Hyperparameter space (e.g., learning rate and maxi- mum epoch) also has great impact energy consumption throughout the training phase. For example, the maximum epoch is almost proportional to training time, which is also proportional to training energy cost. Different learning also causes different convergence rate and different tal training time, resulting in different training energy cost. Thus, for hyperparameter search space, EA-HAS-Bend considers training epochs, and the most important factors in training schemes, i.c.. base learning rate, decay policy of learning rate, optimizer, and data augmentation.
As result, the search space of EA-HAS BennhBBehhh contains a total of 3.26 x 10 configurations, in- cluding 6 x 107 architectures and 540 training hyperparameter settings with variant training epochs.
2.2 EVALUATION METRICS
The Ha-HAA-Beedde dataset provides the following three types of metrics to evaluate different con- figurations.
3

---

Published as a conference paper at ICLR 2023
1:3:0000,, oglin: "agl" my
2636 spti. m))
Inisal Points,
HC 540 1xl. opti: "sge" m Trinung Hypespanmmttr Configs
pepprrmmter VecsoE %
Feer
Hyperparameter Enooder
tritial Brarch
Mocal
Model
Lesting Prediction
Priducted Botter Charee
254h
Model RogNeet Condigs
Antilecture vector, V
Architseture Enooder
Branch
Figure I: Overview of Bézier Curve-based Surrogate Model. HC denotes Hyperparameter configu- ration.
Model Complexity. Metries related to model complexity include parameter size, FLOPs, number of network activations (the size the output tensors of each convolutional layer), as well as the inference energy cost of the trained model.
Model Performance. In order to support multi-fidelig NAS algorithms such Hyperband (Li ee al. 2017). AA-HAS-Bende provides the full training information including training, validation, and test accuracy learning curves for each hyperparameter and architecture configuration.
Search Cost. Firstly, the energy cost (in kWh) and time (in seconds) to train model under a cer- tain configuration for one epoch are obtained. Then, by accumulating the energy consumption and runtime each epoch, we obtain the total search cost of a configuration, which allows NAS/HPO methods to search optimal models under a limited cost budget.
2.3 BÉZIER CURVE-BASED SURROGATE MODEL
Due to the large EA-HAS-BEE -ench search space, directly training all configurations in the space is infeasible even for small dataset like CIFARIO. As a result, some of the metrics can not be directly measured, including the model performance curve, search energy cost, and runtime. Thus, similar to other recent works (Zela et al. 2022 Yan al., 202T), we develop a surrogate model that expands the entire search space from a sampled subset of configurations.
As for energy cost and training time, we follow the Surrogate NAS Benchmark (Zela et al., 2022) and train LGB (LightGBM) (Ke et al. 2017) models to predict these. However, for learning curve prediction, surrogate models proposed by the existing NAS-Bench are not applicable to EA-HAS- Bench. Since EA-HAS-Beaee B-BnncBe contains various maximum training epochs in the search space, non of the existing surrogate model can cope with dimensionally varying inputs. More importantly, it not possible to directly constrain the learning curve after using the dimensionality reduction operation (e.g.. the error rate should be between and 1). In our early experiments, the implementation of the NAS algorithm the NAS-Bench- X11 surrogate model would yield results with more than 100% accuracy. Therefore, we propose a Bézier Curved-based Surrogate (BCS) Model to solve the above problems.
Convert Learning Curve to Bézier Curve Control Points. For each configuration of network architecture and training hyperparameters, the surrogate model outputs the learning curve containing the accuracy or
of each epoch. Inspired by the success of the Bézier curve in other areas (Liu et al. 2020). we choose the Bézier curve to fit learning curves of arbitrary length. The shape of Bézier curve is entirely determined by its control points, and degree control points correspond to Bézier curve of order n - 1. The Bézier curve can be formulated in recursive way as follows:
P(t) = Σ P,B [0
(2)
i=0
4

---

Published as a conference paper at ICLR 2023
where P. denotes control points, (t) = (1-) and i 0,1,-.....
t) -1
As a result, the task of regressing a learning curve of arbitrary length is simplified to predicting where ey is the enror of the
Bézier curve control points. Given learning curve, {ez, i Cy, e, epoch and m is the maximum epoch, we need to get the optimal control points to generate Quartic Bézier curves fit the learning curve. The control points are learned with the standard least square method. Since the horizontal coordinates of the start and end points of the learning curve and an maximum epoch), the surrogate model only predicts the
fixed (i.e., hrrsuur vertical coordinates of these two control points. An illustration of generated Bézier curves is shown in Figure 7 in Appendix B.4
Surrogate Model Structure. Given a RegNet architecture and hyperparameter configurations, BCS estimates the Bézier curve control points with a neural network. As shown in Figure [T] the proposed Bézier curve-based Surrogate model is composed of hyperparameter encoder En. ar- chitecture encoder E,, and learning curve prediction network f. The training hyperparameter con- figurations represented one-hot vector 2/ and then fed into E,, The RegNet configuration parameters are normalized to values between 0 and 1, concatenated to a vector, and fed into Eg. Finally, the hyperparameter representation and architecture representation are fed into the learning curve predictor to estimate Bézier curve starting/ending points and control points:
(iustart 1 Vend C21 Cy1
C25 Cys } (va) En (v%)
The learning curve predictor consists of two branches. One predicts the vertical coordinates of the starting and ending points of the Bézier curve, and the other branch predicts the other control points.
With the control points obtained, we can generate Bézier curves with equation ( and then obtain the accuracy of each epoch based on the horizontal coordinates ex; of the curve.
2.4 DATASET COLLECTION
Some previous works (Eggensperger et al. 2015) propose to sample more training data from the high-performance regions of the search space because effective optimizer spends most of its time in high-performance regions of the search space. However, this sampling strategy causes a distribution shift between the sampled training data and the actual search space, which hurts the prediction accuracy of the surrogate model. As discussed and verified in recent work (Zela et al., 2022), search space containing hyper-parameters is more likely to produce dysfunctional models which are rarely covered in a sampled sub-set focusing on high-performance regions, and hence purely random sampled training data yields more satisfactory performance. In summary, for EA- HAS-Bench's search space that contains both model architectures and hyperparameters, we use random search (RS) to sample unbiased data to build a robust surrogate benchmark.
The sampled architecture and hyperparameter configurations are trained and evaluated on two of the most popular image classification datasets, namely CIFAR-10 (Krizhevsky et al., 2009) and MicrolmageNet challenge's 's (Tiny ImageNet) dataset (Le & Yang 2015).
2.5 SURROGATE BENCHMARK EVALUATION
Comparison Methods. We compare the proposed method with six parametric learning curve based models (Domhan et al. 2015) (Exp.. ilog2. pow2, log power, log linear, vapor pressure) and three surrogate models (SVD-XGB, SVD-LGB, SVD-MLP) from NAS-Bench-XI1 Yan et al. (20TT) For fair comparison, the parametric learning curve-based model applies the same network structure as our proposed BCS. For lengths (50, 100, and 200 epochs) into the hidden space with the same dimension with three dif- ferent SVDs respectively (although this is not convenient to cope with learning curves of arbitrary length).
we compress learning curves of different
(TPE) (Bergstra et al. (011) is adopted for all surrogate models to search for the best hyperparameter configuration. The details of the experiments and ablation study are in Appendix
Testing Set and Ground Truth (1 seed). All surrogate model methods are evaluated separate testing set trained on two sets random seeds. One the two seeds of the test set is used as the ground truth, and data from the other seed can be seen tabular benchmark baseline (results in
5

---

Published as a conference paper at ICLR 2023
Table 3: Compare Bézier-based Surrogate model with NAS-Bench-XI1 and parametric learning
curve model on CIFAR-10 10 and ImnlenagNet. "GT (1 seed)" means a 1-seed tabular benchmark.
CIFARIO
TinyImageNet
Methods
Avg.R2
Final R2
KT
Final KT
Avg.R2
Final R2
Avg.
Final KT
Parametric learning curve neural network ( (Domhan et al.,
2015
exp3
0.397
0.791
0.769
0.789
-1,128
0.935
0.807
0.849
ilog2
0.799
0.830
0.820
0.830
0.297
0.978
0.879
0.915
pow2
0.212
-0.056
0,564
0.506
0.321
0.396
0.571
0.547
log power
0.195
0.583
0,544
0.519
-1.933
0.872
0.807
0.873
logx linear
0.808
0.825
0,810
0.793
0,779
0.969
0.893
0,906
vapor
0.790
0.671
0.830
0.829
0.897
0.957
0.858
0.883
NAS BAS-Bench--X XII
(Yan al.,
2021
SVD-XGB
0.762
0.731
0.827
0.836
0,890
0.897
0.848
0,862
SVD-LGB
0.838
0.850
0.787
0.795
0,967
0.976
0.899
0,908
SVD-MLP
0.869
0.835
0.859
0.852
0.967
0.972
0.913
0,919
BCS(Ours)
0.892
0.872
0.860
0.841
0.968
0.979
0.922
0,928
GT (1 seed)
0.857
0.821
0.928
0.931
0.979
0.975
0.961
0.962
The sampled configurations on CIFARIO and TinylmageNet are split into training, validation, and testing sets containing 11597, 1288, and 1000 samples respectively.
"GT" row in Table B}
Evaluation Metrics. Following Wen al. 2020). White et al. (2021b) and Zela al. 2022 we use the coefficient of determination R² and the Kendall rank correlation coefficient as the evaluation metrics. These two metries only evaluate the performance based the overall statistics of the curve rather than anomalies. However, a few spike anomalies on a validation curve could significantly affect the final accuracy prediction. As result, we further adopt spike anomalies Yan et al. 2021 as extra metrics ( detailed descriptions in appendix).
Evaluation Results. The performance of surrogate models is shown in Table First, the paramet- ric learning curve-based models function not well fit the real learning curve EA-HAS-Bende and some of the higher order functions even fail to converge, such as pow; (c az ") and pow (c (ax: + b) 9)). This is because the importance of the different parameters in surrogate model varies considerably, especially the parameter which is in the exponent of an exponential function. The percentage of spike anomalies for real BCS is 3.72% and 4.68% CIFAR-I0 and 0.83% and 1.31% on TinyImageNe respectively. We further evaluate the consistency between the real measured energy and the predicted energy cost by the surrogate model. Specifically, on CIFAR- 10, the energy cost surrogate model achieves R2 of 0.787, KT of 0.686, and Pearson correlation of 0.89. On TinylmageNet, achieves R2 of 0.959, KT of 0.872, and Pearson correlation of 0.97.
DIFFERENCE WITH EXISTING NAS BENCHMARKS
Compared with existing NAS benchmarks such as 201 Dong & Yang. 2020), NAS-Bench-301 (Siems et al. (220)) or Surrogate NAS Bench (Zela ct al. 2022), EA-HAS Bennhh has three significant differences.
101 (Ying el. 2019). NAS--Bech--
Diverse and Large Scale Joint Search Space. EA-HAS-Bende is diverse in terms of the types of configurations in the search space, which contains both model architectures and training hyperparameters. Although NAS-HPO-Beat
(Klein Hutter 2019) and HAS---PO-Beeh-ll (Hi- rose et al.) (202T) also consider both architectures and hyperparameters, both benchmarks are based on small and homogeneous search spaces. Specifically, NAS-HPO-Bent focuses only on 2-layer feed-forward network training on tabular data, and the hyperparameters search space of NAS-HPO- Bench-ll only contains learning rate and batch size. Besides the search space, diversity is also reflected in the evaluated performance ranges. As shown in Figure 2 the performance range of DARTS Liu et al. 2019) used by on CIFAR-I0. Although DARTS contains more architectures. the performance of the models in this space is significantly less diverse (Yang et al. 2020). Compared with NAS- sench-101/201 the configurations in
is the smallest for the validation performance
cover much larger performance range.

---

Published as a conference paper at ICLR 2023
curvistive prob.
cumulative prob.
6-3
contii rimertmmax
curnulstive
0.1
mmo apt. S/84
64
confajin rimearmmax opocheadtoye 6.000)
1m 5013 1J55
optim=spt sj110
(mw0.18.2 1...1.34..0
opdm--ddamm5220 2211
emror
enror
(a) Split=Learming Rate
(b) OplizzOptimizer
(c) Split=Max Epoch
Figure 3: The empirical cumulative distribution (ECDF) of all real measured configurations on
TinyImageNet for 3 different splits.
100
Modeling Learning Curves for complex joint space. To the best of our knowledge, NAS-Bench-X11 (Yan et al. (021) the
80
only existing surrogate model that provides the full training sta-
60-
tus over the entire training process. However, PAS-Bench-XI1 is only available for learning curves with fixed maximum epochs. The number of training epochs required for convergence is not always the same for different architectures and it also directly affects the training energy cost. As a result, for a more realistic search space propose BSC predict learning curves
40-
20
N8-201 NB-301 Ours
like FA-HAASBennce with different maximum epochs.
Figure 2: Validation accu- racy box plots for each NAS benchmark in CIFAR-10. The whiskers represent the mini- mum and maximum accura- cies in each search space. The black hollow circles represent outliers
Full-cycle Energy Consumption. Most existing benchmarks use model training time as the training resource budget, which is not an accurate estimation of energy cost. Firstly, the GPU does not function at stable level throughout the entire training process, or it may not even be working at all for a significant time period during training, and hence longer training time does not always mean more GPU usage and more energy cost. Secondly, energy cost not only corresponds to training time but also relates to the energy cost per unit time, which is affected by the architecture and hyperparameter configurations. Our experimental analysis in the next section also verifies that training time and en- ergy cost not equivalent. HW-NAS-Bead but its focus is on the model inference energy cost. On the other hand, EA-HAS- -encchh provides a full-cycle energy consumption, both during training and inference. The energy consumption metric allows HPO algorithms to optimize for accuracy under an energy cost limit (Section 5
(Li et al., (2021b) also considers energy consumption,
4 ANALYSIS ON EA-HAS- -BENH
Impact of Hyperparameter Configurations. Since most existing large-scale computer vision NAS benchmarks focus solely network architectures and apply the same hyperparameters for all models, we examine how different hyperparameters affect the searched model performance. We empirical cumulative distributioo (ECDF) (Radosavovic al. 2020) to assess the quality of search space. Specifically, we take a set of configurations from the search space and characterize its error distribution. Figure shows the empirical cumulative distribution (ECDF) of different training hyperparameters on CIFAR-10 and TinyImageNee of Appendix.) We observe that the learning rate and the choice of optimizer may have the most significant impact on the search space quality. The size of the maximum number of epochs is also positively correlated to the quality of the search space.
(The full version is in the Figure [3
Correlation Between Training Energy Cost and Training time. Figure investigates the re- lationship between the training energy consumption (TEC), training time, and the test accuracy of models in TinylmageNe scattered. This means the correlation between training time and energy cost is not strong. Although training the model for longer period is likely to yield higher energy cost, the final cost still de- pends on many other factors including power (i.c., consumed energy per hour). The middle and right plots of Figure also verifies the conclusion, where the models in the Pareto Frontier the
Firstly, we observe that the points in the left figure of Figure is rather

---

Published as a conference paper at ICLR 2023
350
62
Dest Accuracy
Best Accurney Acc and Rumrine
300
50
Rokue the Farsto hurcas
ACd and Runtime
50
250
Nest Accuracy (%)
Runtinee
37
Test sccuracy
lest Accurscy
40
40
200
20
90
150
25
20
100
10
50
260
500
Training Energy Consumption
Training Energy Consumption
Figure 4: (left) Training time vs. training energy consumption (TEC), color-coded by accuracy. (middle) Test Accuracy vs. TEC. (right) Test Accuracy vs. training time. TEC and training time are the per epoch training energy consumption (Kw*h) and runtime (seconds) on the Tesla V100. The orange cross in the middle plot denotes the models in the Pareto Frontier on the accuracy-runtime coordinate but are the Pareto Frontier on the accuracy-TEC.
runtrme coordinate (right figure) are not always in the Pareto Frontier on the accuracy-TEC coordinate (middle figure), showing that training time and energy cost are not equivalent.
Meanwhile, training model longer (or with more energy) does not guarantee better accuracy. In the middle and right plots of Figure we see that many models with high TECs, still fail train, due to improper neural architectures or hyperparameter configurations. On the other hand, simply searching for the best accuracy might not be cost-efficient, since there are quite few configura- tions with the same level of accuracy. The observation motivates finding the best trade-off between accuracy and energy consumption.
Lr GW WM VWt WC:EPHH
4..00
DanigurattiossAccurrrr//EEEee Correlation. Figure 5 shows the correlation between architecture/ hyperparameter configurations and ac- curacy, runtime, TEC, and inference energy cost (IEC). We observe that hyperparameters like learning rate also have high correlation with model performance, which further verifies the importance to consider hyperparameters in NASBench. Both network architecture like network depth and width and hyperparameters like training epoch has relatively strong correlation with energy cost, showing the importance of consid- ering both perspectives in energy-aware Benchmark.
0.081 0.004
0.12
0.1
0.027 4.015 0.002
0.000
0.2
0.021 -3.01
0.035
0.024
0.27
Epoch Aug Porcy Opiim
0.02 0.002
0.004
0.003
0.002 +0.011
0.000
0.000
-0.043 4.004
-04
5 EA-HAS-BENCH AS A BENCHMARK
Runtime
Correlation between Figure 5: coefficient RegNet+ HPO and Accuracy, TEC, and inference energy cost (IEC) on TinylmageNet.
CA-HAS-Bende saves tremendous resources to train and evaluate the configurations for real. We demonstrate how to leverage the proposed dataset to conduct energy-aware AutoML research with two use cases. Firstly, we evaluate the trade-off between search energy cost and model performance of four singlc-fidelity algorithms: random search (RS) (Li Talwalkar 2019) local search (LS) (White et al., 2020), regularized evolution (REA) (Real et al. 2019). BANANAS (White et al.) 2021a).
Runtime,
and two multi-fidelity bandit-based algorithms: Hyperband (HB) (Li
et al. 2017) and Bayesian optimization Hyperband (BOHB) Falkner et al. tion details of the above algorithms are in Appendix D Then, in the second sub-section, as the first step toward a long journey of energy-awant AutoML, arm several existing AutoML algorithms with another energy-related objective and verify the effectiveness energy-aware AutoML on our hyperpprraeteeracchiteeeuee benchmark.
2018 The implementa-
8

---

Published as a conference paper at ICLR 2023
EA-HAS-Beeme CIFAR1O
BAvHAS-Bench en TinylmageNet
10
Sest Regge
Acouracy
0.996
20
0.964
BAMAVAS
0.962
GAMANAS DONB
REA 15
60+B
REA
10"
10
10
15
10f
R5
10"
0.960
Energy
Energy
(kWhh)
Figure 6: NAS results CIFARIO (left) and TinyImageNet (middle). Energy-aware BANANAS and Energy- Local Search (LS) vs. origin BANANAS and on CIFARIO (right).
5.1 BENCHMARKING EXISTING ALGORITHMS
Experimental setup. Since AL-HAS-Bencls focuses the trade-off between model performance and search energy cost, in this experiment we use the total search energy cost as the resource limita- tion, instead of training time. As result, we the maximum search cost roughly 40,000 kWh for CIFAR 10 and 250,000kW Wh for TinyImageNet, which is equivalent to running a single-fidelity HPO algorithm for about 1,000 iterations.
Results. In Figure 6 left and middle, we compare single- and multi-fidelity algorithms on the search space of EA-HAS-Bench. For single-fidelity algorithms, LS is the top-performing algorithm across two datasets. This shows that similar to NAS-Bench, MAS-Bench with joint search space also has locality, property by which "close" points in search space tend to have similar perfor- mance.
5.2 A NEW ENERGY-AWARE HPO BASELINE
Energy Aware HPO Most existing HPO methods do not directly consider the search energy cost. In order to verify the importance to consider the energy cost during the search process, we propose energy-aware HPO baseline by modifying existing HPO methods. Following MnasNet (Tan) et al. 2019), we modify the BANANAS (White et al., 202 and LS (White et al. 2020) by chang- ing the optimization goal a metric that considers both accuracy and energy cost: ACCx (TEC) ) where To is the target TEC. For CIFAR1O, we the To to 0.45 and -0.07.
Experimental Setup and Result We explore another important usage scenario, where the goal is to achieve a target model performance using as little energy cost as possible. As result, we use model performance rather than energy consumption as resource limitation and stop search when the model hits target performance and compare the corresponding search energy cost. For CIFARIO, the target accuracy is set to 97%. As shown right, EA algorithms that consider TEC save close to 20% of search energy consumption compared to the origin algorithms in achieving the target accuracy. The ablation study is shown in Appendix D.4
CONCLUSION
EA-HAS-Bende the first large-scale mergy-aware hyperparameter and architecture search bench- mark. The search space of EA-HAS-Beas -eenchh consists of both network architecture scale and training hyperparameters, covering diverse configurations related to energy cost. A novel Bézier curve-based surrogate model is proposed for the new joint search space. Furthermore, between existing NAS benchmarks and EA-HAS- -ennh and dataset statistics and the correlation of the collected data. Finally, provide algorithms can save significant energy in the search process. We expect that our expedites and facilitates the EA-NAS and HAS research innovations.
analyze the difference
cases of EA-HAS-Beat
to show that energy- aware

---

Published as a conference paper at ICLR 2023
7 ACKNOWLEDGMENTS AND DISCLOSURE OF FUNDING
S. Dou and C. Zhao acknowledge that the work was supported by the National Natural Science Fund of China (62076184, 61976158, 61976160, 62076182, 62276190), in part by Fundamental Research Funds for the Central Universities and State Key Laboratory of Integrated Services Net- works (Xidian University). part by Shanghai Innovation Action Project Science and Technology (20511 100700) and Shanghai Natural Science Foundation (22ZR 1466700)))) We thank Yuge Zhang (Microsoft Research Asia) for suggestions on the design of the search space and for revising the writing of the paper. We thank Bo Li (Nanyang Technological University) for help with the codes.
REFERENCES
Archit Bansal, Danny Stoll, Maciej Janowski, Arber Zela, and Frank Hutter. Jahs-bench-201: A foundation for research on joint architecture and hyperparameter search. In Thirty-sixth Confer- ence on Neural Information Processing Systems Datasets and Benchmarks Track.
Gabriel Bender, Hanxiao Liu, Bo Chen, Grace Chu, Shuyang Cheng, Pieter-Jan Kindermans, and Quoc V Le. Can weight sharing outperform random architecture search? an investigation with tunas. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14323-14332, 2020.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parameter optimization. In Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information P Processing Systems 2011. Proceedings of a meeting held 12-14 December
2011, Granada, Spain. pp. 2546-2554, 2011.
Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. Speeding up automatic hyperparam- eter optimization of deep neural networks by extrapolation of learning curves. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015, pp. 3460--468. AAAI Press, 2015.
Xuanyi Dong and Yi Yang. Nas-bench-201 Extending the scope of reproducible neural architecture search. In Sth International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL /tttp:://epenreeee
n
FFormmid=Hie 2a..mm221dd.xxx2XXXXX
Xuanyi Dong, Mingxing Tan, Adams Wei Yu, Daiyi Peng, Bogdan Gabrys, and Quoc V Le, Auto- has: Efficient hyperparameter and architecture search. arXiv preprint arXiv:2006.00 036566 2020.
Katharina Eggensperger, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Browe Efficient bench- marking of hyperparameter optimizers via surrogates. In Blai Bonet and Sven Koenig (eds.), Pro- ceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, Junuary 25-30, 2015, Austin, Texas, USA, pp. 1114-1120. AAAI Press, 2015.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. J. Mach. Learn. Res., 10:55:1-55:21, 2019.
Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: mization at scale. In Proceedings of the 35th International Conference on Machine Learning, ICML, volume 80 of Proceedings of Machine Learning Research, pp. 1436-1445. PMLR, 2018.
robust and efficient hyperparameter opti-
Matthias Feurer and Frank Hutter. Hyperparameter optimization. In Automated machine learning, pp. 3-33. Springer, Cham, 2019.
Xin He, Kaiyong Zhao, and Xiaowen Chu. Automl: A survey of the state-of-the-art. Knowledge- Based Systems, 212: 10622,, 2021.
Yoichi Hirose, Nozomu Yoshinari, and Shinichi Shirakawa. Nas-hpo-bench-ii: A benchmark dataset on joint optimization of convolutional neural network architecture and training hyperparameters. In Vineeth N. Balasubramanian and Ivor W. Tsang (eds.), Asian Conference on Machine Learn- ing. ACML 2021, 17-19 November 2021. Virtual Event, volume 157 of Proceedings of Machine Learning Research, pp. 1349-1364. PMLR, 2021.
10

---

Published as a conference paper at ICLR 2023
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang. Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. In Isabelle Guyon, Ulrike von Luxburg. Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp- 3146-3154, 2017.
Aaron Klein and Frank Hutter. Tabular benchmarks for joint architecture and hyperparameter opti- mization. CoRR, fos//195504990, 2019. URL
04970
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3 2015.
Bo Li, Xinyang Jiang. Donglin Bai, Yuge Zhang, Ningxin Zheng, Xuanyi Dong, Lu Liu, Yuqing Yang, and Dongsheng Li. Full-cycle energy consumption benchmark for low-carbon computer vision. CoRR, abs/2108. 1 13465, 2021a.
Chaojian Li, Zhongzhi Yu, Yonggan Fu, Yongan Zhang, Yang Zhao, Haoran You, Qixuan Yu, Yue Wang, Cong Hao, and Yingyan Lin. Hw-nas-bench: Hardware-awar aware neural architecture search benchmark. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021b.
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019, volume 115 of Proceedings of Machine Learning Research, pp- 367-377 AUAI Press, 2019.
Lisha Li, Kevin G. Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar, Hyper- band: A novel bandit-based approach to hyperparameter optimization. Mach. Learn. Res., 18: 185: 188:1-18 2017.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Yuliang Liu, Hao Chen, Chunhua Shen, Tong He, Lianwen Jin, and Liangwei Wang. Abenet: Real-time scene text spotting with adaptive bezier-curve network. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, pp. 9806-9815, 2020.
Yash Mehta, Colin White, Arber Zela, Arjun Krishnakumar, Guri Zabergja, Shakiba Moradian, Mahmoud Safari, Kaicheng Yu, and Frank Hutter. Nas-bench-suite: NAS evaluation is (now) surprisingly easy. In The Tenth International Conference on Learning Representations, ICLR, Virtual Event. OpenReview.net. 2022.
Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüssan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 4092- 4101. PMLR. 2018.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross B. Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10425-10433, 2020.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image classifier architecture search. In The Thirly-Thrrd AAAI Conference on Artificial Intelligence, AAAL, pp. 4780-4789. AAAI Press, 2019.
Michael Ruchte, Arber Zela, Julien Niklas Siems, Josif Grabocka, and Frank Hutter. Naslib: a modular and flexible neural architecture search library. 2020.
11

---

Published as a conference paper at ICLR 2023
Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green ai. Communications of the ACM, 63(12):54 63, 2020.
Julien Siems, Lucas Zimmer, Arber Zela, Jovita Lukasik, Margret Keuper, and Frank Hutter. Nas- bench-301 and the case for surrogate benchmarks for neural architecture search. arXiv preprint MwXiv:2008.09777 2020.
David R. So, Quoe V. Le, and Chen Liang. The evolved transformer. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, Cali- fornia, USA, volume 97 of Proceedings of Machine Learning Research, pp. 5877-5886. PMLR, 2019.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv: meoe:o:200.044444 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2820-2822 2019.
Wei Wen, Hanxiao Liu, Yiran Chen, Hai Helen Li, Gabriel Bender, and Pieter-Jan Kindermans. Neural predictor for neural architecture search. In Computer Vision ECCV 2020 16th Euro- pean Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIX, volume 12374 of Lecture Notes in Computer Science, pp. 660-676. Springer, 2020.
Colin White, Sam Nolen, and Yash Savani. Local search is state of the art for NAS benchmarks. CoRR, abs/2005.02966 2020.
Colin White, Willie Neiswanger, and Yash Savani. BANANAS: bayesian optimization with neural architectures for neural architecture search. In Thirty-Fiftt AAAI Conference on Artificial Intelli- gence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 10293-10301. AAAI Press, 2021a.
Colin White, Arber Zela, Robin Ru, Yang I Liu, and Frank Hutter. How powerful are performance predictors in neural architecture search? In Advances in Neural Information Processing Sys- tems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 28454-28469, 2021b.
Bichen WU, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10734-10742, 2019.
Shen Yan, Colin White, Yash Savani, and Frank Hutter. Nas-hench-x11 and the power of learning curves. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems, pp. 22544, 2021.
Antoine Yang, Pedro M. Esperança, and Fabio Maria Carlucci. NAS evaluation is frustratingly hard. In 8th International Conference on Learning Representattons, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. Openkeview.net, 2020.
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas- bench-101: Towards reproducible neural architecture search. In Proceedings of the 36th Interna- tional Conference Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 7105-7114 7114. PMLR, 2019.
Arber Zela, Julien Niklas Siems, Lucas Zimmer, Jovita Lukasik, Margret Keuper, and Frank Hutter. Surrogate nas benchmarks: Going beyond the limited search spaces of tabular nas benchmarks. In International Conference on Learning Representations, 2022.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint 1rX15:1611.01178, 2016.
12

---

Published as a conference paper at ICLR 2023
A RELATED WORK
A.1 EXISTING NAS BENCHMARKS
Tabular NAS Benchmarks. While neural architecture search (NAS) has succeeded in various practical tasks such as image recognition (Pham et al. 2018) and sequence modeling (So et al. 2019), the non-reproducibility of NAS has been contested (Li & Talwalkar 2019). One of the main reasons that complicate NAS reproducibility studies is the high computational cost as well as the large carbon emissions that result(Li et al.) 2021a). For the reproducibility of NAS research, NAS- Bench-101 (Ying et al. 2019) and NAS-Bench-201 (Dong & Yang. (020) have been proposed. Due the success of these tabular benchmarks in image classification, corresponding benchmarks have been proposed in areas such as NLP and speech recognition (Mehta et al. 2022).
Surrogate NAS Benchmarks. The recent Surrogate NAS Benchmarks (Zela et al. 2022) builds surrogate benchmarks on a realistic search space and shows strong generalization performance. To extend the surrogate model, NAS-Bench-XI1 (Yan al. 2027 uses singular value decomposition and noise modeling to output the learning curve.
NAS Benchmarks joint HPO Most NAS Benchmarks have fixed training hyperparameters, but the training hyperparameters strongly influence the performance of the model obtained by NAS
(Dong al. 2020). To alleviate this problem, NAS-HIP--Bnnch (Klein & Hutter 2019) and NAS-
HPO-Bench-II Hirose et al. (202T) are proposed. However, as shown Table I the size of the two Benchmarks small and the search spaces are simple. The architecture space of NAS-HPO- HPnhhBnchh is multi-layer perception (MLP) trained on the tabular datasets for regression tasks. NAS-HPO-
Bench-II only really measures 12 epochs using the CIFAR-I0 dataset Krizhevsky et (2009) and the training ours is hyperparameters space only contains learning rate and batch size. A similar work to 201 (Bansal et al.)). also with large-scale joint search space. provides FLOPS, latency, and runtime in addition to performance and loss. However, -AHS-Bench- 201 does not focus on energy consumption during the search.
A.2 RESOURCE-AWARD NAS
Early NAS algorithms Zoph Le 2016 Real et al. 2019) focused only on performance and ig- nored some of the associated hardware consumption. To this end, many resource-aware NAS algo- rithms are proposed to balance performance and resource budgets (He et 2021). These resource- aware NAS algorithms focus on the four types of computational costs that are included the FLOPs, parameter size, the number of Multiply-ACcumuln the NAS algorithms, two classic works are MnasNet (Tan al. 2019) and FBNet (Wu et al. 2019). MnasNet proposes a multi-objective neural architecture search approach that optimizes both accu- racy and real-world latency on mobile devices. Similar to MansNet, FBNet designs a loss function to trade off the cross-entrope loss and latency of architecture. However, none of the above NAS algorithms focus on the huge energy consumption in search.
(MAC) operations, and real latency. Among
B MORE DETAILS OF SECTION 2
B. MORE DETAILS ON EVALUATION METRICS
Details of Spike Anomalies. Although R2 and KT can evaluate the surrogate model by measuring the overall statistics between the surrogate benchmark and the ground truth, they are not sensitive to anomalies. Following, NAS-bench-XI Yan et al. (202I), to evaluate the performance of surrogate models based on anomalies, we introduce the Spike Anomalies metrics. We first calculate the largest value & such that there are fewer than 5% of learning curves whose maximum validation accuracy is higher than their final validation accuracy, the true learning curves. Next, the percentage of surrogate learning curves whose maximum validation accuracy is r higher than their final validation accuracy was computed.
13

---

Published as a conference paper at ICLR 2023
B.2 MORE DETAILS ON BÉZIER CURVE-BARS BASED SURROGATE MODEL
Network Structure of BSC. The encoder of architecture and hyperparameters adopts a simple Multi-Layer Perceptron (MLP) structure, consisting of two linear layers with ReLU activation func- tions. Then, the encoded features are fed into the learning curve prediction network, which is also an MLP with an extra dropout layer, whose output fed into two linear regressors that output the coordinates of the control points. We the sigmoid activation function the regressor, which directly constrains the initial final performance between and The control points are learned with the standard least square method follows:
Bo,s (to)
B5,5 (to)
Prstart Cz1
Vatart Cyn
Cur Cri
eoo Cyr
Bo,s (t1)
B5,5 (f1)
(4)
Bo,5 (tm)
B5,5 (fm)
Wwond
Eyend
ex
eym
Details on the Compared Parametric Learning Curves based Methods. Several parametric learning curves-based methods are selected as the comparison methods. The detailed formulation of those parametric models is shown in Table 4 Following Domhan al. (2015), we first try the Levenberg- -arbbuurdd algorithm and fall back to Broyden- Giolaae Shanoo (BFGS) in case that fails. In the experiment, found that the initial parameters process. Some parametric models of a high order cannot be fitted to the learning curve in the training set because suitable initial parameters cannot be found. In contrast, the initial point of the BSC model is the starting and ending point of the learning curve.
important in the fitting
Table 4: The formula of parametric learning curve
Reference name
Formula
exps
exp
ar
+ b)
iloga
Jog
pow2
ara
log power
I+(#) alog(a (x) +
logx linear
exp a + b + c clog(r))
vapor
Hyperparameters Surrogate Models. Table 3 shows the optimal hyper-parameters searched by TPE for different surrogate models. Due to the page limit, here we only listed the hyperparame- ters of the three models that achieve the best performance in Table We used a fixed budget of 500 trials for all surrogate models and average R2 as the optimal target.
Surrogate Models of Runtime Metrics Although our benchmark focuses on energy consump- tion. we also provide runtimes to allow NAS methods to use runtimes as budgets. We train an LGB model with the average runtime of each epoch as target and the model achieves 0.926 for R2 and 0.849 for KT on runtime prediction on CIFARI0.
B.3 MORE DETAILS ON DATA COLLECTION
Here we provide a more detailed introduction to energy consumption measurement for data col- lection. Intuitively, the search energy cost is the total energy consumption to complete search algorithm. Since the majority of the energy cost comes from training each deep model the search algorithm traverses, in our dataset, the search energy cost is defined the total energy cost (in kWh) or time (in seconds) to train the model configurations traversed by the search algorithms.
Specifically, denote training configuration in the EA-HAS-Bench search space as Nd. where e is d-dimentional vector containing d training parameters. eep(c) is the energy cost mea- sure function that returns the training energy cost train a model with training configuration e for epoch. A (c(0) the set of configurations NAS/HPO search method traversed. As
14

---

Published as a conference paper at ICLR 2023
Table 5: Hyperparameters of the surrogate models and the optimal values found via TPE.
Model
Hyperparameter
Range
Type
Optime Value
Num. components
[1.20]
uniform
Num. rounds
constant
3000
Early Stopping
100
Max. depth
[1.24]
uniform int
9
Num. leaves
[10, 100]
uniform int
84
SVD-LGB
Min. child weight
[0.001, 10)
log uniform
0,4622
Lambda I.I
[0.001, 1000]
log uniform
0.0056
Lambda L2
[0.001, 1000]
log uniform
0.0054
Boosting lype
constant gbdi
Learning rate
[0.001, 0.1]
log uniform 0.5822
Num. components
[1.20]
uniform 3
Num. epochs
[5,200]
uniform
hidden dim
[32, 250]
SVD-MLP
Num. layers
constant 4
Learning rate
[0.0001, 0.1]
log uniform
0.0008
drop out
[0.0, 0.1, 0.2, 0.3, 0.4, 0.5]
uniform int
0.2
batch size
[64, 128, 256]
uniform int
64
Num. epochs
[5,200]
uniform int
180
hidden dim Num. layers Learingg rate
[32, 250J
constant
247
BSC
constant
4
[0.0001, 0.1]
log uniform
0.0003
drop out batch size
10.0. 0.1. 0.2. 0.3, 0.4, 0.5]
uniform
[64, 128, 256]
uniform
64
Num. rounds
3000
Early Stopping
constant
100
Max. depth Num. leaves Min. child weight
[1,100]
uniform int
31
[10, 1000]
uniform int
315
LGB-E
[0.001, 10]
log uniform
0.0046
Lambda L1
[0.001, 1000]
log uniform
20.3216
Lambda 1.2
[0.001, 1000]
log uniform
11.6866
Boosting type
constant gbdt
Learning rate
[0.001, 0.1]
log uniform
0.0451
result, the total search energy e, cost is defined as:
(A(A) = Σ (ep(c! c
(5)
efiEA
where n the index of that stores the number of total training epochs to train the deep model under configuration
Next, we introduce how to measure the per epoch energy consumption for different training configu- rations. Following [i et al. (2021a), we collect the real-time power of the GPU during the operation of the algorithm through the interface of pynvml. In the following, we will provide an implementa- tion of the GPU tracker accurately describe its functionality.
import re
import subprocess
import threading
import time
import рyпvml
import torch import xmltedict
10
class Tracer (threading Thread))
11
def _init_ (self, gou_num= (0,), profillig__ineevaloo..ll 1):
12
threading. Thread. init iisi__iiit______________________
15
def run (sclf):
16
pynvml . nv
17
handle pynvml wwwpp//.ww.eeviice.eppppcm www.eeeeeeeettttandeee...eeee (0)
lower_list while self runninnu
15

---

Published as a conference paper at ICLR 2023
self. counters
21
power_u_info pynvml www..w/www....ooo.rowweeggggggg (handle)
22
power_list.app append (power_u
23
time.sleop (sclf. htt.::.........iinnnnooteeaa..
class GPUTracer:
26
all mode =I normall)
27
def _nit__(eef,, mode, gpu (0,), inotiiiing_itterv---00
verbose=False):
it not mode in GFUTracer.aL) modes: mode:::s::
raise Valuekrror (f' Invalid mode {mode}()
30
self.mode mode
31
gou_num
self profiingg i interval = profiling
self.verbose
34
35
def wrapper (sclf, *args, **kwargs);
36
if not GPUTracor. GPUTracor.1s_cnabl. is_cnable
return self. func (+args, **kwargs), None
tracer Tracer (wr.arrggpuuuumslle....uu profiling_intervai]
profiliingiinerralleed
39
start torch.cuda . Rvent (enable (enable_tiimnnggrrre)
40
end = torch. cuda. Event (enable Satable_timing-True start.
42
tracer.start
43
results self. func (*args, **kwargs)
44
tracer. tercentterminate
end.
47
48
if ctracr.counteed == 0:
49
print ("*" 50)
print ("No tracing infe collected, incrcasing sampling rate if
needed.
51
print 50)
52
tracer.join()
return results, None
else:
55
tracer.join(
56
power, avg temperature, avg_gpu utils, avg. mem utils, total total_power, total gpu
qpu_utils, utils, total mem utils tracer. communicate ()
time_elapse start. elaat.elapsed__inee (end) / 1000 energy_consumption time_clapse avg_power 3600
58
Listing 1: GPU Tracer
Specifically, we implemented this tracer using Python's decorator function and then just logged the GPU information at runtime. In the following, we provide user case for collecting energy data.
BGFUTracer (mode=' normal', verbose=True) deff train_epoch (loader, model, ...): "**performs one epoch of training. ...
BGFUTracer (mode=' normal', verbose=True)
7 @torch.no
grad() ()
def test spoch (loader, model, ....):
**#Evaluates the model on the test set. .**
Listing 2: GPU information Collection by GPUTracer
The details of the machines used collect energy consumption are in Table
16

---

Published as a conference paper at ICLR 2023
Laont
68
Betier Curve
Boaer
Nerker Curve
38
30-
22
12
-12
100
-20
50
100
10
(e) Cubic Bazier Curve
(b) Quarlic Bezier Curve
(c) Quimite Basier Curree
Figure 7: Visualization of generated Bézier Curves and original learning curves (Ground Truth).
Table 6: Details of the machines used to collect energy consumption
Property Name
Value
CPU
Intel(R) Xcon(R) CPU ES-2690 v3 @ 2.60GHz 2600 MHz
Memory-GB
112
Operation system
Linux Ubuntu 20.04 LTS
Hard drive-GB
1000
GPU
Nvidia Tesla V100 with 32 GB memory
B.4 MORE ABLATION STUDY ON SURROGATE MODEL
The effect of Degree n In Table 7 we show the effect of different degrees of the Bézier curve on the prediction performance. The higher the degree, the better the Bézier curve fits the real learning curve, but it also leads to overfitting. As the degree increases, the prediction performance becomes worse instead. When the Degree is 4, the Quadratic Bézier curve achieves the best results.
Bézier curves vs. Polynomial functions Compared to general n order polynomial functions, the coefficients of the Bézier Curve are explainable and have real-world semantics (i.e. the control points that define the curvature). As result, we can leverage the prior knowledge of the learning curve by adding constraints to the control points and fitting a better learning curve. For example, in our implementation, we constrained the starting and ending points of the learning curve to make the accuracy value stay within the [0, 1] range.
Empirically, conduct ablation study in which instead of predicting the Bézier Curve control points, directly predicts the coefficients and intercept of polynomial functions. However, we observe that for polynomial functions of higher order (n=4), the model is almost impossible fit. The possi- ble reason that the scales the parameters differ too much, and the magnitude of the coefficients varies widely, making it difficult to learn the model. When in Table g In contrast, regardless of the order of Bézier's curve, the size of the control points is basically in the same order of magnitude and the model can be easily fitted (as shown in Table 7)
set
2, the results
shown
B.5
API ON How TO USE EA-HAS-BENCE
Here is an example of how to AA-HAA-BBenhh
def @et__a_has_bennh__ee (dataset):
full_api = ()
# load the ea-nas-bench surrogate models
if cataset=="cifrr1""
sa_bas_bench_modea [cad_ensemble(' htwccpooott/eeeeessssaooooo
())
train_energy_modee Load_nnsmblle checkpolntssea_hassaoooo trainE-v0.2')
test_energy_mooet ioad_ensembl 1e checkpof nts.ea hats/eaahas_bencheeeeeeee
vv).1))
17

---

Published as a conference paper at ICLR 2023
Table 7: The predicted performance of different degrees the Bézier Curve on CIFAR-
Method
Avg R2
Final R2
Avg KT
Final KT
Cubic Bézier (Degree = 3)
0.870
0.859
0.858
0.855
Quadratic Bézier (Degree 4)
0.892
0.872
0.860
0.841
Quintic Bézier (Degree = 5)
0.891
0.862
0.855
0.834
Sextic Bézier (Degree
0.852
0.796
0.771
0.736
Table 8: The polynomial function model on CIFAR10
Degree
Avg.R2
Avg.KT
Final.R2
Final.KT
n=2
0.0437
0.547
-2.66
0.182
runtime model Sead_ensemlle" checkkointsseea has renche-untime- v0.1')
elif dataset=
ca_has_bench_meee lcad_ensemble tisckponntt/aa--ssoooootr- -v0.2')
12
crain_energy_model Load_ensemble checkpoints/ee-nss-beee trainš-v0.1')
test_energy_mont modell Load_ensemble htttee/pttttttteeeeesstteeee ha
-v0.1')
14
runtime model boad_ensemble hacttooitt//a--a---ee-------- v0.1')
15
16
full ['ea_pp bench m mea_has_bench_modll runtime_modei brain_energy_mode) dest_energy_mode
17
return full_api
api get has bench api ("cifar10")
20
21
output the learning curve, train time, TEC and IBC
22 1s ca_api ca_caa_hsssaaaaa waaaaaaoncc_moeelleollloeeo [[0] predict (config-arch_stti
train_time ea ea_api @ea_has_bench_meeeee
[1] prreiitt (configearch_strr)
train_cost ea_api I'ea_ ['ea_has_benchh______aaaaaaaaa__
[2] pprddict (config=arch_strr
25 test_ ea_api ['ea has s____encchheeee [ predict (configearch_stry
Listing 3: EA-HAS-Bent API
C MORE ANALYSIS ON EA-HAS-BENT
C.I MORE ANALYSIS ON IMPACT OF HYPERPARAMETER CONHGURATTONS
Table 13 compares how different hyper-parameter configurations affect search space quality on both CIFARIO and TinylmageNe datasets. Besides the learning rate, optimizer, and the number of total training epochs discussed in the main paper, here we further examine the influence of learning rate policy and data augmentation. For the investigation of data augmentation, found that cutout achieves opposite effects on different datasets.
C.2 COMPARING WITH ORIGINAL REGNET SPACE
As shown in the figure we compare the learning curves of one hundred samples actually mea- sured from under two search spaces-RegNet and RegNet+HPO. The different architectures con- verge effectively under well-designed training hyperparameters. However, many architectures fail to converge under the search space of RegNet+HPO. But the latter may find better combination to produce better performance. With fixed training hyperparameter, the best model obtained by searching only in the network architecture space not necessarily the true best model. At these fixed training hyperparameters, the sub-optimal model obtained by searching may get better performance under another set of training hyperparameters. In addition, the optimal hyperparameters difficult
18

---

Published as a conference paper at ICLR 2023
Error
60
40
20
Step
500
IK
1.5K
2K
(a) RegNet
Error
60
20
Step
500
1K
1.5K
2K
(b) RegNet
НPO
Figure 8: Comparing the learning curve under RegNet and RegNet+HPO search spaces. (a) The RegNet search space. (b) The RegNet HPO search space.
determine when facing a new dataset, while searching for training hyperparameters and model architecture is more realistic scenario.
C.3 MORE ANALYSIS ON ENERGY COST
Power Distribution runtime Aug-power. As shown Figure 9 since the power of different models trained on Tesla V100 is not constant, the energy consumption is not linearly related to the runtime.
The correlation between energy consumption and runtime energy
Distribution of Average Power on CIFAR10
Distribution of Average Power on TinylmageNet
1750
1750
1500
1500
1250
500
250
8030
0.035
0.000
800o 3.005 0.010
0.050 Average Power
0.020
0.025
0.035
0.040
Figure 9: Visualization distributions of power on CIFARIO (left) and TinyImageNet (right).
19

---

Published as a conference paper at ICLR 2023
Correlation Between Training Energy Cost and FLOPs. Although HW-NAS- BenshBBechhhh has dis- cussed the relationship between FLOPs and energy in detail on six different hardware devices and concluded that the theoretical hardware consumption does not always correlate with hardware sumption in both search spaces, we still analyze the correlation between the two our search space. The Kendall Rank Correlation Coefficient between FLOPS and TEC TinyImageNet is 0.396 which is less than 0.5, As shown in Figure I0 we observe that the distribution of scattered points triangular rather than linear. Based on the qualitative and quantitative results, the correlation between FLOPs and TEC is not strong.
Overall Energy Consumption of Sampled Configurations build EA-HAS-Bence in Table
The total energy consumption to
Table 9: The energy consumption (kWh) to build EA-HAS-Bench
Dataset
Tranining & Validation Testing sets GT(1 seed)
Total
CIFARTO TO
660,313k 1,715,985
46,813
707,126
Tiny ImageNet
124,088
1,840,074
Total
2,547,200
1e8
3.0
2.5
1.0
0.5
0.0
3
Training Energy Consumption
Figure 10: FLOPs vs. training energy consumption(TECC on TinylmageNet.
D MORE DETAILS OF SECTION 5
D.1 INTRODUCTION ON COMPARED NAS/HPO METHODS
All algorithms in Section 5 are based NASLib (Ruchte al., 2020). For regularized evolution and local search, we modily them to suit our joint search space. The algorithm description and implementation details are as follows.
Random Search is a classic baseline of HPO algorithms and is the basis for some complex algorithms (Li & Talwalkar 2019). For EA-HAS-Bendd we sample randomly both the architecture and the hyperparameter space. The sampling type is the same as the dataset that build for training the surrogate model.
Local search iteratively evaluates all architectures in the neighborhood of the current best architecture found so far (White et al. 2020). For the search space of AA--HA--Bendh we define neighborhood as having the same network architecture the same training hyper- parameters, i.e., points with the same subspace are neighboring points.
Regularized evolution mutates the best architecture from sample of all architectures evaluated so far. For EA-HAS EA-HAS-Benchh we define a mutation as a random change of one dimension in the architecture hyperparameters.
20

---

Published as a conference paper at ICLR 2023
BANANAS is based Bayesian optimization and samples the next point by acquisition function (White et al. 2021a). We used a modified version from NAS-Bench. to con- struct the candidate pool by mutating the best four points 10 times each.
Hyperband based on the random search with successive halving (Li et al. 2017). D. Since the space size of EA-NAS-Bees S-Bench is x 1010, expect explore more points and set the maximum budget to 512 and the minimum budget to 4.
Bayesian optimization Hyperband is based Hyperband with Bayesian optimization (Falkner et al. 2018). We use the same parameters as Hyperhand.
D.2 MORE DETAILS ON EXPERIMENTAL. SETUP
Following Ying et al. 2019) and Yan et (202T) during the architecture search we keep track of the best architectures found by the algorithm after each evaluation and rank them according to their validation accuracy. When the metric we specify (e.g., total energy consumption or target accuracy) exceeds the limit we set, stop the search. After the search, we query the corresponding best accuracy of the model. We then compute regret:
regref; Acc; Acc
where Ace; denotes the accuracy of the best architecture after each evaluation i and Acc denotes the model with the highest average accuracy in the entire dataset. For experiments in section S we run trials each AutoML algorithm and compute the mean and standard deviation.
0.570
0.565
Accuracy
0.555
0.550
BA-AAAANAS
BANANAS
EA-LS
0.545
2500
5000
7500
10000
12500
15000
Energy cost (kWh)
Figure 11: Energy-aware BANANAS and Energy-aware LS vs. origin BANANAS and LS on Tiny- Imaget.
D.3 MORE RESULTS ON ENERGY-AWARE BASELINES
We also conduct experiments to evaluate the proposed energy-aware baselines the TinylmageNet dataset. Specifically, we set T to 2.5 and target performance to 57%. The result is shown in Figure Same the experiment on CIFAR-I0, compared with the baseline without the energy cost penalty, our EA algorithm costs significantly lower search energy to reach the target accuracy.
D.4 ABLATION STUDY ON ENERGY-AWARE BASELINES
In this section, we conduct a more comprehensive empirical evaluation of the proposed new energy- aware baselines. Specifically, ablation study is conducted to examine how different reward func- tions and hyper-parameterd in the proposed objective affect the performance the energy enargy-aware aware HPO.
21

---

Published as a conference paper at ICLR 2023
Choice of the reward function. Inspired by existing work joint optimization of accuracy and latency (Tan et al.) 2019) Bender et al. 2020). we modify the existing NAS methods (LS and BA- NANAS) by using molti-objective optimization functions, including soft exponential reward func- tion (SoftE), hard exponential reward function (HardE), absolute reward function (Absolute). The soft exponential reward function is expressed
r(a) Acc(o) x (T(o)/Th)
where Acc(a) denotes the accuracy of candidate architecture a, T(a) is its training energy con- sumption, To denotes the TEC target control the total TEC in the search process, and / < 0 is the cost exponent. The hard exponential reward function imposes "hard" limit constraint:
{ Acc(a), [((a) x (T(a)/Ta)",
if T(a) S To T( > To
r(a:)
(8)
The absolute reward function aims to find configuration whose TEC is close to Tu:
r(a:) = Acc(a) + (TT(o) 1|
(9)
For the three functions, we set TEC target Tu = 0.45 and / = 0.007 In Table 10 we compare the total energy required by these methods achieve the target performance. Since SoftE requires the least amount of energy, we choose SoftE to convert the current NAS algorithm to the EA-NAS algorithm.
Table 10: The total TEC in reaching target performance on CIFAR10
Algorithms
Origin
Soft Exp
Hard Exp
Absolute
Local Search
5,521
3,218
3,595
6,070
BANANAS
4,966
3,630
5,227
5,227
Choice of the scaling. Following MnasNet, we simply set the beta to -0.07. Based the TEC distribution sampling points mainly between 0.4 and 0.5, we try three different parameters. The results shown in the table. After To 0.45, the final Total TEC is the same, which indicates that EA-NAS is robust to the parameter To-
Table 11: The impact of different To in CIFAR10 Algorithms To =0.4
To=0.45
To =0.5
EA-BNANAS
3,692
3,630
3,630
EA-LS
4,751
3,218
3,218
E SMALL REAL TABULAR BENCHMARK
Besides providing large-scale proxy benchmark and the tens of thousands of sampling points used to construct it, we also provide a small real tabular benchmark. As shown in Table [2] we redefine very small joint search space with a size of 500. As with the previous tabular benchmark, evaluate all models within this space.
NAS algorithms Tabular Benchmark. Similar Section 5.1
implement NAS algo- rithms on the small tabular benchmark. In this experiment, the maximum search cost is set to 20,000 kWh, which equivalent to running a single-fidelity HPO algorithm for about 200 iterations. We run 10 trials of each AutoML algorithm and compute the mean and standard deviation. The result shown in Figure 12, Due the different search spaces and budgets, the conclusions drawn differ slightly from the previous ones on the surrogate benchmark.
22

---

Published as a conference paper at ICLR 2023
Table 12: Overview of the toy search space
Type
Hyperparameter
Range
Quantize
Space
Depth d
[.15]
10
[80. 112]
8
5
RegNet
Wa
20
Wmn
2.75
Group Width
16
Total of Network Architectures
Learning rate
0..011, 0.003, 0.005, 0.01, 0.03, 0.05, 0.1, 0.3, 0.5, 1.0)
Max epoch
100)
Optim
Decay policy
"cus'"
Optimizer
sgd
Training
Data augmentation
None
Total of Hyperparameter Space
10
Tabular Benchmark CIFAR10
2 10
10-2
Test Regret
60-30
4 10
80H8 HB
REA
30-3
103
10*
Search Energy Cost (kWh)
Figure 12: NAS results on small tabular benchmark CIFARIO.
23

---

Published as a conference paper at ICLR 2023
1.0
cumulative prob.
0.8
cumulative
0.6
config mmnn/menn|max
0.6
config/ninnman
Ir=0, (01|5.6626.5/00 6/26.5)81.0
0.4
0.00044 5/71
1r=3..15..3....366.7
Ir=0.01 74.664444400
0.2
1r=00.75.3]144.8.6677
0.2
Ir-0.1/42 3/65 9/98.0
/m=100443337.55889
|==10037..1/788 8/999
0.0
70 error
error
(a) CIFAR-10 10 Split=Leaming Rate
(b) TinyImageNet Split=Learning Rate
1.0
1.0
curuulative prob
0.8
cumulative
0.6
0.6
0.4
configlneinnmnenn(rta
0.4
corfigi minjmeannmax
uptim-sgdj44./2555588..
optim=ssdj37..1rr..oo
aptimmadarm/5 2/21.7/69.4
0.2
(01imm=addm/403/77777799 7199.3
0.0
0.0
20
40
60
80
60
100
error
error
(c) CIFAR- OplitzOt
(d) TinylmageNet Split=Optimizer
1.0
1.0
cumulative prob.
cumulative prob.
0.6
config min mean|max
0.4
contgjminjmman/mm
epochs=050066.om 2/..6////2/9..... 9/89.4
epochs=050jt0.811 8/700.0.88//.33// 99
cpochs-100j4.11 2/0.0000472222.0 2
1pochs=100/]8.22677.
epochs=200/4..20
epochs=200037.1165.6 99.5
0.0
0.0
60
70
80
100
error
error
(c) CIFAR-10 10 Split=Max Epoch
(f) TinylmageNet Split=Max Epoch
1.0
cumulative prob.
curnulative
0.6
0.4
config mijjmmanjjmax
confi//minjmneenmmax c_pollcccoss37.226..22
polic=cos|4.2 (2999/////9/8.//
Ir. p_.plic=exp/41.221..222
policelinj4 3/19. 7189.4
0.0
0.0
20
40
80
80
40
50
70
80
90
100
error
error
(g) CIFAR-10 Split=Decay Policy
(h) TinyImageNet Split=Decay Policy
cumulative prob.
cumulative prob.
0.8
0.4
0.4
confeniminjmeannnaa
corfig mminmmean max
oug=none|4.7 9/994
0.2
aug-none|37.1166.
Buggcuuouut422/22.60
aug=:ubbut133.068..719
20
60
60
80
70
80
90
100
error
ii CIFAR- Split=Data Augmentation
TinylmageNet Split=Data Augmentation
Figure 13: The empirical cumulative distribution (ECDF) of all real measured configurations on CIFAR- 10 and TinylmageNet for 5 different splits.
24


Reviews:
[{"review_id": "h_99DInoYin", "reviewer": "Reviewer_ZDyf", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new benchmark dataset for joint neural architecture search (NAS) & hyperparmeter optimization (HPO). It includes information about the energy cost of training a model so that people studying NAS-HPO algorithms can incorporate energy cost. They use a large search space for joint NAS-HPO search and develop a new surrogate model for this search space. In particular, this surrogate model can predict learning curves of arbitrary length. They also demonstrate how to use this dataset.", "summary_of_the_review": "Overall, I like this paper, although the writing & reproducibility could be improved. I'll tentatively mark this paper as \"accept,\" with the assumption that the authors can make at least some of the points more clear. Also, I'm hoping to confirm that the validation curves are provided in the dataset, not just training & test. If validation curves are not provided, I'll lower the score. \n\n**Update post-rebuttal**\n\nThanks for improving the clarity and confirming that the validation curves are included. I'll keep the score at \"8: accept, good paper.\"", "strength_and_weaknesses": "I agree that this is an important problem, and I think that this dataset could be very helpful for saving energy during future research. Also, including HPO makes a lot of sense to me, since a lot of energy could be saved by a model that converges faster. \n\nI have various questions and comments below to improve the writing of the paper. I am also concerned about the reproducibility, as discussed below.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BCDRuy53Dy", "reviewer": "Reviewer_uqiJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes EA-HAS-Bench, which is a surrogate neural architecture search and hyperparameter benchmark that can be used to query the energy cost (in kWh) of training and inference of neural networks in the search space. This can be used as another metric when benchmarking AutoML algorithm to compare them with the energy consumption, which according to the empirical results in the paper, is not well aligned with the training costs. The authors create the benchmark on 2 popular image classification datasets, namely CIFAR-10 and Tiny-ImageNet.", "summary_of_the_review": "In general the paper provides a useful contribution to the NAS community, by proposing a new benchmark that provides the energy consumption utilized to train and evaluate a configuration. Even though suited mainly for black-box algorithms the search space is useful and the surrogate model used to predict the learning curves is novel in the NAS literature. The claims are backed with empirical evidence. I lean towards acceptance for this paper.", "strength_and_weaknesses": "Below I mention some pros and cons of this submission:\n\n(+) Provides novelty in the field of NAS benchmarking by creating a new surrogate benchmark that includes the AutoML search energy consumption.\n\n(+) Well-written and mainly easy to read.\n\n(+) The surrogate model proposed seems to outperform most of previous surrogate models used in the literature.\n\n(+) Claims supported by empirical evidence and analysis.\n\n\n(-) I wasn't able to find the codebase for the benchmark, which is one of the most important features when proposing a new NAS benchmark\n\n(-) No description of the API on how to use the benchmark.\n\n(-) Does not support one-shot NAS algorithms.\n\n(-) Section 5 could benefit of more empirical evaluations, as for instance evaluating multi-objective NAS algorithms on the benchmark.\n\n**Questions and other comments**\n- I guess the models used to encode the architecture and hyperparameters are MLPs? I could not find more details on them. Did the authors provide these in the text? Also, did they try other surrogate models such as GINs for the architecture?\n- It would be interesting to compare the learning curves in Fig 6 to the same counterparts ran on the real benchmark, i.e. without the surrogate predictions, as done in the Surr-NAS-Bench paper.\n\nMinor:\n- Wrong reference to NAS-Bench-301 in Section 3.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "55wHtlCPIB", "reviewer": "Reviewer_QejS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a surrogate model-based benchmark dataset for neural architecture search, which includes search energy consumption and architecture/hyperparameter joint search space. The proposed dataset, called EA-HAS-Bench, enables us to compare the NAS method in energy-aware settings. A Bezier curve-based predictive model is used to provide the surrogate learning curves. The authors exhibit the benchmarking results of existing HPO/NAS algorithms on the energy-aware setting by using the proposed EA-HAS-Bench.", "summary_of_the_review": "This paper provides a novel HPO/NAS benchmark dataset that is valuable for the community. However, several unclear points should be addressed.", "strength_and_weaknesses": "[Strength]\n* A novel benchmark dataset for AutoML/NAS is provided, which includes search energy consumption and architecture/hyperparameter joint search space. As the common benchmark datasets are useful to develop and compare the HPO/NAS algorithms and EA-HAS-Bench considers a novel aspect of benchmarking, it will be valuable in the community.\n* The demonstration and benchmarking of existing algorithms using EA-HAS-Bench are sound.\n\n[Weaknesses]\n* The surrogate model-based benchmark datasets might have a gap with a really collected dataset. It might be better to provide not only the large-scale surrogate dataset but also the small \"real\" dataset.\n* There exist several unclear points. Please see the following comments.\n\n[Comments for Authors]\n* The reviewer is not sure of the definition of energy consumption in this paper. How did the authors measure the search energy consumption? The detail of the data collection method, including machine spec, should be reported.\n* The reviewer could not find the number of actually evaluated samples to build the surrogate model. Such information should be reported to ensure the reliability of datasets.\n* The authors use the Bezier curve and train the model that predicts control points for learning curve prediction. The reviewer thinks that the Bezier curves of order $n$ are equivalent to polynomial functions of order $n$. Therefore, we can use the model that predicts the coefficients of polynomial functions of order $n$ instead of predicting the control points of Bezier curves. The advantage of predicting the control points of Bezier curves should be clarified.\n* It would be better to report the total energy consumption to build EA-HAS-Bench.\n* What does \"GT\" mean in Table 3?\n* What does \"LC\" mean in Figure 6?\n* Why is the result of BOHB for CIFAR-10 in Figure 6 omitted?\n* It would be nice if the authors mentioned the following recent paper. The following paper seems to relate to this work, although it might be published after this paper's submission.\n\nArchit Bansal, Danny Stoll, Maciej Janowski, Arber Zela, Frank Hutter, \"JAHS-Bench-201: A Foundation For Research On Joint Architecture And Hyperparameter Search,\" NeurIPS 2022 Track Datasets and Benchmarks.\nhttps://openreview.net/forum?id=_HLcjaVlqJ\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "22DRlwEA33D", "reviewer": "Reviewer_bERE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents a surrogate tabular benchmark for neural architecture search (NAS) -- EA-HAS-Bench -- with two primary contributions: 1. Joint space comprising architectures and hyperparameter configurations, and 2. Additional metrics to measure efficiency that include training and inference energy costs. This work then uses the EA-HAS-Bench dataset to demonstrate the use of different NAS strategies when constraining the overall budget to a fixed energy/power consumption criterion instead of max. epochs. The work also demonstrates the use of energy consumption per model cost as a factor in the model selection criterion to obtain more energy efficient architectures. Comprehensive experiments are carried out on a large joint space of architectures showing the usefulness of the proposed tabular benchmark dataset.\n", "summary_of_the_review": "A new surrogate tabular benchmark which also includes training and inference energy costs, along with other standard performance measures. The dataset itself is large, comprehensive and well designed. The experiments pertaining to surrogate model predictions are reasonably done. The main contribution -- which is the use of energy consumption to design energy efficient architectures is not evaluated rigorously. For instance, the joint objective proposed to integrate energy costs with accuracy is a heuristic with fixed parameters presented without any justification; the impact of using such an objective on the class of obtained architectures is also not clearly investigated. For a tabular benchmark aiming to demonstrate energy efficiency the experiments about this aspect are not strong enough.\n", "strength_and_weaknesses": "**Strengths:**\n* The main contribution of this work is the EA-HAS-Bench tabular benchmarking dataset which not only has conventional performance measures, but in addition tabulates training- and inference energy costs. This can become an important contribution as most existing tabular benchmarks mainly report number of parameters or training time as the proxies for efficiency.\n\n* The scale of the new surrogate dataset presented in this work is massive. $6\\times 10^7$ unique CNN architectures and 540 hyperparameter configurations are explored yielding about $3\\times 10^{10}$ datapoints.\n\n* A novel surrogate model that approximates the learning characteristics using Bezier Curve-based surrogate (BCS) model is presented and validated. Energy and training times are predicted using Surrogate NAS Benchmark approach in [3].\n\n**Weaknesses:**\n\n* **Energy measurements**: The main contribution in this work is on updating the energy consumption as an additional metric. However, the details on how this energy was measured is missing. Which of the components were monitored and on which hardware? It is mentioned that GPU costs are measured; while this is one of the large contributors the CPU, DRAM costs are not small [1]. Factors such as Power usage effectiveness further have impact on the energy efficiency of the models.\n\n* **Surrogate Energy measurements**: As with other metrics, I assume the energy consumption is also estimated using some surrogate methods. In Sec. 2.2 the authors describe that energy costs are measured for one epoch and aggregated over total number of epochs. Authors do mention that energy costs were predicted according to [3] but in [3] there was no energy costs reported. Is this a linear scaling of the energy cost of one epoch? Does this extrapolation extend simply linearly?\n\n* **Updated objective**: The objective that integrates energy consumption with performance is highly heuristic, and presented without any motivation or justification. Given that the key contribution in this work is to use energy consumption as an additional constraint when performing NAS, the specific formulation of the composite objective is quite vague. It is mentioned to be $ACC\\times {\\frac{TEC}{T}}^w$ with fixed values for T and w. Are there any other ways of incorporating these measurements that were tried? Would the choice of scaling influence the solutions obtained. Do the gains achieved still hold? Why is this specific formulation the preferred choice?\n\n* **Efficiency in terms of FLOPS**: In Sec.4, the correlation (or the lack thereof) between training energy and training times is clearly captured. This is an important result to show the additional resource costs that are missed when only using training time. However, the authors do not extend this analysis or at least discuss how FLOPs as a measure of efficiency correlates (or not) to measuring energy. For instance, works such as [2] use FLOPs as a measure of efficiency and obtain the now famous efficient net architectures.\n\n* **Missing details**: Although the proposed benchmark is a surrogate one, there are no clear reportings of the total cost of obtaining this dataset. For instance, how many architectures/hyperparameter configurations were sampled in the first place to train the two neural networks. This information was either missing or difficult to obtain from the main paper. In Sec 2.4 there's a mention of using random sampling of the search space -- but how many architectures from the $3\\times 10^{10}$ architectures were sampled to be approximated using the BCS model? Further, what was the total energy cost/ GPU costs for obtaining this new dataset? \n\n[1] Henderson, Peter, et al. \"Towards the systematic reporting of the energy and carbon footprints of machine learning.\" Journal of Machine Learning Research 21.248 (2020): 1-43.\n\n[2] Tan, Mingxing, and Quoc Le. \"Efficientnet: Rethinking model scaling for convolutional neural networks.\" International conference on machine learning. PMLR, 2019.\n\n[3] Zela, Arber, et al. \"Surrogate NAS benchmarks: Going beyond the limited search spaces of tabular NAS benchmarks.\" Tenth International Conference on Learning Representations. OpenReview. net, 2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%