Energy Consumption-Aware Tabular Benchmarks for Neural Architecture Search
Authors: Pedram Bakhtiarifard
Christian Igel
Raghavendra Selvan
Venue: Submitted to ICLR 2023 (2023)
Keywords: NAS
tabular benchmarks
energy consumption
carbon footprint
deep learning
multi-objective optimisation
automl

Abstract: The demand for large-scale computational resources for  Neural Architecture Search (NAS) has been lessened by tabular benchmarks for NAS. Evaluating NAS strategies is now possible on extensive search spaces and at a moderate computational cost. But so far, NAS has mainly focused on maximising performance on some hold-out validation/test set. However, energy consumption is a partially conflicting objective that should not be neglected. We hypothesise that constraining NAS to include the energy consumption of training the models could reveal a sub-space of undiscovered architectures that are more computationally efficient with a smaller carbon footprint. To support the hypothesis, an existing tabular benchmark for NAS is augmented with the energy consumption of each architecture. We then perform multi-objective optimisation that includes energy consumption as an additional objective. We demonstrate the usefulness of multi-objective NAS for uncovering the trade-off between performance and energy consumption as well as for finding more energy-efficient architectures. The updated tabular benchmark is open-sourced to encourage the further exploration of energy consumption-aware NAS.

Under review as a conference paper at ICLR 2023
ENERGY CONSUMPTION-AWARR TABULAR BENCHMARKS FOR NEURAL ARCHITECTURE SEARCH
Anonymous authors
Paper under double-blind review
ABSTRACT
The demand for large-scale computational resources for Neural Architecture Search (NAS) has been lessened by tabular benchmarks for NAS. Evaluating NAS strategies is now possible on extensive search spaces and moderate computa- tional cost. But so far, NAS has mainly focused on maximising performance on some hold-out validation/test set. However, energy consumption is partially con- flicting objective that should not be neglected. We hypothesise that constraining NAS to include the energy consumption of training the models could reveal a sub- space of undiscovered architectures that are more computationally efficient with a smaller carbon footprint. To support the hypothesis, an existing tabular bench- mark for NAS is augmented with the energy consumption of each architecture. We then perform multi-objective optimisation that includes energy consumption as an additional objective. We demonstrate the usefulness of multi-objective NAS for uncovering the trade-off between performance and energy consumption as well as for finding more energy- energy-efficient architectures. The updated tabular benchmark, EC-NAS Bench, is open-sourced to encourage the further exploration of energy consumption-awws NAS.
INTRODUCTION
The design of neural architectures is complex task. While general guidelines for producing suit- able neural architectures have been proposed, neural architecture design still requires expert domain knowledge, experience, and not least substantial effort (Philipp 2021 Zoph & Le. 2016 Ren et al.) 2020). This led to an upsurge research on automated exploration and design of neural architec- tures cast optimisation problem neural architecture search (NAS) (Baker al. 2016 Zoph & Lc 2016 Real et al. 2017).
NAS strategies explore neural architectures in predefined search space relying model training and evaluation to determine the model's fitness (i.e., validation/test set score) to adjust the search strategy and extract the best performing architecture (Ren et al.) (2000). NAS strategies have shown great promise in discovering novel architecture designs yielding state-of-the-art model performance (Liu al.) 2017 2018 Lin al., 2021 Baker al. 2017). However, it can be prohibitively expensive to perform NAS (Tan & Lc 2019b) due to the demand for large-scale computational resources and the associated carbon footprint of NAS (Schwartz et al. 2019 Anthony et 2020).
The introduction of tabular benchmarks for NAS significantly lessened the computational challenges mentioned above by facilitating the evaluation of NAS strategies on a limited search space of archi- tectures (Klein & Hutter 2019; Dong & Yang 2020). Predictive models and zero- and one-shot models (Wen et al.) 2019 Lin al. 2021 Zela et al. 2020) have reduced time-consuming model training and thereby increased the efficiency of NAS strategies. Most recently, surrogate NAS bench- marks (Zela et al. 2022) have been proposed for arbitrary expansion of architecture search spaces for NAS.
Notwithstanding the aforementioned major contributions to the advancement NAS research, the prime objective of NAS has been maximising performance objective some hold-out test/valida- tion test. NAS strategies can be evaluated effectively, yet the search strategies do not intentionally aim to find computationally efficient architectures. That is, the NAS may efficiently determine model performance at a moderate computational cost, but energy efficiency is generally not an objective of NAS.

---

Under review as a conference paper at ICLR 2023
We hypothesise that adding the energy consumption of training models NAS objective could reveal a sub-space of computationally efficient models that also have a smaller carbon footprint. In order find efficient architectures without sacrificing cardinal performance requirements, propose the use of NAS strategies that will optimise for multiple objectives.
Our main contributions.
1. We provide energy consumption-awain tabular benchmark for NAS based on NAS- Bench-101 (Ying et al. 2019). For each architecture, we added its training energy con- sumption, power consumption and carbon footprint. We hope that the new data will foster the development of environmentally friendly deep learning systems.
2. We also introduce surrogate energy model to predict the training energy cost for given architecture in a large search space (about 423k architectures)
3. To exemplify the use of the new benchmark, devise simple multi-objective optimisa- tion algorithm for NAS and apply it to optimise generalisation accuracy as well as energy consumption.
4. We demonstrate the usefulness of multi-objectioo architecture exploration for revealing the trade-off between performance and energy efficiency and for finding efficient architectures obeying accuracy constraints. This is also demonstrated with other baseline nulti-objective methods.
2 ENERGY CONSUMPTION-WWAR BENCHMARKS EC--AS-Benchh
Our energy consumption-awy tabular benchmark BC-NAS-Bench is based Nas-Bench-100 101 (Ying et al. 2019). We closely follow their specification of architectures; however, the search space of architectures that are considered, the evaluation approach and the metrics provided for each chitecture different. This section will briefly present OC-NAS-Bence and its differences to NAS 10--Bench-101.
2.1 ARCHITECTURE DESIGN
Network Topology. All architectures considered are convo- lutional neural networks (CNNs) designed for the task of im- age classification on CIFAR- 10 (Krizhevsky) 2009). Each neu- ral network comprises convolutional stem layer followed by three repeats of three stacked cells and a downsampling layer. Finally, a global pooling layer and dense softmax layer are used. The space of architectures, X, is limited to the topolog- ical space of cells, where each cell is configurable feedfor- ward network.
12 Pppee
20 sispcc
spech
Figure 1: Visualisation of the 5V space of architectures with the val- idation performance P, and the corresponding training energy cost (E) for four training epoch budgets 14,12,36,108).
Cell Encoding. The individual cells are represented as di- rected acyelic graphs (DAGs). Each DAG, G(V, M), has N [V| vertices (or nodes) and edges described in the bi- nary adjacency matrix M E 10 X The set of op- erations (labels) that each node realise is given by L' (3x3conv, 1xiconv, 3x3maxpool]. Two the N nodes are always fixed as input and output the network. The remaining N nodes can take up one of the labels in L. The con- nections between nodes of the DAG are encoded in the upper-triangular adjacency matrix with no self-connections (zero main diagonal entries). For a given architecture, A, every entry as E MA denotes an edge, from node i to node with operations i.j E L and its labelled adjacency matrix, LA E M. x L'
(input, output) L. where
Search space. The number DAGS grows exponentially with N and L YYing et al. 2019). We restrict the search space in EC-NAS-Bench by imposing [V| 5 and |A * 9, referred to as the 5V space. The search space with [V| & called 4V space is also considered. In contrast, NAS-Bench- 101 considers the search space for |V] < With these imposed restrictions the
2

---

Under review as a conference paper at ICLR 2023
search space of EC-NAS- 91, 2532 and 423k unique architectures are identified from the 4V, 5V and 7v spaces, respectively.
2.2 ENERGY CONSUMPTION-AWARENN
Resource-constrained NAS for obtaining efficient architectures has been explored mainly by opti- mising the total number of floating point operations (FPOs) (Tan & Le, 2019a). Optimising for FPOs, however, might not be entirely indicative of the efficiency of models (Henderson al. a (020)_ It has been reported that models with fewer FPOs have bottleneck operations that can consume the bulk of the training time (Howard et al. 2017), and some models with high FPOs have lower infer- ence time (Jeon & Kim 2018). Energy consumption optimised hyperparameter selection outside of NAS settings for large language models has been recently investigated in Puvis de Chavannes et al. 202T.
The energy consumption during the training of model encapsulates facets of architecture efficiency that are not entirely taken into consideration when using standard resource constraints such as FPOs, computational time and the number of parameters. Energy consumption accounts for both hardware and software variations in the experimental set-ups. To foster a new direction for NAS to find more efficient architectures, we use energy consumption as the additional objective along with standard performance measures.
2.3 QUANTIFYING ENERGY CONSUMPTION
About 75% of the total energy costs during training a neural network are incurred by hardware ac- celerators such graphics processing units (GPUs) or tensor processing units (TPUs) (Dodge al, 2022). The remaining energy consumption is mainly due to the central processing units (CPUs) and dynamic random access memory (DRAM). Additional energy consumed by the supporting in- frastructure, such as cooling- and power systems and dissipation, is usually accounted for by the power usage effectiveness (PUE), which is overhead factor. Several open-source tools have been published in the past couple of years, such as pnperimenttimpact-rruuee Carbontracker (Anthony et al.) 2020) and CodeCarbon (Schmidt et al. 2021) provide convenient ways to track and log the energy consumption of neural networks by taking these factors into con- sideration.
(Henderson et al.) (020).
In C--NNNSSBBenis the energy consumption of training and evaluating the neural architectures is estimated by modifying the tool Carbontracker (Anthony et al.) (020)). Our version of the tool mon- itors the GPUs, CPUs and DRAM and estimates the total energy costs, E (kWh), aggregate carbon footprint (kgCO,eq) based on the instantaneous carbon intensity of the regions and the total compu- tation time, T(s). The complete set of metrics that measured and reported in EE-NAS-Bencee are listed in Table [
2.4 ARCHITECTURE PERFORMANCE, AND RFFICIENCY
Training Pipeline. Architectures from the 4V and 5V space are trained on CIFAR-10 (Krizhevsky) 2009) using 40k samples and evaluated on 10k validation and test samples (60k total). Each model is trained in-house Slurm cluster on a single NVIDIA Quadro RTX 6000 GPU with 24 GB memory and two Intel CPUs. The training strategy, or hyper-parameter setting, is similar that of NAS-Bench-101 (Klein & Hutter 2019). Pre- dicting the energy consumption of longer model runs from a few training epochs has been shown to be robust when performed on the same hardware (Anthony et al. 2020). To refrain from training and re-evaluating all the models in S-S--Bench--0- we train each model for only 4 epochs and then obtain surrogate time and energy measurements by linear scaling. We then tabulate these measurements along with the corresponding mean performance metrics for each model from NAS- Bench- 101 and obtain metrics for training and evaluating each model for 12, 36 and 108 epochs.
Metries
Unit of measurement
Notation
Model parameters
Million (M)
[0]
Test/TainnEvaal time
Seconds (s)
T(s)
Test/Train/all Acc.
R [0: 1]
P.
Energy consumption
Kilowatt-hour (eWh)
B@Wh))
Power consumptin
Joule (J). Wall (W)
BUJ, E(W)
Carhon footpeint
kgCO,ey g/kWh
Carbon intensity
Table 1:
Metrics reported in EC-NAS Bensc.
3

---

Under review as a conference paper at ICLR 2023
Nare Abveate Eree
Produced
50
500
3000
1000
2000
2500
3900
Actast Eareeg Caspaempiiaa (W71)
a)
b)
Figure 2: a) Scatter plot showing the correlation between the predicted and actual energy consump- tion for subset of the architectures from the 7V-space. b) Influence of the number of training data points on the test set performance in the Surrogate 7V-space model.
Metrics. We report the operations, no. parameters, and performance metrics in EC-NAS-Benchl in NAS-Bench- 101, and additionally, include efficiency measures in terms of energy sumption and the carbon footprint for training each model. The primary focus for efficiency metrics to quantify the resource costs specific to model training; however, we also report the total source costs, which include computational overhead, c.g., data movements. For completeness, we also provide carbon intensity measures at training time, timestamp, and average energy consumption of computing resources. We have made the metrics of cach architecture readily accessible to encour- age the development of NAS strategies for exploring efficient architectures. The metrics reported relevant to this work can be seen in [
2.5 SURROGATE DATASET FOR 7V-SPACE
The 4V and 5V search spaces are the primary spaces used in this work to reduce the overall resource consumption populate the energy measurements in the tabular benchmark datasets. However, even the 5V space has only a fraction of possible architectures compared to the 7V space published in Ying al. (2019), which has about 423k architectures. Computing the energy consumption as done for 4V and 5V datasets on the 7V space is prohibitively expensive
We instead sample subset of architectures from the 7V space and obtain the actual energy costs for 4300 architectures. Using these measurements we train multi-layered perceptron (MLP) based surrogate energy prediction model. The MLP takes the graph-encoded architecture and the number of parameters as input and predicts the energy consumption for given number of epochs. This surrogate model is similar to recent surrogate NAS methods that have shown to be more efficient/Zela al. (2022). Details of the surrogate model used to predict the energy measurements for the 7V space are provided Appendix D
The resulting surrogate 7V dataset with the energy measurements yields a close approximation of the actual training energy costs as shown in Figure 2 a). The Pearson correlation between the actual and predicted energy measurements is 0.9977. In Figure we also show that the mean absolute error of the predicted- and actual energy measurements plateau with about 3000 architectures, justifying its to predict the remaining 7V space. The standard deviation is estimated over 10 random initialisations of the surrogate model per training dataset size.
2.6 INFLUENCE OF HARDWARE ON CC-NAS- Mas--Bench
The energy consumption of the architectures in the 4V and 5V spaces were obtained on single RTX Quadro 6000 GPU. While the energy measurements tabulated in EC-NAS-Bench are specific to these hardware settings, we argue that the trends across the architectures hold independent of actual hardware used. To demonstrate this, we trained the architectures in the 4V space on four different (Nvidia) GPUs spanning multiple generations: Titan XP, RTX 3060, RTX 3090 and RTX Quadro 6000.
While the energy consumed by each model on specific hardware is different, the trends compared to other models are maintained across different GPUs. This is captured in Figure where the energy
Our estimates showed that it would require 770 GPU days of compute.
4

---

Under review as a conference paper at ICLR 2023
RTX 3260
lltan
40
Quadro RTX 2003
Energy (Wh)
Overall
gndd aagaag
20
tha thy pengungaatttee Blaggee Raad
Model number
Figure 3:
Energy consumption of all 4V models on four different GPUs. The models are sorted based on their energy consumption for improved visualisation.
consumption for each architecture in the 4V space on all four GPUs is reported. This trend confirms the fact that when NAS is constrained on energy consumption and performance the resulting models would remain the same irrespective of the specific hardware used.
NAS STRATEGIES WITH EC-NAS
Given a tabular benchmark which can be used to query for model training energy consumption in addition to other standard metrics such in EC-NAS-Benchh NAS strategies be used to search for energy-efficient architectures. We next present multi-ohjective optimisation as a suitable strategy to uncover the trade-off between performance and efficiency, which supports energy- aware architecture choice.
3.1 MULTI-OBIECTIVE OPTIMISATION
Multi-objective optimisation (MOO) simultaneously optimises several, potentially conflicting ob- jectives. The goal of MOO find or to approximate the set of Pareto-optimal solutions, where solution is Pareto-optimal if it cannot be improved in one objective without getting worse in another.
In this work, we introduce a simple evolutionary MOO algorithm (SEMOA) based on Krause et al. 2016). The algorithm is simple, but derived from canonical principles of derivative- multi- criteria optimisation, such as hypervolume maximisation. Details SEMOA are presented in Ap- pendix A.2 We also several existing MOO algorithms: random search, Speeding up Evolu- tionary Multi-Objective Algorithms (SHEMOA) and Mixed Surrogate Expected Hypervolume Im- provement (MSEHVI), implemented in Izquierdo al. (202T) to demonstrate the usefulness of EC-NAS-BEI -aas-BB
3.2 EVALUATION OF NAS STRATEGIES
Experimental Setup. We conduct experiments on EC-NAS-Bench by adapting the presented MOO-algorithm to perform both single-objective optimisation (S0O) and MOO. In the former, we will naturally find only one solution when optimising single objective. In contrast, when optimising multiple, diverse objectives, we will find the empirical Pareto-front in the latter. We run the algorithm in the 4V and 5V space of models trained for 108 epochs. The optimisation is performed over 100 evolutions with a population size of 20. All the experiments are conducted desktop workstation with single NVIDIA RTX 3090 GPU with 24GB memory and Intel(R) Core(TM) i7-10700F CPU 2.90GHz.
Performance Criteria. For the multi-objective optimisation, and the training energy cost, Wh),h), as the two objectives to be jointly optimised using the MOO algorithm. For the single-objective optimisation, we only P, the performance objective. We
the validation accuracy (P.)

---

Under review as a conference paper at ICLR 2023
65
1000
Tu F(Wh))
T..B(MM)
200
300
-00)
0.4
0.1
0.2
0.3
0.
Yo.1 11-PP
b)
C)
Figure 4: Multi-objective exploration of 5V space. a) The attainment curve showing the median solutions and the inter-quartile variations for 100 random initialisations of the four MOO algorithms on the CC--NA--Bench dataset. The two objectives being optimised (1 P... F)) are shown on the two axes. b) Pareto front obtained for one of the MOO runs shows the family of solutions as discrete points c) Radar plot showing three solutions from a Pareto front that close to the median The three architectures
in subfigure
chosen to correspond to the two extrema (Ar, in blue, Ar, in green) and the knee point (Ar, in yellow). The plot has four axes capturing the performance P, and resource consumption measured in E, T, 10].
use energy cost rather than, e.g., training time, considering that E is agnostic to parallel computing. We note that it is possible to use any of the provided metrics in Table for the purpose of single- and multi-objective optimisation. As the MOO algorithm minimises the objectives, we simply use the negative of the objectives in cases where the quantities to be maximised; for instance, optimise - P. as accuracy is a maximisation objective.
Training costs In aggregate, EC-NAS-Bende had total estimated training cost of 124.214 GPU days, 2021.02 kWh and 259.047 kgCO,eq for the 5V space. The 4V space had total estimated training cost of 3.854 GPU days, 63.792 kWh and 5.981 kgCOzeq. The actual training costs for the 5V space were only 3. 3.105 GPU days. 50.525 kWh and 6.476 kgCOzeq. Actual training costs of the 4V space were 0.096 GPU days, 1.594 kWh and 0.149 kgCO,eq.
In total, we saved an estimated compute cost of 121.109 GPU days. 1970.495 kWh and 252.571 kgCO,eq for the 5V space, and 3.758 GPU days, 48.931 kWh and 6.327 kgCO,eq for the 4V space. We obtain = 97% reduction in computing resources and energy consumption in all efficiency mea- sures.
4 RESULTS
Multi-objective exploration of 5V space. The key results from the experiments EC-NAS N--NA--Bench using the multi-objective optimisation of E and -P,, are shown in Fa)
and c). Pareto fronts over multiple random initialisations of the four MOO algorithms: SEMOA
(ours), Random Search, SHEMOA, MSEHVI, are visualised as attainment curves in Figure 4 which summarises the median solutions attained over the multiple runs(Fonseca et al. 2001). All
the MOO algorithms are able to explore the search space reasonably well, yielding attainment curves that largely look similar.
The Pareto front obtained from the our MOO algorithm, SEMOA, for one run is shown in Figure b). It also shows the extrema (No. F1) on both ends of the front preferring one of the objectives, whereas the knee point (rx) offers the best trade-off between the two objectives. These three points are shown in different colours and markers, where the two extrema (Ar,/Blue, Ar, /Green) and the knee point (Ap, /yellow). We compute the bend-angles to find the knee point as suggested by Deb & Gupta (2011).
The architectures corresponding to the two extrema (Ar) and the corresponding knee point (Ar.) for a single MOO run are visualised in the radar plot in Figure 4b. The exact performance
metrics for the three models in Figure b) are also reported in Table 2 The solution covering the
largest area is one of the extremal points (Arr, blue) with high accuracy (0.944) but also a larger
footprint in the energy consumption (1.032kWh), computation time (5482.78s) 78s) and the number of

---

Under review as a conference paper at ICLR 2023
Model
Strategy
(M)()))
T(e) 4
Po 1
EORWEy
Arg (B)
SEMOA
12.08
3878.66
0.942
0.709
Ar
SEMOA
0.137
Arg (Y)
SEMOA
5.09
1916.65
0.932
0.324
A,- (R)
S00
21.22
5482.78
0.944
1.032
Table 2:
Metrics for 5V space models single- and multi-objective setting seen in Figure 4 For SOO the optimal solution (Ar-/Red) reported. For MOO the two extrema (Arg/Blue, Ar, /Green) and the knee point (Ar, /yellow are reported.
250
600
Tt. (..((W))
100
1700
50
0.3 n.. P.
Tt.1 P.
a)
b)
Figure 5: Multi-objective exploration of 7V space. The attainment curve showing the median solutions and the inter-quartile variations for 100 random initialisations of the four MOO algorithms on the EC-NAS-Bench dataset. The two objectives being optimised (1 P., E) shown on the two axes. b) Pareto front obtained for one of the MOO runs shows the family of solutions as discrete points
parameters (21.22M) compared to the other extremum (Ar,, green) or the knee point (Arr. yel- low). The model corresponding to the knee point (Ar.) provides large reduction in the energy consumption (0.324kWh) at the expense of small reduction in performance (0.932).
Single-objective exploration. We optimise only the validation accuracy, Pos simulate standard NAS practices. The resulting solution is shown the last row of Table 2 This SOO model achieves the highest validation accuracy (0.944). However, the footprint of the solution along the energy consumption, computation time and the number of parameter axes are larger than those from the MOO algorithm.
Multi-objective exploration 7V space. The MOO results for the surrogate 7V space resemble the trends observed the 5V space, as shown Figure As with the 5V space all the attainment curves of all the four MOO algorithms look similar. Visibly, the MSEHVI method seems underperform compared to other models due to the protrusion around the knce-point compared to the other models, which are largely overlapping. A single Pareto front of SEMOA also showed in Figure b), with trends comparable with those in the 5V space results in Figure
5 DISCUSSIONS
Single versus multi-objective optimisation. The performance trends of the SOO and MOO solu- tions clearly captured in Table The knee point solution, Ar,, from MOO, yields an architecture that consumes about 70% less energy and has only about 1% degradation in performance. Depend- ing on the downstream tasks, this could be reasonable trade-off. the degradation in performance cannot be tolerated, the Pareto front offers other candidate solutions for the practitioners to choose from. For instance, the extremum solution (Arr) offers basically the same performance as the SOO solution by consuming about 32% less energy.
Training time is not an alternative to energy consumption. The original NAS-Bench- 101 al- ready reports the training time (Ying et al. 2019). In single hardware regimes, this could serve as measure of the energy consumption, as training time mostly correlates with the energy consumption. However, most neural architecture training is performed lelism, training time alone cannot capture the efficiency of models. Aggregate energy consumption can take parallel hardware and the associated overheads into consideration. Even in single GPU
multiple GPUs with large-scale paral-
7

---

Under review as a conference paper at ICLR 2023
training settings, energy consumption could optimise for energy- models. For instance, small architecture trained on a large GPU still has larger energy consumption due to the under- utilisation of the hardware resources. In such instances, larger model could (to a certain extent) yield more performance improvements for the total energy consumed (P./ FE).
Energy efficient tabular NAS benchmark for obtaining efficient architectures. Tabular bench- marks such as NAS-Bench-101 (Ying et al. 2019) were introduced to reduce the resources required to perform NAS. However, even the one-time cost of generating tabular benchmark dataset is massive. Surrogate NAS benchmarks are being studied to alleviate these costs, where the models are not exhaustively trained and evaluated. Instead, the performance metries of architectures are estimated based on smaller training costs. For instance, this is achieved using predictive modelling based on learning curves (Yan et al. 2021), gradient approximations (Xu et al., 202T), or by fitting surrogate models to subset of architectures (Zela et al., (022). Similar to these attempts, the pro- posed EC-NAS- BBenhhenhhhhhc dataset does not train all the models but bases its predictions on training the models only for 4 epochs, described in Section 3.2 This results in about 97% reduction if the dataset were to be created from scratch, as shown in Table 3 Thus, EC-NAS CCC-NAS-Benci is an energy- efficient tabular benchmark that can be used to obtain energy -efegy-efficient architectures as demonstrated in Section
Corbon-footprint aware NAS. The RC-NAS-Bence dataset reports several metries per architecture, as shown in Table [T Combinations of these metrics and the use of MOO could allow for the exploration of architecture spaces that have interesting prop- erties. For instance, NAS can be performed to directly optimise the carbon footprint of deep learning models. Although instantaneous energy consumption and carbon footprint linearly correlated, when measured over a longer duration (>5m) these quantities differ due to the fluctuations of the instantaneous carbon intensity (Anthony et al. 2020). These carbon intensity fluctuations are caused by the variations of the power sources to the grid (Henderson et al. 2020). This can have implications when training models for longer duration on cloud instances that can be distributed over data centres in different countries (Dodge et al. (022). By reporting instantaneous and aggregate carbon footprint of model training the possibility of carbon footprint aware NAS (Selvan et al.) (2022). In this work, we focused only on energy consumption awareness work around the temporal- and spatial variations of the carbon intensity.
Space
Red. GPU days
Red. kWh
Red. kgCO.sq
3.758
48.931
6.327
SV
121.109
1970.495
2.22.571
Table Estimated reduction in total training costs w.r.t GPU days, kWh and kgCO,eq of the 4V and 5V space.
CC-NAS- Bench we facilitate
Energy Consumption aware Few-shot NAS. Tabular benchmarks such as et al. (219)) provide an efficient way to explore different NAS strategies where the model training cost is only incurred once. One restriction with such tabular benchmarks is that they are specific to set of architectures (fx: feedforward convolutional neural networks) and datasets (fx: CIFAR- 10). Developing tabular benchmarks for all possible network architectures and datasets is alleviated using one- few- shot learning methods (Zhao et al., 2021: Zela et al. 2020). Integrating surro- gate models for predicting learning dynamics (Zela et 2022) and energy measurements using the surrogate model in Section 2.5 could bridge the divide between few-shot and surrogate tabular benchmark datasets that are also energy consumption-awwe
101 (Ying
We have demonstrated the integra- tion of surrogate energy models with existing tabular benchmark datasets, and extending these to surrogate benchmark datasets is straighttorward.
Limitations. Constraining the number of vertices in the DAGS results in sparser search spaces for the optimisation strategy. The optimisation strategy will therefore be more sensitive to initialisation and choice of random seeds, and the empirical Pareto front will appear to be more rigid. as seen in the attainment plot in Figure f even when multiple initialisations and trials are carried out. We also only demonstrated experiments on the 4V and 5V spaces.
To reduce the computation cost, in EC-NAS-Bench we used the surrogate time and energy mea- surements that do not model training time variability. We also query the performance metrics from the three repeats of NAS-Bench- 101 and update EC-NAS- CC-NAS-Ben.h with their mean performance metries.
8

---

Under review as a conference paper at ICLR 2023
All these limitations primarily driven by the need minimise the energy consumption of these experiments. While these are at the expense of variability, we argu that the resulting reduction the energy consumption justifies these choices. Further, the results from these small-scale experiments have been shown to extend to larger space of architectures (Ying et al. 2019).
6 CONCLUSIONS AND FUTURE WORK
In this work, we presented an updated tabular benchmark dataset, NCC-NNS- -ench,, which tabulates the energy consumption and carbon footprint training models, in addition to standard performance measures. Using multi-objectivve optimisation strategies, showed that Pareto-optimal solutions offer appealing trade-offs between the performance measures and the energy consumption of model training. We qualitatively showed that large reductions (about 70%) in energy consumption possible with < % reduction in performance.
In addition to providing energy consumption measures, the EC-NAS- benchmark provides metrics such as average carbon footprint and power consumption of CPUs, GPUs and DRAM. We hope this will foster interest in the development of models that are efficient and environmentally friendly by optimising for their energy consumption and carbon footprint.
REFERENCES
Lasse Wolff Anthony, Benjamin Kanding, and Raghavendra Selvan. Carbontracker: Tracking and predicting the carbon footprint of training deep learning models. ICML Workshop on Challenges in Deploying and monitoring Machine Learning Systems, July 2020. arXiv:2007.03051.
Rhonda Ascierto and Andy Lawrence. Uptime Institute global data center survey 2020. Technical report, Uptime Institute, 07 2020.
Johannes Bader and Eckart Zitzler. Hype: An algorithm for fast hypervolume-based many-objective optimization. Evolutionary computation, 19(1):45-76, 2011.
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architec- tures using reinforcement learning, 2016. URL http::::////aii.ooor//////////// 02167
Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural architec- ture search using performance prediction, 2017. URL httpp::/aarxiv.orrr//bb////
10:23
James Edward Baker. Adaptive selection methods for genetic algorithms. In Proceedings of an International Conference Genetic Algorithms and their Applications, volume 1, 1985.
Nicola Beume, Boris Naujoks, and Michael Emmerich. Sms-emoa: Multiobjective selection based on dominated hypervolume. European Journal of Operational Research, 18533)1663--169, 2007.
Karl Bringmann and Tobias Friedrich. Approximation quality of the hypervolume indicator. Artifi- cial Intelligence, 195:265-291 290, 2013.
Karl Bringmann, Tobias Friedrich, Christian Igel, and Thomas VoB. Speeding up many- optimization by Monte Carlo approximations. Artificial Intelligence, 204:22-221 29, 2013.
Kalyanmoy Deb and Shivam Gupta. Understanding knee points bicriteria problems and their im- plications as preferred solution principles. Engineering Optimization, 1173111111111 Novem- ber 2011. doi: 10. 1000000005215x.20055555555
URL lattps://dei.org///
03052 5x 2010 548863
Jesse Dodge, Taylor Prewitt, Remi Tachet des Combes, Erika Odmark, Roy Schwartz, Emma Strubell, Alexandra Sasha Luccioni, Noah A Smith, Nicole DeCario, and Will Buchanan. Mea- suring the carbon intensity of ai in cloud instances. In 2022 ACM Conference on Fairness, Ac- countability, and Transparency, pp. 1877-1894 2022.

---

Under review as a conference paper at ICLR 2023
Xuanyi Dong and Yi Yang. Nas-bench-201. Extending the scope of reproducible neural architecture search, 2020.
Viviane Grunert da Fonseca, Carlos M Fonseca, and Andreia O Hall. Inferential performance sessment of stochastic optimisers and the attainment function. In International Conference on Evolutionary CuitiiCrriterion Optimization, pp. 213-225. Springer, 2001.
John J Greffenstette and James Baker. How genetic algorithms work: critical look at implicit parallelism. In Proceedings of the 3rd International Conference Genetic algorithms, pp. 20- 27, 1989.
Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pineau. To- wards the systematic reporting of the energy and carhon footprints of machine learning. Journal of Machine Learning Research, 21(248): 2020.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.0486. 2017.
Christian Igel, Nikolaus Hansen, and Stefan Roth. Covariance matrix adaptation for multi-objective optimization. Evolutionary Computation, 15(1):1-28, 2007.
Sergio Izquierdo, Julia Guerrero-Vin Sven Hauns, Guilherme Miotto, Simon Schrodi, André Biedenkapp. Thomas Elsken, Difan Deng, Marius Lindauer, and Frank Hutter. Bag of base- lines for multi-objective joint neural architecture search and hyperparameter optimization. In 8th ICML Workshop on Automated Machine Learning (AutoML), 2021.
Yunho Jeon and Junmo Kim. Constructing fast network through deconstruction of convolution. Advances in Neural Information Processing Systems, 31, 2018.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations, 2015.
Aaron Klein and Frank Hutter. Tabular benchmarks for joint architecture and hyperparameter opti- mization, 2019. URL
org. abs 1905. 04970
Oswin Krause, Tobias Glasmachers, and Christian Igel. Multi-objectivv optimization with un- bounded solution sets. In NeurlPS Workshop on Bayesian Optimization (BayesOpr 2016), 2016.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Ming Lin, Pichao Wang, Zhenhong Sun, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, and Rong Jin. Zen-nas: zero-shot nas for perh-performmnce deep image recognition, 2021. URL https: //arxiv.org/aant 21 02.00.22 01063
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search, 2017. URL
htttps://aae http://///xxxi..gg////////////// wwtp///gg//////ooo
00559
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search, 2018. 09055
URL htttps://aaxiv..og//eee.e
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc. E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019.
George Philipp. The nonlinearity coefficient a practical guide to neural architecture design, 2021.
URL https: //a///axii// http:///wwxivvooaaa/////////// 12210
10

---

Under review as a conference paper at ICLR 2023
Lucas Høyberg Puvis de Chavannes, Mads Guldborg Kjeldgaard Kongsbak, Timmie Rantzau, and Leon Derezynski. Hyperparameter power impact in transformer language model training. In Proceedings of the Second Workshop Simple and Efficient Natural Language Processing, pp- 96-118, Virtual, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 2021.sustainlp-1.22 httpttss:: //acc anthology.........oo wtt.yyorrgg22200 www.ssilssaaaannlpp.s..
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoe V. Lc, and Alexey Kurakin. Large- scale evolution of image classifiers. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pp. 2902-2911. PMLR, 06-11 Aug htmll
2017. URL http:::tt ltttps:/pproceedingg ml.
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po- Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. A comprehensive survey of neural architecture search: Challenges and solutions, 2020.
URL htt.hhtss///axxiv.orgg/ httprr/ggab///ee/aeeeee/// 6006. 02903
Victor Schmidt, Kamal Goyal. Aditya Joshi, Boris Feld, Liam Conell, Nikolas Laskaris. Doug Blank, Jonathan Wilson, Sorelle Friedler, and Sasha Luccioni. CodeCarbon: Estimate and Track Carbon Emissions from Machine Learning Computing. 2021. doi: 10.5281/7enodo 46584444.
Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai, 2019. URL https: //arsvv.rr//abb/1000 10597
Raghavendra Selvan, Nikhil Bhagwat, Lasse F Wolff Anthony, Benjamin Kanding, and Erik B Dam. Carbon footprint of selecting and training deep learning models for medical image analysis. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp- 506-516. Springer, 2022.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International conference on machine learning, pp. 6105-6114 14. PMLR, 2019a.
Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. 2019b. doi: 10.48550/ARXXI.19955.
URL httppss://axiii..rr////////
1905. 11946
Wei Wen, Hanxiao Liu, Hai Li, Yiran Chen, Gabriel Bender, and Picter-Jan Kindermans, Neural Neural pre- dictor for neural architecture search, 2019. Uttpstts:://rxx wwtpp/wwwwwwww.p./////oa/// 100p///0111000088888 00848
Jingjing Xu, Liang Zhao, Junyang Lin, Rundong Gao, Xu Sun, and Hongxia Yang. Knas: green neural architecture search. In International Conference on Machine Learning, pp. 11613-11625. PMLR, 2021.
Shen Yan, Colin White, Yash Savani, and Frank Hutter. Nas-bench-x11 and the power of learning curves. Advances in Neural Information Processing Systems, 14:22534-225444 2021.
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. NAS- bench-101: Towards reproducible neural architecture search. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 7105-7114, Long Beach, Cali- ml
fornia, USA, 09-15 Jun 2019. PMLR. http:::tttcceeeeeeeeeeeee ptw.ss............ ying19a html
Arber 7ela, Julien Siems, and Frank Hutter. Nas-bench-Ishotl: Benchmarking and dissecting one- shot neural architecture search, 2020. Utl.htttss///arxii http:///////rii..oo//aaaaa//e http://bs/000110 10422
Arber Zela, Julien Niklas Siems, Lucas Zimmer, Jovita Lukasik, Margret Keuper, and Frank Hut- ter. Surrogate NAS benchmarks: Going beyond the limited search spaces of tabular NAS benchmarks. International Conference on Learning Representations, 2022. URL. https:
7/openrevi ew, bett/foruue d=t.:mmmmmdd0omaa55mm
Yiyang Zhao, Linnan Wang, Yuandong Tian, Rodrigo Fonseca, and Tian Guo. Few-shot neural architecture search. In International Conference on Machine Learning, pp. 12707-12718, PMLR, 2021.
11

---

Under review as a conference paper at ICLR 2023
Eckart Zitzler and Lothar Thiele. Multiobjective evolutionary algorithms: A comparative case study and the strength Pareto approach. IEEE Transactions on Evolutionary Computation, 3(4):257- 271. 1999.
Eckart Zitzler, Lothar Thiele, Marco Laumanns, Carlos M. Fonseca, and Viviane Grunert da Fon- seca. Performance assessment of multiobjective optimizers: An analysis and review. IEEE Trans- actions on Evolutionary Computation, 7(2):117-132, 2003.
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning, 2016. URL https://arxiv wwwpp/pppp/.w..oo///..opp/...eet 01578
12

---

Under review as a conference paper at ICLR 2023
A MULTI-OBJECTIVE OPTIMISATION
Formally, let the MOO problem be described by f: x RT,, f(z) (f( (z)). Here
X denotes the search space of the optimization problem and m refers to the number of objectives.
We assume w.l.o.g. that all objectives are to be minimized. For two points æ, E X say that o dominates x and write x' < x if Ví E (1...., m..: fi(x') x [x) A 3j E (I,..., m): [5x²) < (z). For X', , C X we say that X' dominates X" and write X
X" if Yz"
X"
Er' E X x < X", The subset of non-dominated solutions in a set X' C X is given by
ndom (X') = {o x X' A Aal E X' (a): : x' x x). The Pareto front of set X' c X defined as F(X') (f x E adom(X')) and, thus, the goal of MOO can be formalised as approximating F(X).
In iterative MOO, the strategy is to step-wise improve set of candidate solutions towards a suffi- ciently good approximation of F(X). For the design of MOO algorithm. it important to have a way to rank two sets X' and X" w.r.t. the overall MOO goal even if neither X' < X" nor X" x X'. This ranking can be done by the hypervolume measure. The hypervolume measure or S-metric (see Zitzler & Thiele 1999) of set X' S X is the volume of the union of regions in Rm that are dominated by X" and bounded by some appropriately chosen reference point r E I.:
Sx(X)) = A (u [f(()).]] x... x [Jm (on(()) rm] WE
where A(-) is the Lebesgue measure. The hypervolume is, up to weighting objectives, the only strictly Pareto compliant measure (Zizzlrr et al. 2003) in the sense that given two sets X' and X" have S(X') > S(X") if X' dominates X". As stated by Bringgman et al. (0033), the worst-case approximation factor of a Pareto front F(X') obtained from any hypervolume-
set X' with size |X|| = p is asymptotically equal the best worst-case approximation factor achievable by any set of size μt, namely (11/p) (1) for additive approximation and 1 0 /a) for relative approximation
(Bringmann Friedrich 2013). Now we define the contributing hypervolume of an individual
E Exxa. as
x= =
S,(X fx))
The value Max, X') quantifies how much candidate solution contributed to the total hypervolume of X' and can be regarded as a measure of the relevance of the point. Therefore, the contributing hypervolume is a popular criterion in MOO algorithms (c.g. Beumee al. 2007 Igel et al.) 2007 Bader & Zitzler 2011 Krause et al. 2016). If we iteratively optimize some solution set P. then points x with low A(x, P) are candidates in an already crowded region of the current Pareto front F(P), while points with high Azz, P) mark areas that are promising to explore further.
A.1 SEMOA: SIMPLE EVOLUTIONARY MULTI-OBJECTIVE OPTIMISATION ALGORITHM
In this study, we used a simple MOO algorithm based on hypervolume maximisation outlined in Algorithm by Krause et al. ( The algorithm iteratively updates set P of candidate solutions, starting from a set of random network architectures. Dominated solutions are removed from P. Then À new architectures are generated by first selecting À architectures from P and then modifying these architectures according to the perturbation described in Procedure 2 The À new architectures are added to P and the next iteration starts. In Procedure 2 the probability Pedge for changing (i.e., either adding or removing) an edge is chosen such that in expectation, two edges are changed, and the probability Paode for changing a node is set such that in expectation every second perturbation changes the label of a node.
The selection of the À > m architectures from the current solution set described in Procedure 3 We always select the extreme points in P that minimize single objective (thus, the precise choice of the reference point r is of lesser importance). The other m À points are randomly chosen preferring points with higher contributing hypervolume. The points in P are ranked according to their hypervolume contribution. The probability of being selected depends linearly on the rank. We use linear ranking selection (Baker 1985; Greffenstette & Baker 1989,, where the parameter
controlling the slope is set to n7 =2. 2. Always selecting the extreme points and focusing on points with large contributing hypervolume leads to wide spread of non-dominated solutions.
13

---

Under review as a conference paper at ICLR 2023
Algorithm SEMOA for NAS strategy
Input: objective f = (fi...., fm). maximum number of iterations
Output: set of non-dominated solutions
1: Initialize P C. x (e.g., randomly)
b Initial random architectures
2: P <- ndom(P)
Discard dominated solutions
for <- to
Loop over iterations b Get À points from P Change the architectures Evaluate architectures
4: O < LinearRanSammleePP... A)
5:
O - Pcrturb(O)
6: Compute fz) for all x E O
P 4-ndom(PU) UO)
p Discard dominated points
end
9: return P
Procedure 2 Perturb(O)
Input: set of architectures O variation probabilities for edges and nodes Padge and poode Output: of modified architecture O
1: for all Ma E do
Loop over matrices
2:
repeat
3:
for all on. E M. do With probability Padee flip oijj
Loop over entries
4:
end for
6: for all E LA do
Loop over labels
With probability Puode change the label of 8: end for
7:
until MA has changed
10:
end for
11:
return O"
A.2 MULTI-OBJECTIVE OPTIMISATION BASELINES
Hyperparameters for the MOO baseline methods All baseline methods utilise the tabular bench- Benchfor exploring and optimising architectures. The methods" hyperpa-
marks of rameters are chosen to circumvent unfair advantages gained by increased compute time, e.g., allocate similar resources for the bascline
no- no--itrationss function evaluations. Although methods, it is difficult to reason for the fairness when comparing the baselines, when considering the disparity in the algorithmic approach of the baselines.
The bag-of-baselines baselines implementation discussed in Izquierdo et al.) (2021) are used and modified for for 10 trials using different initial seeds. All developed code will be made public upon the blind-review period ending.
compatibility with tabular benchmarks of EC-NAS- N-NN-AA-BBe CC-NAS-BBnch. Bench. Each experiment
Random Search The baseline methods, except for Random Search, apply evolutionary search heuristics to optimize architectures the search space. The random search implementation sam- ples architectures from the architecture uniformly at random, each time querying an architecture for random epoch budget. Random search is done over 1000 iterations, as the other baseline methods, where applicable, will also run for 1000 iterations.
Speeding up Evolutionary Multi-Objective Algorithm (SH-EMOA) As with all our baselines, we use the implementation in Tzquierdo et al.) (2021). We define a problem for and search space following the bag-of-baseling API allow model evaluation for different epoch budgets simply by querying the tabular benchmarks of N----NA---eeeo
We initialize the algorithm with population size of 250 and restrict the search to 1000 function evaluations for budgets between 4 and 108. However, force the algorithm only to use budgets 4, 12, 36 and 108, which available in search space. The remaining hyperparameters we leave as default, which covers a uniform mutation type for architecture perturbation and tournament style parent selection for off-spring generation.
Mixed Surrogate Expected Hypervolume Improvement (MS-EHVI) This evolutionary algo- rithm, too, is initialized with population size of 250. We choose to generate 50 samples lessen
14

---

Under review as a conference paper at ICLR 2023
Procedure 3 LinearRankSample((.,
Input:
of candidate solutions, numher À of elements to selected; reference point E R parameter controlling the preference for better ranked points 1) [1,2]
Output: O C P. =
1: O ==0
2: for i 1 to m do
3: 0+ Urrgin (x)
> Always add extremes
4: end for
5: Compute (r P) for all P
Compute contributing hypervolume
6: Sort P according A(x,
Define discrete probability distribution ı over where
Pl n1-
1 n1'
2(n
i i- P| -1
is the probability of the element with the ith largest contributing hypervolume
for 4- to
m do
I Randomly select remaining points Select points with larger A, with higher prohahility
Draw J
10:
O ++ xr
11: end for
12: return
computation time, and we merely pass an auxiliary function to discretize parameters to with experimental setup using tabular benchmarks.
Simple Evolutionary Multi-Objective Algorithm (SEMOA) Our MOO algorithm is described in subsection A.2 The key hyperparameters are the initial population size, which we set to 250, similar to the baseline methods, and likewise, we run the algorithm for 1000 iterations.
B MEASUREMENTS FROM CARBONTRACKER
We modify the open-source tool Carbontracker (Anthony al. 2020) to measure the additional met- ries reported in Table I Measurements take into account the energy usage of Graphical Processing Units (GPU), Central Processing Units (CPU), and Dynamic Random Access Memory (DRAM). Note that the energy usage for CPUs will include the power usage of DRAM. Power usage informa- tion monitored, logged every 10 seconds, and reported as the average power usage during model training. Power is measured during model training. The integral power consumed over the time time interval, energy, is then reported in units of kilowatt-hours (kWh) with IkWh = 3.6 3.6-10% Joule (J). Additionally, the emission of greenhouse gasses (GHG) measured by equivalent units measured grams of carbon dioxide (CO,eq). The CO is then estimated by using the carbon intensity COAeq units necessary to produce one unit of electricity a kilowatt per hour (kWh)- to express the carbon footprint of model training. The quantities for carbon intensity fetched from the carbon intensity data provider every 15 minutes during model training.
the average of total units of watt (W) over 10-second intervals
Measurements from the aforementioned components alone do give an accurate depiction of the carbon footprint model training when taking into account the energy consumption of the supporting infrastructure (e.g., data centre) is not considered. Therefore the quality of energy and carbon foot- print estimations is amended by multiplying the power measurements by the PUE of the data centre hosting the compute resources. We use PUE of 1.59, which is the global average for data centres in 2020 (Ascierto & Lawrence 2020).
C ADDITIONAL RESULTS
The results in Figure and Figure S were reported for the 5V- and 7V spaces respectively. The also consists of the complete 4V space. In this section report the MOO solutions based on the 4V search space. The trends observed for the 5V. and 7V spaces also hold for this smaller space as well.
15

---

Under review as a conference paper at ICLR 2023
-251
a)
b)
Figure 6: Results for the 4V space. a) Pareto front obtained for one of the MOO runs shows the family of solutions as discrete points b) Radar plot showing three solutions from a Pareto front that close to the median in subfigure a). The three architectures are chosen to correspond extrema (Ar, in blue, Ar, in green) and the knee point (Ar, in yellow). The plot has four axes capturing the performance P. and resource consumption measured in E, T, [0]. с) The attainment curve showing the median solutions and the inter-quartile variations for 100 random initialisations of the MOO algorithm on the EC-NAS-Bench dataset. The two objectives being optimised (-Po, E)
the two
shown on the two axes.
Model
Strategy
[0
T(s) 4
P. 1
E(KWhy
Aro (I)
MOD
21.22
5461 .666
0.944
1.015
Arg (G)
MOO
0.88
815.93
0.83
0.153
MOO
3.20
1782.80
0.925
0.355
A, (R) S00
21.22
5482.78
0.944
1.032
Table 4: Metrics for models in single- and multi-objective setting seen in Figure 6 For SOO the optimal solution (Ag-/Red) is reported. For MOO the two extrema (Ar,/Blue, Arree)) and the knee point (Ar,/yelloo are reported.
D SURROGATE ENERGY MODEL
The MLP-based surrogate model used to predict the training energy consumption of the 7v space, R
E is given as: fo(-): features obtained from the architecture specifications. Using the cell/graph encoding of architectures introduced in Section 2.1] we populate to consist of the upper triangular entries the adjacency matrix, operations (input, 1x1conv 3x3conv, 3x3maxpool, output} mapped categorical vari- ables 2,3, 4, 5). respectively and the total number parameters. For the space this results in 36
R. where f the trainable parameters and x comprises the
We use a simple four layered MLP with gelu(-) activation functions, except for the final layer, 64 + 32
which transforms the input in this sequence 36 - 128
The surrogate energy model is trained using actual energy measurements from 4300 randomly sampled architectures from the 7v space. The model was implemented in Pytorch (Paszke et al. 2019) and trained on an Nvidia RTX 3060 GPU. Using a training, validation and test split of ratio 0.16, 0.3) we train fa(.) for 200 epochs with initial learning rate of x 10-3 to minimise the the L1-norm loss function between the predicted and actual energy measurements using the Adam optimiser (Kingma & Ba 2015).
16


Reviews:
[{"review_id": "zUS-BLbrVV", "reviewer": "Reviewer_Bb6X", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes an energy consumption-aware tabular benchmark for NAS based on NAS-Bench-101. For each architecture, it adds the training energy consumption, power consumption, and carbon footprint. This work also demonstrates the usefulness of multi-objective architecture exploration for finding energy-efficient architectures without sacrificing much accuracy.\nThe MOO algorithm used is based on existing work from Krause et al. 2016.\n\n", "summary_of_the_review": "The established energy-aware NAS benchmark is a very good contribution. However, the presentation of results could be improved. And benchmarked baselines should be added. ", "strength_and_weaknesses": "**Strengths**: Energy consumption is indeed an important factor to consider in NAS. This benchmark can help facilitate further research and development on energy-efficient NAS. This work presents an interesting showcase of using MOO to find energy-efficient NAS.\n\n**Questions and concerns**: This benchmark changed the search space of NAS-Bench-101. For example, instead of evaluating a 7V space, this work tests 5V and 4V spaces. Can the authors justify why this change is made? Clarity of the results: The results presented in Figure 2 are not very clear to me. Figure 2a shows the Pareto front obtained from one of the MOO runs. But what do the different markers and colors mean? It is my understanding that the blue, green, and yellow points correspond to the three types of architecture, including two extrema and a knee point. Why the red one, i.e., the optimal solution in SOO is, not shown in the figure? It would be great if the authors could establish benchmarked baseline performance on this new benchmark.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qCPCP-WRYfn", "reviewer": "Reviewer_gjRN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a tabular benchmark that includes various energy consumption metrics for a subset of networks in the NAS-Bench-101 search space. The authors provide simple multi-objective and single-objective baselines and run those on the benchmark.", "summary_of_the_review": "The motivation for having a benchmark that provides energy consumption metrics is valid and would be beneficial for the NAS community. However, I think this submission is not ready for acceptance due to the reasons I mention above.", "strength_and_weaknesses": "I find the proposed benchmark and the motivation for using such benchmark beneficial to the community as the awareness for resource efficient NAS increases. The paper is in general easy to follow and well-structured. The authors also release their codebase together with the API. However, I find the paper could improve with the following:\n\n- **A larger and novel search space**. The smaller versions of the NAS-Bench-101 search space do seem way far from being realistic in my opinion. With 91 and 2532 architectures, respectively, they do not provide a realistic testbed and even the simplest search algorithms would lead to optimal solutions pretty quickly. Moreover, these spaces do not provide support for one-shot NAS methods.\n\n- **More multi-objective algorithms to evaluate**. The authors only evaluate a single multi-objective algorithm in Section 4. It would be beneficial if they would add more methods to this section. Check [1] for some simple methods.\n\n- **More on the benchmark than the algorithms used**. The authors spend most of Section 3 by describing the multi-objective optimization algorithm they use to evaluate on their benchmark. While such a detailed description is appreciated, it does not serve the main purpose of the paper and therefore I do not find it necessary to be included in the main paper. Rather, more statistics and analysis of the benchmark would be more useful.\n\n-- References --\n\n[1] Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimization. Guerrero-Viu et al. 2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "GoPtyu7BGrc", "reviewer": "Reviewer_FG4d", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper present a new benchmark that extends NASBench-101 with the training energy footprints. The author also presents a MOO algorithm to perform the NAS.", "summary_of_the_review": "It is a nice paper, but the authors seem to strike the reviews with some novelty. Frankly, it is okay for such tool paper to be not that novel as long as it solves practical problems. So I'd hope authors focus more on analyzing the dataset itself and establish a set of canonical baselines for using the dataset. Thank you.", "strength_and_weaknesses": "Strength:\n1) curate a new dataset and perform NAS on it. \n2) the authors also evaluate their algorithms on NAS benchmark.\n3) the paper is quite easy to follow, it is more like a tool paper that proposes a new benchmark for others to use.\n\nWeakness:\n1) I don't think it is necessary for such paper to introduce a new optimizer then spent a huge amount of content on the optimizer. Instead, I think it is more reasonable for authors to reuse the latest MOO algorithm such as [1] and [2] to perform NAS on their datasets, then release them as a baseline.\n\n[1] Zhao, Yiyang, et al. \"Multi-objective Optimization by Learning Space Partitions.\" arXiv preprint arXiv:2110.03173 (2021).\n\n[2] Daulton, Samuel, Maximilian Balandat, and Eytan Bakshy. \"Parallel bayesian optimization of multiple noisy objectives with expected hypervolume improvement.\" Advances in Neural Information Processing Systems 34 (2021): 2187-2200.\n\n2) The experiments spent too much on the optimizer but not on the dataset, but I'd like to see more analysis on the dataset since I assume the focus of this paper is on introducing a new dataset.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4ed9_9wrrs", "reviewer": "Reviewer_bAGs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed an energy consumption-aware tabular benchmark for NAS based on NAS-101, called EC-NAS-Bench. EC-NAS-Bench contains the training energy consumption, power consumption and carbon footprint of each architecture in the benchmark. The author performs single-objective optimization and multi-object optimization (based on the algorithm proposed in the paper \"Multi-objective optimization with unbounded solution sets\") on EC-NAS-Bench, and noticed that multi-objective optimization is able to figure out architectures with about 70% energy reduction and <1% performance degradation\n", "summary_of_the_review": "I vote for weak rejection because EC-NAS-Bench may soon outdated when new hardwares appear. Thus, this benchmark may not be useful to the research community.\n", "strength_and_weaknesses": "Strength\n\n- Incorporating energy consumption data in NAS is an important problem. The author took the first step in this direction. Experiments show that it is able to find architecture that consumes much less energy but with comparable performance.\n\nWeakness\n\n- Different from the performance of the model, energy consumption of neural networks is a moving target. This is related to factors like global economy and the computational hardware. The author has not discussed how they will incorporate potential changes in the energy/power consumption and carbon footprint of the models.\n- The multi-objective optimization algorithm is largely based on the cited paper \"Multi-objective optimization with unbounded solution sets\". Are there novel components proposed in the paper?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject