CE-NAS: An End-to-End Carbon-Efficient Neural Architecture Search Framework
Authors: Yiyang Zhao
Yunzhuo Liu
Bo Jiang
Tian Guo
Venue: NeurIPS 2024 poster (2024)
Keywords: Neural architecture search; Carbon efficient; Optimization; Environment friendly
Primary area: deep_learning_architectures

Abstract: This work presents a novel approach to neural architecture search (NAS) that aims to increase carbon efficiency for the model design process. The proposed framework CE-NAS addresses the key challenge of high carbon cost associated with NAS by exploring the carbon emission variations of energy and energy differences of different NAS algorithms. At the high level, CE-NAS leverages a reinforcement-learning agent to dynamically adjust GPU resources based on carbon intensity, predicted by a time-series transformer, to balance energy-efficient sampling and energy-intensive evaluation tasks. Furthermore, CE-NAS leverages a recently proposed multi-objective optimizer to effectively reduce the NAS search space. We demonstrate the efficacy of CE-NAS in lowering carbon emissions while achieving SOTA results for both NAS datasets and open-domain NAS tasks. For example, on the HW-NasBench dataset, CE-NAS reduces carbon emissions by up to 7.22X while maintaining a search efficiency comparable to vanilla NAS. For open-domain NAS tasks, CE-NAS achieves SOTA results with 97.35% top-1 accuracy on CIFAR-10 with only 1.68M parameters and a carbon consumption of 38.53 lbs of CO2. On ImageNet, our searched model achieves 80.6% top-1 accuracy with a 0.78 ms TensorRT latency using FP16 on NVIDIA V100, consuming only 909.86 lbs of CO2, making it comparable to other one-shot-based NAS baselines. Our code is available at https://github.com/cake-lab/CE-NAS.

CE-NAS: An End-to-End Carbon-Efficient Neural Architecture Search Framework
Yiyang Zhao
Yunzhuo Liu Shanghai Jiao Tong University
Worcester Polytechnic Institute
Bo Jiang Shanghai Jiao Tong University
Tian Guo
Worcester Polytechnic Institute
Abstract
This work presents novel approach to neural architecture search (NAS) that aims to increase carbon efficiency for the model design process. The proposed framework CE-NAS addresses the key challenge of high carbon cost associated with NAS by exploring the carbon emission variations of energy and energy differences of different NAS algorithms. At the high level, CE-NAS leverages a reinforcement-lemming agent to dynamically adjust GPU resources based on carbon intensity, predicted by time-series transformer, balance energy-efficient sampling and energy-intensivy evaluation tasks. Furthermore, CE-NAS leverages a recently proposed multi-objective optimizer effectively reduce the NAS search space. We demonstrate the efficacy of CE-NAS in lowering carbon emissions while achieving SOTA results for both NAS benchmarks and open-domain NAS tasks. For example, on the HW-NasBench, CE-NAS reduces carbon emissions by up to 7.22X while maintaining a search efficiency comparable to vanilla NAS. For open- domain NAS tasks, CE-N NAS achieves SOTA results with 97.35% top-1 accuracy on CIFAR-10 with only 1.68M parameters and carbon consumption of 38.53 lbs of CO2. On ImageNet, our searched model achieves 80.6% top-1 accuracy with a 0.78 ms TensorRT' latency using FP16 on NVIDIA V100, consuming only 909.86 lbs of CO2, making it comparable to other mneeshot-basee NAS baselines. Our code is available at https:/ggthubbommmcccllaa/aaaaa
Introduction
Deep Learning (DL) has become an increasingly important field in computer science, with wide applications such as healthcare and finance [8] 60
63 35 38 Neural architecture
search (NAS) has emerged
means to automate the design of DL models, which involves training many DL models from a massive architecture design space with hundreds of millions to trillions of candidates [102, 66 79 19779999 98 46) Consequently, NAS can be energy-intensive and significantly
contributes to today's ' carbon emissions (68 [02). Many NAS works have reported using thousands of GPU-hours [102 66 79 78 188. For example, Zoph et al. [102] used 800 GPUs for 28 days, resulting in 22,400 hours, to obtain the final architectures. Strubell et al. [68] found that a single
NAS solution can emit as much carbon as five cars during its lifetime.
The environmental impact of NAS, if left untamed, can be substantial. While recent works have significantly improved the search efficiency of NAS [102] [6] 79, 78, 97]. e.g., reducing the GPU- hours tens of hours without sacrificing the architecture quality, there still lacks conscious efforts in reducing curbon emissions. As noted in recent vision paper by Bashir et al. [9], energy efficiency can help reduce carbon emissions but is not equivalent to carbon efficiency. This paper aims to
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Section 3.5.2
Time Series Transformer
CO2 trace with the same energy intensity
Curfenty
Historical
Section agent
Trace forecasting state
Carbon cost
RL rewards
budget
New sompless
#Searched arch
Hypervclume
GPus
Allocation policy
Section 3.3
Section 3.4
Search Space
Pasx
Update
Evaluited samples
Search Algonthmm
for
OLO
Dominance number ranking
Searched
Architccture samples
Accuracy
Archstecture samoles
Energy-efficient sampling
Energy-intensive evaluation
Figure I: An overview of CE-NAS. The sampling and evaluation tasks will be allocated different GPU resources based on carbon emission intensity during the neural architecture search. Specifically, when the CO2 intensity is low, we allocate more resources to energy-intensive evaluation, which focuses on assessing the performance of sampled architectures generated through energy-efficinn sampling. Conversely. when CO2 intensity is high, more GPUs are dedicated to energy-efficient sampling, which utilizes various NAS algorithms sample new architectures from search space partitioned based on previously evaluated architectures. The specific GPU allocation strategy is learned through a reinforcement learning agent, as detailed in the middle section of the figure.
bridge the gap between carbon and energy efficiency with new NAS framework designed to be carbon-aware from the outset.
The proposed framework, termed CE-NAS, tackles the high carbon emission problem from two main aspects. First, CE-NAS regulates the model design process by deciding when use different NAS evaluation strategies based on the carbon intensity, which varies geographically and temporally with the mix of active generators as observed in [9]. To elaborate, given fixed amount of GPU resources, CE-NAS will allocate more resources to energy-efficient NAS evaluation strategies, e.g., one-shot during periods high carbon intensity. Conversely, during periods of low carbon intensity, NAS will shift the focus to running integy-intensive but more effective NAS evaluation strategies, e.g., vanilla NAS 102 66 79 78. We design and train reinforeement learning (RL) agent ($3.5) based on historical and predicted carbon emissions ($5..2 and observed search results to make such allocation decisions. Second, the CE-NAS framework will support searching for DL. models satisfying multiple metrics beyond the commonly used metric of inference accuracy. CE-NAS can consider more metrics such as inference latency, #PARAMs, #FLOPs, etc. For example, CE-NAS search for models with high accuracy and low inference latency. Specifically, both the search and deployment efficiency are achieved by integrating a recent learning- based multi-objective optimizer LaMOO 98 to NAS. Figure depicts the overall workflow of CE-NAS.
NAS 66] 10
To demonstrate CE-NAS'S efficacy in addressing the energy and carbon issues, we conduct com- prehensive evaluation using both commonly used NAS benchmarks and real-world NAS tasks. For example, utilizing carbon emission data from Electricity (58), CE-NAS achieves better per- formance with reduced carbon costs compared to existing methods various NAS benchmarks, including HW-NASBench [40] and NasBench301 93]. In open-domain tasks, CE-NAS achieves the state-of-the-art (SOTA) top-1 accuracy of 97.35% with only 1.68M parameters on the CIFAR-10

---

image classification task, with CO2 costs similar to those of one-shot-based NAS algorithms. For the ImageNet dataset, CE-NAS achieves the SOTA top-1 aceuracy of 80.6% under the same inference latency of 0.78 ms with TensorRT on NVIDIA V100, while maintaining carbon cost compara- ble to SOTA NAS methods [3]. We also evaluate our carbon forecasting model and show that
outperforms existing models 11 56 55], including the SOTA method [55]
In summary, we make the following key contributions.
We introduce a carbon-efficient NAS framework named CE-NAS that can dynamically allocate GPU resources among different NAS evaluation methods naabhod.-namlly.... vanilla NAS and one/few-shot NAS-
carbon emissions data and current search results. CE-NAS is an end-to-end framework that synergistically integrates an RL-based agent for GPU resource allocation, a time-series transformer [2] for carbon intensity prediction, and multi-bjecctive optimizer [98] for reduced search space.
We implement and evaluate CE-NAS on different NAS benchmarks, including HW- NasBench [40] and Nasbench301 [93] We show that CE-NAS can achieve the best search performance compared to only using vanilla NAS, one-shot NAS, and a recently proposed heuristic GPU allocation strategy given the same carbon budget.
CE-NAS delivers SOTA architectural performance in open-domain NAS tasks while main- taining carbon costs comparable to one-shot NAS methods. For example, on CIFAR- 10, CE-NAS achieves a top-1 accuracy of 97.35% with only 1.68M parameters and carbon cost of 38.53 lbs CO2. On ImageNet, CE-NAS reaches a top-1 accuracy of 80.6% with a 0.78 ms TensorRT latency using FP16 NVIDIA V100.
2 Related Work
Efficient neural architecture search (NAS) often focuses on improving the evaluation phase, e.g..
via weight-sharing (one-shot) [65] 46 88
18
T3 zero-shot proxy 44 85| 3 37
73 76
95
74
59
44
70
16,
17), performance predictor 45
13
79 93]. low-fidelity NAS evaluation 102
66
79
78
46 36 and gradient proxy NAS [87. A comparison of these methods can be found in Table in Appendix. Weight-sharing leverages the accuracy estimated by supernet
proxy for the true architecture performance, while gradient proxy NAS uses the gradient as a proxy. These proxy-based methods, although incurring smaller search costs in terms of energy, can have lower search efficiency because their estimated architecture accuracy may have poor rank correlation [97]. Zero-shot proxy NAS eliminates the need for supernet training entirely, serving
fully training-free proxy for network evaluation. However, as noted in [41], mainstream zero-shot NAS methods still struggle to achieve high rank correlations between predicted and true accuracies, even when compared to weight-sharing-ban based NAS evaluation. Performance predictors provide a more accurate performance prediction than weight-sharing and gradient proxy NAS. Still, their accuracy heavily relies on the volume and quality of the training data, which can be very expensive to create (90) [93. Low-fidelity evaluation still requires training each searched architecture. leading limited energy savings. In practice, simply employing these methods without modification might not effectively balance carbon emissions and performance during the search process. CE-NAS successfully achieves good search efficiency, search quality, and carbon efficiency by integrating the weight-sharing NAS method into vanilla NAS. We also leverage the low-fidelity evaluation method in the open-doman NAS search.
The high carbon emission of NAS is a pressing issue. A study by (68] found that single NAS solution emit as much carbon as five cars over its lifetime. For example, if we design a transformer- based model, training a base Transformer model can use 96 GPU hours on NVIDIA's P100 (68), while training larger model can take 28 GPU days. A recent NAS work [68] on transformer revealed that their comprehensive architecture search used 979 million training steps, totaling 274, 120 GPU
hours and resulting in 655 pounds of CO2, emissions. Although several NAS studies [8 B [2] have attempted to address carbon issues during the search process, they often treat carbon emissions uniformly, regardless of time and location variances. These carbon-agnostic NAS methods therefore miss the opportunity to explore the inherent carbon variations 99. Concurrently, systems researchers have explored the temporal and spatial variations exhibited in electricity's 's carbon emission for popular data center applications like training 4 g and inference [39] This work targets a special form of
training workload, i.e., NAS, by leveraging the domain knowledge to optimize the end-to-end NAS process to explore carbon variations. Specifically, concentrate on implementing the temporal
3

---

variations in carbon intensity rather than spatial variations, as managing the carbon costs associated with spatial shifts poses significant challenges. Transferring large volumes of data over the Internet can incur substantial carbon costs, and accurately estimating these costs is complex. Data often passes through numerous intermediate routers across large geographic regions, making difficult to ensure that the benefits of a spatial shift outweigh its associated overhead.
3 The Design of a Carbon-Efficient NAS Framework
In this section, present overview of the proposed DE-NAS framework (.. We first introduce the initial setup of our CE-NAS in $3.2 We then propose a sampling strategy based on the one- shot/few-shot NAS evaluation method in $3.3] Finally, we discuss the architecture evaluation part in $3.4) which is both time and computing ressurce-intensive In $3.5 we describe how to use an RL-based agent automatically allocate GPU resources based on predicted hourly carbon intensity.
3.1 CE-NAS Overview
As observed in 19. grid carbon emissions vary geographically and temporally based on the mix of active generators. Consequently, systems can emit different amounts of carbon even when consuming the same electricity at different locations or times. Currently, neither one (few)-shot nor vanilla NAS methods account for these variations carbon emissions, which can lead to inefficiencies in terms of and vanilla NAS exhibit different search and carbon
NAS carbon usage. Furthermore, one efficiency trade-offs. Vanilla NAS is carbon-intensive but effective, while one (few)-shot NAS can be carbon-friendly but sample-inefficient.
CE-NAS is a NAS framework that directly addresses the high-carbon issues by balancing energy consumption across high-carbon and low-carbon periods. The primary objective of this framework is efficiently search for optimal neural architectures while minimizing the carbon footprint associated with the NAS process. Specifically, CE-NAS decouples the two parts of NAS search process- energy-efficient sampling and energy-intensio evaluation different carbon periods. For each carbon period, CE-NAS uses an RL-bbased agent to automatically allocate GPU resources for the NAS search process to achieve good carbon efficiency.
handles them independently across
3.2 Search Initialization
CE-NAS considers more than search objective(e.g. accuracy, mAP, latency, #FLops, etc), formulate the search problem as a multi-objective optimization (MOO). Similar to other NAS and optimization problems [24 77 98 66), CE-NAS initializes the search process by randomly selecting several architectures, a, from the search space, 22 and evaluating their accuracy, E(a), and other deployment metrics (#Params, #FLoop, inference latency, inference energy, etc) T(a). Compared to the accuracy (aaa). Taa can be quickly measured without much cost. The resulting samples are then added to the observed samples set, P.
We distinguish two types of methods for evaluating the accuracy of an architecture. One actual training. which trains architecture from scratch until convergence and evaluates it to obtain its true accuracy. E(a). To optimize computing resources during this process, we implement the low-fidelity training strategy described in [102 66 46 79 36). This strategy reduces computing costs by simplifying the parameters of neural architectures (e.g., #depth, #channels, #resolutions) or shortening the training process under controlled conditions. The other method is using one-shot which leverages a trained supernet, or zero-shot-based NAS evaluator [95] as performance proxy to estimate the accuracy of the architecture. denoted as E" (a). Note that obtaining E (a) is very cheap and carbon-efficient, however, due to the co-adaption among operations (97) Ea(a) is often not as accurate as E(a). We train all the sampled architectures in the initialization stage to obtain their true accuracy for further search.
evaluation [10
65
(6)]
74 50 59
70
16
17
3.3 Energy-Efffcient Architecture Sampling
Multi-objective search space partition. We leverage the recently proposed multi-objective opti- mizer called LaMOO [98] that learns to partition the search space from observed samples to focus on

---

promising regions likely to contain the Pareto frontier. LaMOO is an optimizer for general black-box optimization problems; we can apply it to NAS as follows.
We utilize LaMOO [98] to partition the search space Ω2 into several subspaces, and find the optimal subspace denoted by Pbest- Next, we construct and train supernet [TQ 977, Sbest- for Ibest- We then use a NAS search algorithm to identify new architectures that will be used to refine the search space. In this work, we employ the state-of-the-ard multi-objective Bayesian optimization algorithm QNEHVI 25. This algorithm will generate new architectures On from Prest, and estimate their approximate accuracy E" (a)) using Shest. At the same time, these architectures an are added to a ready-to-train set R, consisting of candidates for further actual training as described in $3.4
We define the maximum capacity of R Cap(R), hyperparameter in CE-NAS. Note that the architectures in R are removed once they are actual trained as descrihed in $3.4 When the capacity is reached, i.e., when there are more architectures to train than we have resources for, the sampling process blocks until spaces free up in R. The accuracy of architectures estimated by Shest will be fed back into the search algorithm shown in Figure to repeat the process described above.
As mentioned in $3.2 obtaining estimated accuracy through supernet is energy-efficient because these architectures can be evaluated without the time-consuming training. Therefore, during high carbon emission periods, CE-NAS will try to perform this process to save energy and produce as little carbon possible, as shown in the energy- energy-efficient sampling part of Fig [
3.4 Energy-Intensive Architecture Evaluation
If we perform the entire NAS only using the process described $3.3 CE-NAS will be essentially performing one/few-shot NAS only within the subspace Shest. Although these methods are efficient, they typically underperform compared to vanilla NAS. As Zhao al. showed, it is possible to improve LaMOO's space partition with more observed samples through actual training 198] This section describes the process to evolve Stest during low carbon emission periods.
At the high level, we will pick new architectures from the ready-to-train set R to train to convergence and use them to refine the search space partition. That is. the architecture a, with its true accuracy. E(a), will be added to the observed sample set P to help identify more advantageous subspace, Ohest for the architecture sampling process as described in (98]. In this work, we sort the archi- tectures in the ready-to- mady-to-train set R. by their dominance number. The dominance number o(a) of an architecture is defined as the number of samples that dominate search space f2:
o(a) = Σ Ha; -s a, a a. a.En
(I)
where II[:] is the indicator function. With the decreasing of the o(a), a would be approaching the Pareto frontier; v(a)= = O when the sample architecture is located in the Pareto frontier. The use of dominance number allows us to rank an architecture by considering both the estimated accuracy E'(a) and other metrics T(a) at the same time. CE-NAS will first train the architectures with lower dominance number values when GPU resources are available. Once an architecture is trained, is removed from R.
This process depicted the energy-consuning evaluation component of Figure I Note that this process includes actual time-consuming DL model training, which is time-consuming and highly energy-intensive, leading to significant CO2 emissions. Hence, CE-NAS will try to prioritize this process during periods of low carbon intensity.
3.5
Carbon-Eflicient GPU Allocation
The carbon impact of the above two processes (i.e., $3.3 and $3.4) in NAS search is materialized through the use of GPU resources. A key decision CE-NAS needs to make is how to allocate GPUs among these two interdependent processes. Assigning too many GPUs architecture sampling could impact the search efficiency, i.c., the searched architectures are far from the true Pareto frontier; assigning too many GPUs architecture evaluation could significantly increase energy consumption and carbon emission. CE-NAS must consider these trade-offs, under varying carbon intensity and
We define dominance - x as fe(x) < f((y) for all functions fú. and exists at least one i s.t. f((w) < fil)). 1 sis M, where M is the number of objectives.
5

---

Avg Carbon intensity. (g/KWH)
Actual
380
Prediction
350
320
290
260
200
64
Time (hour)
Figure 2: An overview of CE-NAS. This carbon trace is based on the US-CAL-CISO data from 2021, specifically covering the period from 0:00, July 2, 2021, , to 8:00, July 4, 2021. The blue trace its actual carbon trace and the yellow trace the prediction trace by carbon predictor described in sec. 3.5.2
re-evaluate the GPU allocation strategy. design an RL-based strategy ($3.5.1 to automatically allocate GPU resources based on predicted carbon intensity in every one hour ($3..2).
3.5.1 Design of RI.-Based GPU Allocation Strategy
Although the heuristic-basee strategy proposed by can adjust the tendency for sampling and evaluation based on carbon intensity, it is still far from making the most carbon-efficient decision. It overlooks critical influencing factors such as the varying gains in accuracy from performing sampling and evaluation at different stages of the search. These factors are specific the NAS algorithms and are challenging to model, which complicates its inclusion in heuristic strategy design. To address this issue, introduce method based reinforcement learning that automatically accounts for these diverse factors to develop allocation strategies. We propose a Reinforcement Learning (RL)-based method to automatically customize GPU allocation strategy for specific NAS algorithms. In this paper, we choose LaMOO
(8 with qNEHVI (25).
The key to applying RL lies in the design of effective state inputs, actions, and rewards according to the specific task. Our CE-NAS focuses on tailoring these elements for carbon-efficient NAS tasks, with the guiding principle of simplifying the state and action spaces to expedite efficient and rapid training. Specifically, Our RL agent, which is simple-structured neural network with only four fully connected layers, takes the remaining carbon budget, the number of already searched architectures and their hypervolume (The definition the hypervolume refers to Appendix E.2.44 ), as well as the future carbon traces predicted by the time transformer input. Then it outputs the probabilities of different GPU allocation ratios for architecture evaluation as action. These possible allocation ratios discretized boost learning efficiency. After the action executed, CE-NAS generates a reward based on the improvement of the performance of the already searched architectures and then uses the reward to refine the policies of the agent by updating its network parameters with the actor-critic [61] policy gradient algorithm. The specifics of our RI. design can be found in Appendix C and the middle part of Fig.
3.5.2 Design of Transformer-Basee Carbon Intensity Forecasting
We utilize machine learning model to predict future carbon intensity, which serves as the input of the RL agent. We leverage time series transformer-based model, as shown in the top left part of Fig. [ to forecast future carbon intensity because of their efficacy [62] 84 99]. Specifically, we leverage the architecture design by employing standard encoder-decoder Transformer for time series forecasting (2]. This method incorporates temporal features that act as positional encodings within Transformer's encoder/decoder framework. Past values are input into the encoders, and future values are input into the decoders in the training stage. For instance, as described in [2] if time series data point corresponds to the 11th of August, the temporal feature vector would (11, 8), where 11 denotes the day of the month, and 8 signifies the month of the year. We leverage the SOTA

---

Table 1: Daywise MAPE comparison of our time-series transformer and other baseline methods.
Region
Day-1 Forecast
Day-2 Forecast
Day-3 Forecast
STCF 11
DACF [6]
CC [55
STCF
111
DACF
[56
CC
65
STCF 01
DACF [56
CC
55]
18.99
DE
15.54
7.21
7.81
5.34
31.56
11.82
10.69
8.23
42.16
13.95
12.80
10.68
PJM
4.27
3,08
3.69
2,58
7,11
5.1
3.53
8,90
7,06
5.87
4,02
CE-MASiour) Vanilla LaMOO
00,926620g
3.0
38
Log Hypervolume Diff
Gneehot LaMOO
(4,246626g
2.8
Hypervolume
36
c0.2227222g
2.6
cog:26131g
34
10-111682g
2.4
co 62664g
32
CE (ASi
2.2
58279g
30
Oneshot LaMOO
6o 188927g
Hauredi
20
40
60
80
0
100
200
300
400
500
Time(Hours)
Time(Hours)
(a) HW-NAS-Becs
(b) NasBench301
Figure 3: Search progress over time. CE-NAS has the second lowest relative carbon emission while achieving the second best HVog.di diff on HW-NAS NAS-Bench, and CE-NAS has the second lowest relative carbon emission while achieving the second best HV on NasBench301.
NAS algorithm LaMOO 98| to design the architecture of the time series transformer. The details of search space and founded architectures in Appendix D
4 CE-NAS Experimental Evaluation
We develop a prototype of the CE-NAS framework as outlined Section 3 This section analyzes CE-NAS's carbon efficiency and search efficacy through trace-driven simulations and emulation. We first demonstrate the performance of our designed time-series transformer for carbon forecasting in CE-NAS across distinct NAS scenarios: the first leveraging commonly used NAS benchmarks, HW-NAS-Bench (40) and NasBench301
4.1 We then
[93), and the second including real-world computer vision applications, such image classification.
Our experiments utilize carbon trace data sourced from ElectricityMap (58], an independent carbon information service. We selected the CISCL trace beginning July 2, 2021, due its variable carbon intensity, which provides an opportunity to evaluate CE-NAS's performance over time and its adaptability to fluctuating carbon levels. Figure Engureeiilusten illlustraees the initial 80 hours of the carbon trace, demonstrating the actual data against the forecasted trace predicted by our designed time series transformer, detailed in Section 3.5.2 Throughout the CE-NAS search phase, predicted carbon intensity is utilized for GPU allocation, while the actual carbon trace informs the calculation of carbon costs in the evaluation phase.
We also conduct several ablation studies detailed in Appendix These studies include variations of hyperparameters in the RL agent settings, validation of LaMOO's effectiveness, and comparison of search performance using predicted versus actual carbon traces, among others. Our findings reveal only minimal difference in search performance between using predicted and actual carbon traces.
4.1 Time-series Transformer for Carbon Forcasting
We assess the performance of our time series forecasting transformer across three distinct regions: CISO, DEI and PJM We compare our approach against other SOTA models referenced in [1]] 56
CISO: California Independent System Operator
PDE Germany
MPJM: Pennsylvania-Jerss Jersey- Interconnection

---

Table 2: Search Results CIFAR-10 using the NasNet search space. The two optimization objectives we are searching for are #params and accuracy.
Search&Training Cost
Method
Test Error(%)
#Param (M)
CO2 (lbs)
NAS Method
(GPU Hours)
PNAS
C5
3.41 4.41++0.09
3.2
5400
3836.62
vanilla
NAO
3.14:10.09
5400
3836.62
NASNet-
102
2.65
3.3
48000
30534.78
vanilla
LEMONADE
29
2.58
13.1
2160
1514.07
vanilla
AlphaX
70
2.54+0.06
2.83
24000
16196.72
vanilla
AmochaNet
2.50++0.05
2.8
75600
43972.18
vanilla
BayeNAS 100
2.81+0.04
3.4
31.2
21.70
one-shot
DARTS 46
2.76+0.09
3.3
50.4
34.04
one-shot
MergeNAS 80
2.68:0.01
2.9
40.8
27.01
one-shot
One-shot REA
2.68+00033
3.5
44.4
29.33
one-shot
CNAS
2.6010.06
3.7
33.6
23.19
one-shot
PC-DARTS
88
2.57=0.07
3.6
33.6
23.19
one-shot
Fair-DARTS
2.54
3.32
98.4
65.87
one-shot
P-DARTS
2.50
33.6
23.19
one-shot
CE-Net-PI
2.65++0.03
1.68
228.6
38.53
hybrid hybrid
CE-Net-P2
0.06
3.30
228.6
38.53
The total cost includes both NAS search duration training urchitectures identified. When applying one-shot NAS search architectures CIFAR-10, estimate that training these architectures from scratch until convergence requires approximately 26.4 GPU hours.
3.0
Difurerce
Sanee Oifurenee 10
2.9
2.8
2.9
2.7
2.7
2.6
2.5
2.7
2.6
244
2.8
22
24
2.3
22
CE-NAS Verila Onedhol Heuritic
CE-NAS
Viriils GraehalHHuuedte
CE-NAS Varile Heesttt
CE-NAS Vanill Onedhol Heurteii
Mehood
Methoo
Medhod
(a) CO2 cost: 10000g
(b) CO2 cost: 20000g
(c) CO2 cost: 30000g
(d) CO2 cost: 50000g
Figure 4: Search efficiency under carbon emission constraints. These results are based on HW-NAS- Bench with carbon trace showing in fig. and each method ten times.
[1 56 employ Feed- Reed-Forward Neural Network (FFNN) for their forecasts, while (55) utilizes combination of FFNN, Convolutional Neural Network, and Long Short-Term Memory networks for multi-day carbon intensity predictions. We use the Mcan Absolute Percentage Error (MAPE) to evaluate forecasting accuracy. Table demonstrates lay-to-day comparison of all methods, with the most effective technique bold. Our model demonstrates the best performance for both single-day and multi-day carbon intensity forecasting scenarios. More experimental details can be found in Appendix E.1
55 The studies
4.2 NAS Benchmarks and Baselines
To date, there are three popular open-source NAS benchmarks, NasBench101 90 HW-NAS- BENCH (28) and NasBench301 [93. For evaluation, we focus the latter two, the network architectures within these benchmarks cover the entirety of the search space. In contrast, the NasBench101 101 benchmark comprises only limited subset of potential architectures. Evaluating using NasBench101 101 is challenging because we will not have access to important information, such as accuracy, during the search for the architecture types of baselines according to different GPU allocation strategies and NAS evaluation algorithms, including vanilla LaMOO (98), one-sin one-shot LaMOO (98] and heuristic GPU allocation strategy 4 which is the SOTA result so far. We also define a carbon budget as the termination condition. Once the NAS exceeds this CO2, budget, it will be stopped. More bascline details are Appendix E.2.3
the NasBench100 benchmark. We choose three
4.2.1 Simulation using HW-NAS-Bence

---

Table 3: Search Results ImageNet. The two optimization objectives searching for
TensorRT Latency with FP16 in NVIDIA V100 and accuracy.
TensorKT Lateney
Search& Search&Training Cost
Method
Top-1 Error(%)
FP16 V100 (ms)
(GPU Hours)!
CO2 (lbs)
NAS Method
NASNet-A
2.86
48300
30679.11 13
PNAS 45
25.8
2.79
5700
4063.14
vanilla
MausNet 7
24.8
0.53
40300
26487 44
vanilla
AlphaX
24.5
2.52
3900
2768.21
vanilla
AmoebaNet-C
66
24.3
2.63
75900
44 44177.38
vanilla
AutoSlim
91
25.8
1.64
480
321 321.89
one-shot
Proxyles:NAS
14
25.4
0.98
500
332.07
one-shot
SinglePahNASS
11
25.3
1.13
686
455.99
one shot
PC-DARTS 88
24.2
1.50
411
278.15
one-shot
FBNet-C 83
1.22
381.58
one-shot
OFA-Net
20.0
1.16
1315
888.85
one-shot
CE-Net-G1
21.0
0.56
2706
909.86
hybrid hybrid
CE-Net-G22
19.4
0.78
2706
909.86
The total cost includes both the NAS search duration and the training time the architectures identified.
The architecture searched ImageNet directly, otherwise searched CIFAR-10 by transfer setting.
38
37
35
36
Hyppree me
39
31
32
30
31
CE-NAS
Venil Onweher Heureete Malhod
CE-NAS Verill Gneehoo Heuridic
CE-NAS Vanils Greshol Heuletc Mwlhod
CE-NAS
Veniin Dewehoo/feurttt Melhod
Melhod
(a) CO2 cost: 25000g
(b) CO2 cost: 50000g
(c) CO2 cost: 75000g
(d) CO2 cost: 100000g
Figure 5: Search efficiency under carbon emission constraints. These results are based on Nas- Bench301 with carbon trace showing fig. and we ran each method five times.
We evaluate our experimental results using two metrics: relative carbon emission and log hypervolume difference. Note that log hypervolume difference measures the quality of the search result and the smaller the better. Detailed definitions of these metrics can be found Appendix E.2.4 As depicted in Figure 3(a) as the search time progresses, vanilla LaMOO demonstrates the highest performance in terms of H Vogg_iiff diff- Vanilla LaMOO's superior performance can be attributed to its approach of training all sampled architectures obtain their true accuracy, which effectively steers the search direction. However, when considering the relative carbon emission, vanilla LaMOO consumes 3.24X-7.22X carbon compared other approaches. This is expected because vanilla LaMOO an energy-intensive approach and is not designed to be aware of carbon emissions.
We show that CE-NAS's search efficiency is only marginally lower than that of vanilla LaMOO while having the least relative carbon emission. Note that we are plotting the HV in the Y-axis of (3 the actual HV values achieved by CE-NAS and Vanilla LaMOO are about 4100 and 4117, differing only by 0.034%, even though the two lines look far apart. This result also suggests that only relying on energy-efficee
approaches (e.g., one-shot LaMOO this case) insufficient achieve good search performance. CE-NAS only takes 15 hours to get the comparable HV results with the final results of One-shot LaMOO.
F presents a comparative analysis CE-NAS's performance against various baselines under differing carbon budgets. It is evident that CE-NAS surpasses all baselines in search efficiency under different CO2 budget c Below a consumption level of 10,000g CO2, Notably, the search outcomes associated with CE-NAS, depicted in Figure ([() and those of vanilla/one-shot LaMOO, as shown Figure (d)) are comparable terms of the median log hypervolume difference. This indicates that integrating CE-NAS with the NAS strategy can achieve carbon cost saving of up 5X compared to the conventional vanilla/one-shot NAS methodologies.
'We did not account for the carbon cost of training and inference for the RL models and the time-series transformer, they consume negligible CO2 compared the NAS process.

---

4.2.2 Simulation using NasBench301
On NasBench301, we use two metrics: relative carbon emission and hypervolume(HV). Note that higher HV means better search results. Given the expansive nature of the NasBench301 search space, the maximal hypervolume remains undetermined. Consequently, rather than employing the hypervolume difference as performance metric, utilize the absolute hypervolume value repre- sent the efficacy of our search strategy. As depicted in Figure 3(b) our findings are consistent with those from HW-NasBench. One-shot LaMOO incurs the lowest carbon footprint but yields the least impressive performance in terms of hypervolume. Conversely, Vanilla LaMOO achieves the highest IV but at carbon cost that exceeds CE-NAS by more than 3.76 times. JESNAS markedly surpasses the SoTA heuristic GPU allocation strategy regarding HV. while maintaining comparable carbon expenditure. At equivalent levels of carbon cost, as illustrated in Figure [ CE NAS demonstrates significantly superior search performance compared to other baselines. Figures ((a) and | show that architectures identified through CE-NAS not only yield better results but also do so with a carbon cost that is four times lower than that of one-shot and vanilla NAS methods. Within the NasBench301 framework, CES also substantially outperforms the heuristic GPU allocation approach for NAS.
4.3 Open-Domain NAS Tasks
4.3.1 Searching on CIFAR-10 Image Classification Task
Our CE-NAS framework is compared with other popular NAS baselines, including vanilla and one-shot methods. Table presents the SOTA results using DARTS and NASNet search spaces CIFAR- 10. The first group in the table comprises models discovered using vanilla NAS, and the second group includes those found with one-shot NAS. The CO2 cost is calculated based the duration of the search/training and the carbon intensity in the CISO region. For our CE-NAS, we select the optimal architectures from the Pareto frontier of the search results. Our CE-Net-P1 has only 1.68M parameters, significantly reducing parameter size compared to other baselines while maintaining comparable top-1 accuracy (97.35%) with other SOTA models. Our CE-Net-P2 surpasses all baselines in top-1 accuracy (97.84%) while maintaining a similar parameter count of 3.26M. The performance discrepancy between the one-shot (second group) and vanilla NAS (first group) methods is attributed to the supernet's inaccurate accuracy prediction [31] [2]. Remarkably, our CE-NAS not only outperforms all vanilla-based NAS algorithms but also ineurs mere 38.53 lbs of CO2 for the NAS search, comparable to the cost of one-shot-based NAS methods. More details related to the search space, training/scarch setup in More details Appendix E.3.]
4.3.2 Searching on ImageNet Image Classification Task
demonstrates comparison between our CE-NAS and other sfate-of-the-art (SOTA) baselines. We selected our searched models, designated as CE-Net-G1 and CE-Net-G2, from the Pareto frontier, based on the dual objectives of accuracy and TensorRT latency. Our searched architectures incur a slightly higher carbon cost (909.86 lbs) compared to me-shot-based SOTA baselines but significantly outperform these baselines in terms of top-1 accuracy and TensorRT latency. More details on search space, and training/search setup are demonstrated in Appendix E.3.2
Table 3
5 Conclusion
In this work, we described the design of carbon-efficient NAS framework CE-NAS by leveraging the temporal variations in carbon intensity. To search for energy-efficient architectures, CE-NAS integrates a SOTA multi-objective optimizer, LaMOO [98 with the one/few-shot and vanilla NAS algorithms. These two NAS evaluation strategies have different energy requirements, which CE-NAS leverages an RL-based agent schedule when to use each based on average carbon intensity. NAS has demonstrated very promising results across various NAS tasks. For example, on CIFAR- 10, CE-NAS successfully identified an architecture that achieves 97.35% 5% top-1 accuracy with just 1.68M parameters and emitted only 38.53 lbs of CO2, These compelling results suggest the efficacy and potential of CE-NAS as an effective framework in carbon-aware NAS. Additionally, on ImageNet, CE-NAS discovered offe-of-theearr art models achieving top-1 accuracy of 80.6% with a TensorRT latency of 0.78 ms using FP16 on NVIDIA V100, and top-1 accuracy of 79.0% with a latency of 0.56 ms on the same hardware. The carbon cost for this search was only 909.86 lbs.
10

---

Acknowledgement
This work was supported in part by NSF Grants #2105564 and #2236987, VMware grant, the Worcester Polytechnic Institute's Computer Science Department, and the National Natural Science Foundation of China under No. 62072302. This work also used Expanse San Diego Supercomputer Center through allocation CIS230364 from the Advanced Cyberinfrastructure Coordination Ecosys- tem: Services Support (ACCESS) program, which is supported by National Science Foundation grants #2 #2138259, #21 #2138307, #2137603, and #2138296.
References
[1] CO2Scrap.
[2] Time series transformer, http:://nggiiinieeoooooooossssse hhtp:/osssttottommmmmmmmeeeleeee ddcciime_srree__aassooee:e
transformer.
[3] Mohamed S Abdelfattah, Abhinav Mehrotra, Fukasz Dudziak, and Nicholas D Lane. 7ero-cost proxies for lightweight nas. arXiv preprint arXiv:2101.00 101000111//444 2021.
[4] Anomynous Author(s). Anomynous title. In Anomynous Venue, 2023.
[5] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2):235-22) 256, 2002.
[6] Minkyung Back, Frank DiMaio, Ivan Anishchenko, Justas Dauparas. Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang, Qian Cong. Lisa N Kinch, R Dustin Schaeffer, et al. Accurate prediction protein structures and interactions using a three-track neural network. Science, 373(6557):871-8766 2021.
[7] Yixin Bao, Yanghua Peng, and Chuan Wu. Deep learning-based job placement in distributed machine learning clusters. IEEE INFOCOM 2019 IEEE Conference on Computer Com- munications, pages 505-513, 2019.
[8] Yixin Bao, Yanghua Peng, and Chuan Wu. Deep learning-based job placement in distributed machine learning clusters with heterogeneous workloads. IEEE/ACM Transactions on Net- working, 31(2):634-644 647, 2023.
[9] Noman Bashir, Tian Guo, Mohammad Hajiesmaili, David Irwin, Prashant Shenoy, Ramesh Sitaraman, Abel Souza, and Adam Wierman. Enabling sustainable clouds: The case for virtualizing the energy system. In Proceedings of the ACM Symposium Cloud Computing, pages 350-358, 2021.
[10] Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le. Under- standing and simplifying one-shot architecture search. In Proceedings of the 35th International Conference on Machine Learning. 10-15 Jul 2018.
[11] Neeraj Dhanraj Bokde, Bo Tranberg, and Gorm Bruun Andresen. Short-term co2 emissions forecasting based on decomposition approaches and impact on electricity market scheduling. Applied Energy, 281 1:1 16061, 2021.
[12] Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transformation. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
[13] Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once for all: Train one network and specialize it for efficient deployment. In International Conference on Learning Representations, 2020.
[14] Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct neural architecture search on target task and hardware. In International Conference on Learning Representations, 2019.
[15] Jingxuan Chen, Xianbin Cao, Peng Yang, Meng Xiao, Siqiao Ren, Zhongliang Zhao, and Dapeng Oliver Wu. Deep reinforcement learning based resource allocation in multi-uav-aided mec networks. IEEE Transactions Communications, 71(1):296-30 2023.

---

[16] Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. arXiv preprint arXiv:2102.11533, 2021.
[17] Wuyang Chen, Wei Huang, Xianzhi Du, Xiaodan Song, Zhangyang Wang, and Denny Zhou. Auto-scaling vision transformers without training. arXiv preprint arXiv:2202. 1921, 2022.
[18] Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive DARTS: bridging the optimization gap for NAS in the wild. CoRR, abs/1912.10952, 2019.
[19] Zhaoyun Chen, Lei Luo, Wei Quan, Mei Wen, and Chunyuan Zhang. Poster abstract: Deep learning workloads scheduling with reinforcement learning on gpu clusters. In IEEE IN- FOCOM 2019 IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), pages 1023-1024, 2019.
[20] Zheyi Chen, Jia Hu, Geyong Min, Chunbo Luo, and Tarek El-Ghazawi. Adaptive and efficient resource allocation in cloud datacenters using actor-critic deep reinforcement learning. IEEE Transactions on Parallel and Distributed Systems, 33(8):1911 111999111999,,,1111 2022.
[21] Xiangxiang Chu, Tianboo Zhou, Bo Zhang, and Jixiang Li. Fair DARTS: eliminating unfair advantages in differentiable architecture search. CoRR, abs/1911.12126, 2019.
[22] Jingjing Cui, Yuanwei Liu, and Arumugam Nallanathan. Multi-agent reinforcement learning- based resource allocation for uav networks, IEEE Transactions on Wireless Communications, 19(2):729-743, 2020.
[23] X. Dai, P. Zhang, B. Wu, H. Yin, F. Sun, Y. Wang, M. Dukhan, Y. Hu, Y. Wu, Y. Jia, P. Vajda, M. Uyttendacle, and N. K. Jha. Chamnet: Towards efficient network design through platform- aware model adaptation. In 2019 /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1 11390 1399, 2019.
[24] Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Differentiable expected hyper- volume improvement for parallel multi-objective bayesian optimization. arXiv preprint arXiv:2006.05077 05078, 2020.
[25] Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Parallel bayesian optimization of multiple noisy objectives with expected hypervolume improvement. Advances in Neural Information Processing Systems, 34:2187-2200, 2021.
[26] Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. Speeding up automatic hy- perparameter optimization of deep neural networks by extrapolation of learning curves. In Twenty-fourth international joint conference artificial intelligence, 2015.
[27] Jin-Dong Dong, An-Chieh Cheng, Da-Cheng Juan, Wei Wei, and Min Sun. Dpp-net: Device- aware progressive search for pareto-optimal neural architectures. In Proceedings of the European Conference on Computer Vision (ECCV), pages 517-531, 2018.
[28] Xuanyi Dong and Yi Yang. Nas-bench-201. Extending the scope of reproducible neural architecture search. International Conference on Learning Representations (ICLR), 2020.
[29] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Efficient multi-objective neural architecture search via lamarckian evolution. arXiv preprint arXiv:1804.0000 2018.
[30] Oliver Faust, Yuki Hagiwara, Tan Jen Hong, Oh Shu Lih, and U Rajendra Acharya. Deep learning for healthcare applications based on physiological signals: A review. Computer methods and programs in biomedicine, 161:1-13, 2018.
[31] Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. CoRR, abs/1 abs/1194.0000 904.00420, 2019.
[32] Xin He, Jiangchao Yao, Yuxin Wang, Zhenheng Tang, Ka Chu Cheung, Simon See, Bo Han, and Xiaowen Chu. Nas-lid: Efficient neural architecture search with local intrinsic dimension. arXiv preprint arXivv 2022.
12

---

[33] James B Heaton, Nick G Polson, and Jan Hendrik Witte Deep learning for finance: deep portfolios. Applied Stochastic Models in Business and Industry, 33(1):3-12, 2017.
[34] Shoukang Hu, Ruochen Wang, Lanqing HONG, Zhenguo Li, Cho-Jui Hsieh, and Jiashi Feng. Generalizing few-shot NAS with gradient matching. In International Conference on Learning Representations, 2022.
[35] Jian Huang, Junyi Chai, and Stella Cho, Deep learning in finance and banking: literature review and classification. Frontiers of Business Research in China, 14(1):13, 2020.
[36] Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, and Frank Hutter. Fast bayesian optimization of machine learning hyperparameters on large datasets. In Artificial intelligence and stafistics, pages 236..36.1 PMLR, 2017.
[37] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.02340 2018.
[38] Sang Il Lee and Seong Joon Yoo. Multimodal deep learning for finance: integrating and forecasting international stock markets. The Journal of Supercomputing. 76:8294 8312, 2020.
[39] Baolin Li, Yankai Jiang, Vijay Gadepally, and Devesh Tiwari. Toward sustainable genai using generation directives for carbon-friendly large language model inference. arXiv preprint arXiv:2403. 12900, 2024.
[40] Chaojian Li, Zhongzhi Yu, Yonggan Fu, Yongan Zhang, Yang Zhao, Haoran You, Qixuan Yu, Yue Wang, Cong Hao, and Yingyan Lin. (-aas bencch Hardware-aware neural architecture search benchmark. In International Conference on Learning Representations, 2021.
[41] Guihong Li, Duc Hoang, Kartikeya Bhardwaj. Ming Lin, Zhangyang Wang, and Radu Mar- culescu. Zero-shot neural architecture search: Challenges, solutions, and opportunities. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[42] Weihe Li, Jiawei Huang, Wenjun Lyu, Baoshen Guo, Wanchun Jiang, and Jianxin Wang. Rav: Learning-based adaptive streaming to coordinate the audio and video bitrate selections. IEEE Transactions on Multimedia, 25:5662-5675, 2023.
[43] Heechul Lim, Min-Soo Kim, and Jinjun Xiong. (CNAS): Channel-level neural architecture search, 2020.
[44] Ming Lin, Pichao Wang, Zhenhong Sun, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, and Rong Jin. Zen-nas: A zero-shot nas for high-performance image recognition. Proceedings of the IEEF/CVF International Conference Computer Vision, pages 347-356, 2021.
[45] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shiens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan L. Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In European Conference Computer Vision(ECCV) 2018.
[46] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations//CLR))) 2019.
[47] Ning Liu, Zhe Li, Jielong Xu, Zhiyuan Xu, Sheng Lin, Qinru Qiu, Jian Tang, and Yanzhi Wang. A hierarchical framework of cloud resource allocation and power management using deep reinforcement learning. In 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), pages 372-382, 2017.
[48] Tong Liu, Shenggang Ni, Xiaoqiang Li, Yanmin Zhu, Linghe Kong, and Yuanyuan Yang. Deep reinforcement learning based approach for online service placement and computation resource allocation in edge computing. IEEE Transactions on Mobile Computing, 22(7):3870-388) 2023.
[49] Yunzhuo Liu, Bo Jiang, Tian Guo, Ramesh K Sitaraman, Don Towsley, and Xinbing Wang. Grad: Learning for overhead-aware adaptive video streaming with scalable video coding. In Proceedings of the 28th ACM International Conference Multimedia, 2020.
13

---

[50] Vasco Lopes, Saeid Alirezazadeh, and Luís A Alexandre. Epe-nas: Efficient performance estimation without training for neural architecture search. In International conference on artificial neural networks, pages 552-563. Springer, 2021.
[51] Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with restarts. CoRR. htx/1608.039883 2016.
[52] Zhichao Lu, Kalyanmoy Deb, Erik Goodman, Wolfgang Banzhaf, and Vishnu Naresh Boddeti. Nsganetv2: Evolutionary multi-objective surrogate-asso assiiteed neural architecture search. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I, page 35-51, Berlin, Heidelberg, 2020. Springer-Verte
[53] Zhichao Lu, Ian Whalen, Vishnu Boddeti, Yashesh Dhebar, Kalyanmoy Deb, Erik Goodman, and Wolfgang Banzhaf. Nsga-net: neural architecture search using multi-objective genetic algorithm. Proceedings of the genetic and evolutionary computation conference, pages 419 2019.
[54] Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie- Yan Liu. Neural architecture opti- mization. In Advances in Neural Information Processing Systems 31. 2018.
[55] Diptyaroop Maji, Prashant Shenoy, and Ramesh K. Sitaraman. Multi-day forecasting of electric grid carbon intensity using machine learning. SIGENERGY Energy Inform. Rev., 3(2):19-33 jun 2023.
[56] Diptyaroop Maji, Ramesh K. Sitaraman, and Prashant Shenoy. Dacf: Day-ahead carbon intensity forecasting of power grids using machine learning. In Proceedings of the Thirteenth ACM International Conference on Future Energy Systems, e-Energy '22, page 188-192, New York, NY, USA, 2022. Association for Computing Machinery.
[57] Hongzi Mao, Ravi Netravali, and Mohammad Alizadeh. Neural adaptive video streaming with pensieve. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication, SIGCOMM '17 2017.
[58] Electricity Map. Electricity map.
[59] Joe Mellor, Jack Turner, Amos Storkey, and Elliot J Crowley. Neural architecture search without training. In International conference on machine learning, pages 7588-7599 PMLR, 2021.
[60] Riccardo Miotto, Fei Wang, Shuang Wang, Xiaoqian Jiang, and Joel T Dudley, Deep learning for healthcare: review, opportunities and challenges. Briefings in bioinformatics, 19(6):1236 1246, 2018.
[61] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In Proceedings of The 33rd International Conference on Machine Learning, 2016.
[62] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint 147v:2211.147300 2022.
[63] Ahmet Murat Ozbayoglu, Mehmet Ugur Gudelek, and Omer Berat Sezer. Deep learning for financial applications: A survey. Applied soft computing, 93: 10684, 2020.
[64] Zhiping Peng, Delong Cui, Jinglong Zuo, Qirui Li, Bo Xu, and Weiwei Lin. Random task scheduling scheme based on reinforcement learning in cloud computing. Cluster computing, 18: 18:1595-1607 2015.
[65] Hieu Pham, Melody Y. Guan, Barret Zoph. Quoe V. Le, and Jeff Dean. Efficient neural architec- urre search via parameter sharing. In International Conference on Machine Learnin(ICMMI.) 2018.
14

---

[66] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V. Le. Regularized evolution for image classifier architecture search. In Association for the Advancement of Artificial Intelligence(AAAA)), 2019.
[67] Yoko Sasaki, Syusuke Matsuo, Asako Kanezaki, and Hiroshi Takemura. A3c based motion learning for an autonomous mobile robot in crowds. In 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC), 2019.
[68] Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nip. arXiv preprint arXiv: 02299000.02222222 2019.
[69] Xiu Su, Shan You, Mingkai Zheng, Fei Wang. Chen Qian, Changshui Zhang, and Chang Xu. K-shot nas: Learnable weight-sharing for nas with k-shot supernets. In International Conference on Machine Learning, pages 9880-9890. PMLR, 2021.
[70] Zhenhong Sun, Ming Lin, Xiuyu Sun, Zhiyu Tan, Hao Li, and Rong Jin. Mae-det: Revisiting maximum entropy principle in zero-shot nas for efficient object detection. arXiv preprint arXiv:2111.13336, 2021.
[71] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, and Quoc V. Le. Mnasnet: Platform- aware neural architecture search for mobile. In Conference on Computer Vision and Puttern Recognition CVPR), 2019.
[72] Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. CoRR, abs/1905. 1946, 2019.
[73] Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in neural information processing systems, 33:6377- 6389, 2020.
[74] Lucas Theis, Iryna Korshunova, Alykhan Tejani, and Ferene Huszár, Faster gaze prediction with dense networks and fisher pruning. arXiv preprint (xXiv:880.005788 2018.
[75] Shreshth Tuli, Shashikant Ilager, Kotagiri Ramamohanarao, and Rajkumar Buyya. Dynamic scheduling for stochastic edge-cloud computing environments using a3c learning and residual recurrent neural networks. IEEE Transactions on Mobile Computing, 2022.
[76] Chaoqi Wang. Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07376, 2020.
[77] Linnan Wang, Rodrigo Fonseca, and Yuandong Tian. Learning search space partition for black- box optimization using monte carlo tree search. Advances in Neural Information Processing Systems, 33: 13:19551-2 119522, 2020.
[78] Linnan Wang, Saining Xie, Teng L.,, Rodrigo Fonseca, and Yuandong Tian. Sample-efficient neural architecture search by learning action space. CoRR, abs/1906.06832, 2019.
[79] Linnan Wang, Yiyang Zhao, Yuu Jinnai, Yuandong Tian, and Rodrigo Fonseca. Alphax: exploring neural architectures with deep neural networks and monte carlo tree search. CORR, abs/1903.11059, 2019.
[80] Xiaoxing Wang, Chao Xue, Junchi Yan, Xiaokang Yang. Yonggang Hu, and Kewei Sun. Mergenas: Merge operations into one for differentiable architecture search. In Christian Bessicre, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 3065-3072. ijcai.org, 2020.
[81] Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen Fisenach, Woody Ahern, Andrew Borst, Robert J Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with rfdiffusion. Nature, 1089
2023.
[82] Xuekai Wei, Mingliang Zhou, Sam Kwong, Hui Yuan, Shiqi Wang, Guopu Zhu, and Jingchao Cao. Reinforcement learning-bases qoe-oriented dynamic adaptive streaming framework. Information Sciences, 569:786-803, 2021.
15

---

[83] Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang. Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
[84] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34:22419-22430, 2021.
[85] Meng-Ting Wu and Chun-Wei Tsai. Training-free neural architecture search: review. ICT Exprexs, 2023.
[86] Dongkuan Xu, Subhabrata Mukherjee, Xiaodong Liu, Debadeepta Dey, Wenhui Wang, Xiang Zhang, Ahmed Hassan Awadallah, and Jianfeng Gao. Few-shot task-agnostic neural archi- tecture search for distilling large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[87] Jingjing Xu, Liang Zhao, Junyang Lin, Rundong Gao, Xu Sun, and Hongxia Yang. Knas: green neural architecture search. In International Conference on Machine Learning pages 11613-11625, PMLR, 2021.
[88] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. (Prt-/darts}: Partial channel connections for memory architecture search. In International Conference on Learning Representationss 2020.
[89] Hao Ye, Geoffrey Ye Li, and Biing-Hwang Fred Juang. Deep reinforcement learning based resource allocation for v2v communications, IEEE Transactions on Vehicular Technology, 38(4):3163-3177 2019.
[90] Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. NAS-bench- 101: Towards reproducible neural architecture search. In Proceedings of the 36th International Conference on Machine Learning, 2019.
[91] Jiahui Yu and Thomas S. Huang. Network slimming by slimmable networks: Towards one-shot architecture search for channel numbers. CoRR, 1tbb//993..1100 1128, 2019.
[92] Kaicheng Yu, Christian Sciuto, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Eval- uating the search phase of neural architecture search. arXiv preprint 1wXiv:1902.08144 2019.
[93] Arber Zela, Julien Niklas Siems, Lucas Zimmer, Jovita Lukasik, Margret Keuper, and Frank Hutter. Surrogate NAS benchmarks: Going beyond the limited search spaces of tabular NAS benchmarks. In International Conference on Learning Representattinss 2022.
[94] Yu Zhang, Jianguo Yao, and Haibing Guan. Intelligent cloud resource management with deep reinforcement learning. IEEE Cloud Computing, 4(6):60-6) 69, 2017.
[95] Zhihao Zhang and Zhihao Jia. Gradsign: Model performance inference with theoretical insights. arXiv preprint arXiv:2 10.00666, 2021.
[96] Xiaoyang Zhao and Chuan Wu. Large-scale machine learning cluster scheduling via multi- agent graph reinforcement learning. IEEE Transactions on Network and Service Management, 19(4):4922-497 2022.
[97] Yiyang Zhao, Linnan Wang. Yuandong Tian, Rodrigo Fonseca, and Tian Guo. Few-shot neural architecture search. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12707-12718 PMLR, 18-24 Jul 2021.
[98] Yiyang Zhao, Linnan Wang, Kevin Yang, Tianjun Zhang, Tian Guo, and Yuandong Tian. Multi-objective optimization by learning space partition. In International Conference on Learning Representations, 2022.
16

---

[99] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106-11115, 2021.
[100] Hongpeng Zhou, Minghao Yang, Jun Wang, and Wei Pan. BayesNAS: A Bayesian approach for neural architecture search. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Pro- ceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 7603-7613. PMLR, 09-15 Jun 2019.
[101] Huan Zhou, Zhenning Wang. Hantong Zheng, Shibo He, and Mianxiong Dong. Cost minimization-orienee computation offloading and service caching in mobile cloud-edge computing: An a3c-based approach. IEEE Transactions Network Science and Engineering, 2023.
[102] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe Le. Learning transferable ar- chitectures for scalable image recognition. Conference on Computer Vision and Puttern Recognition (CVPR), 2018.
17

---

A Background Knowledge
A.1 Multi-objective NAS
Multi-objective NAS has gained popularity in the NAS community, as deep neural networks need to address not only traditional metries like accuracy but also practical efficiency metrics. There are two main approaches within Multi-objective NAS. The first category [83 71 13 23 27] leverages linear scalarization of various metries (e.g., accurecy into single metric, followed by the application of #F.OPs
standard single-objective NAS algorithms for architecture search. The second category 98 53 52
approaches MONAS as multi-objective black-box optimization problem. According to existing results 198 52], this latter approach generally outperforms the former in terms of effectiveness and performance. For multi-objective optimization, mathematically, we optimize M objectives f(x) (2)) (M(()) E pM:
min f1(x), f2(2)).... s.t.
x E Q
where æ represents the neural architecture from the search space, f((x)) denotes the function of objective Modern MOO methods aim find the problem's entire Pareto frontier, the set of solutions that are not dominated by any other feasible solutions. Here we define dominance y <5 æ as fx) S fzy)) for all metrics fú, and there exists least one s.t. f((z) fi(y), 1 I M. the condition holds, solution æ is always better than solution
Table 4: Comparison of energy-efficient NAS evaluation methods. Eval. cost refers to the cost of obtaining the evaluation results. Init. cost describes additional dataset preparation and the time required for training the model (c.g., supernet or predictor). Accuracy measures the rank correlation between the evaluation method and the actual rank. Predictor-based methods require Extra data as training set to construct the prediction model.
Method
Eval. cost
Init. cost
Accuracy
Extra data
Zero-shot proxy 41
16
Low
Low
Low
No
One-shot 46
65
97
13
Low
Low
Intermediate
No
Predictor 45 26
Low
Hight
Hight
Yes
Low-fidelity 102 66
36
High
None
High
Gradient Proxy 87
Low
Low
Intermediate
No
It depends the size of extra data.
depends the extent of the fidelity.
A.2 One (few)-shot NAS
One (few)-shot NAS utilizes a weight-sharing technique, avoiding the need to retrain sampled networks from scratch. Specifically, one-shot NAS initially trains an wee-paraameterized supernet, including all possible operations and connections within the entire search space, and employs this supernet as performance estimator to predict an architecture's efficacy [O 92 97 65
The performance any architecture within the search space can be approximated by leveraging the well-
trained supernet. Few-shot NAS [97] 34 86 32 69) further improves the accuracy of architecture
evaluations conducted by supernets. achieves this by using multiple sub- sub-supernets, thereby reducing the co- adaatatioon impact among operations [10]. In this work, employ this few-shot supernet as an efficient proxy for architecture evaluation, with more details provided in Section 3.3
A.3 Zero shot proxy NAS
Zero-shot NAS leverages training-free score functions/proxies to efficiently evaluate the performance of neural architectures [41 85 Compared to vanilla NAS and one/few-shot NAS, zero-shot NAS cost-effective due to its training-free nature at the search stage. However, the main disadvantage of zero-shot NAS is evaluation performance, which can be very inaccurate compared to one/few-shot
NAS and vanilla NAS 41)
18

---

A.4 Reinforcement learning (RL)
RL has been widely used for designing system strategies for tasks such as scheduling and resource allocation. For instance, RL has been deployed to develop resource allocation strategies in wireless networks [22 89 15 48 and cloud platforms [99
94 20), as well as for making bitrate decisions 49, 42 82 and also generating task scheduling strategies in data centers 664, 96, 81. The key advantage of RL-based methods is their capability to learn strategies and easily adapt to system changes automatically. An RL agent begins by taking the system state as input and proceeds to generate an action, i.c. a scheduling or allocation decision, based on its learned strategies, which is typically randomized the start. Upon execution of the action, the system provides feedback in the form of reward, enabling the RL agent to refine its strategies. The key to applying RL lies in the design of effective state inputs, actions, and rewards according the specific task. Our CESNAS focuses on tailoring these elements for carbon-efficient NAS tasks, with the guiding principle of simplifying the state and action spaces expedite efficient and rapid training.
in adaptive video streaming systems [57]
B Learning Search Space Partition
We utilize LaMOO 98 partition the search space, Ω into several sub-search spaces. This partitioning will be based on the architectures' accuracy E(a) and their evaluation metrics T(a) as observed in the sample set, P. Specifically, LaMOO recursively divides the search space into promising and non-promising regions. Each partitioned region can then be mapped to a node in search tree. Using Monte-Carlo Tree Search (MCTS), LaMOO selects the most promising sub-space (i.e.. tree node) for further exploration based on their UCB values [5) This optimal sub-space selected by MCTS is denoted as Qsest.
C Details of Reinforcement Learning Based GPU Allocation Strategy
State Input. We design the RL state input at time step L. as 8, = (br, nt, nt, he, ct),, where b, is the remaining carbon budget of the current searching task, n, is the number of already searched architectures, h the hypervolume of these architectures, and c is the future carbon traces in the upcoming hour (e.g., predicted by our time series transformer from $3.5.2). The general idea is that being aware of b, helps the RL algorithm plan the overall amount of carbon emission to spend for architecture evaluation, while n4, he and ct help it decide the best timing to execute the evaluation that generates the highest hypervolume improvements per carbon cost.
Network architecture. Our network consists of an input layer. 2 hidden layers, and an output layer. All of the layers are fully connected layers, with hidden sizes of 100, 150, 200 and 100 respectively, which are determined by performing a simple grid search. The final output of the network specifies the probability of distribution of actions.
Action.
We design the output of the RL agent as K discrete actions, i.c. the output is a vector where of denotes the probability of allocating ratio of (k-1) K GPUs for
of Or =< of of. >, architecture evaluation. We use K = for our evaluation. We also implement version of the agent with continuous action space. i.e., the network outputs the mean and standard o of Gaussian distribution to indicate the probability distribution of the GPU allocation ratio. Our evaluation in $F44 shows that the two versions achieve comparable performance.
Rewards. As the overall goal is to maximize the cumulative reward that represents the final accuracy obtained by the search task, we design the reward in each time step the improvement of the best accuracy of the already searched architectures: r; (co,, hig, nn.), where co2 denotes the carbon cost in this iteration, hi, represents the hypervolume increase in this iteration, and nın, means the number of new samples in this step.
Policy gradient. We adopt the popular actor-critic algorithm to calculate the policy gradient and update the network. Aclor-critic has been proved to achieve good performance while enabling fast
training tasks of similar scales [57 49 75 67 [0I). Note that our design not coupled with any
specific DRL algorithm. Thus, replacing actor-critic with other algorithms is easy if needed. The key
19

---

Any Catton blurady 0p (pr0000)
Actual
yon biteraay garovt
600
Adual
Calbou iiturally daro0Ao
+.00
crodiction
650
Prediction
480
600
200
450
470
400
Actual
200
00:0
05:00
12:58
18:0
00:09
350
00:00
06.00
12:50
18:00
00
450
0090
06:05
12:59
(8
06
Local time
Local time
Local time
(a) Region: CISO
(b) Region: DE
(c) Region: PJM
Figure 6: Transformer- based carbon intensity predictor. Our predictor makes forecasts that perfectly match actual values over three different regions.
gradient equation of actor-critic as follows:
V PITTO
7're
Eity Vo (o)
where Ei=g 7' the cumulative rewards and y is discount factor. Actor-critic algorithm includes two networks referred to as actor and critic respectively. In our design, they use the same network structure. The actor is responsible for outputting (s, a). which the output probability for action with input state s. The policy parameter is the network parameter of the actor. A a) is the advantage function that indicates the performance difference between the current policy and the average performance of policies learned by actor. A (s, is obtained with the temporal difference method based on the output of the critic. The calculation as follows,
A(s,, rı ~V ($141:0g) ~ (s);
where A(s,, 0z) is an unbiased estimation of A a). 0, is the parameter of the critic. V (st; 0.) the output critic that estimates the cumulative reward follow actor policy my starting from state
D Architecture Design of Time series Transformer
Building upon the foundational design of the time series Transformer, we employ LaMOO 98 as our NAS strategy identify the optimal architectural parameters for carbon dataset (58), Our goal is to devise an architecture that not only achieves high forecasting accuracy but also maintains efficient training and latency times. Given that our search space is straightforward. We focus on searching for: i) the number of encoder layers, ii) the number of decoder layers, iii) the embedding size, and the maximum context length of the time series that the model is capable of considering. Following our NAS search process, we configured a model with 64 encoder layers, 96 decoder layers, with embedding size of 48. Additionally, the context length to 8760.
working with standard encoder-decoder Transformer,
E Experimental Details in Sec.
E.1 Time-series Transformer for Carbon Forcasting
We align our training and testing data with the approach used in [55 by utilizing ElectricityMap [58 a sarry carbon information service. We select three regions (namely, CISC DE and PJM from PlectricityMap [58] to represent distinct carbon trace scenarios for our experiments. The dataset spans from January 1, 2020, December 31, 2021, with an hourly resolution. We employ train vrin(validationn-tes split of 75%-25% 25%. We trained a single transformer model using data from three
CCISO California Independent System Operator
7 DD: Germany
PPMM: Pennsylvania-Jess Jersey--aaaaeeee Interconnection
20

---

different regions simultaneously. Additionally, we conducted separate training sessions for the model using distinct traces from each location. The experimental results from both approaches were similar.
In our training configuration, the context length set to 8760, as determined by our NAS search. We utilize the AdamW optimizer, setting the weight decay the exponential decay rate of the first moment estimates, and beta2 as 0.95 for the second moment estimates. The model is trained over 150 epochs with an initial learning rate of x 10-4. The batch size is configured at 512 on a single NVIDIA A100 GPU.
10-2 and selecting betal 0.9 for
For testing, predict carbon intensity for the forthcoming 24 hours, to the last complete 24-hour block. This block is then replaced with predictions from our model to forecast the carbon intensity for the 25th hour. Subsequent forecasts are generated by updating the most recent hour with its actual value and using the model's 's prediction for the next hour. We measure forecasting performance using the Mean Absolute Percentage Error (MAPE). For multi-day forecasts, we similarly update and replace values for the 24-, 48-, and 72-hour marks.
all historical data up
Figure 6 presents the hourly time series, averaged over a week, demonstrating both actual and predicted carbon intensities for the electricity grids in these areas. The comparison suggests that our model accurately forecasts the actual carbon trace in each region, aligning closely with the observed data.
E.2
Experimental Details on NAS Dataset
E.2.1 Overview of HW-NAS-Bench
HW-NAS-Bendh [40 enhances the original NasBench201 dataset by including additional metrics such as inference latency, parameter size, and number of FLOPs, thereby expanding the evalua- tive dimensions available for NAS research. NasBench201 itself is a comprehensive open-source benchmark designed for the systematic assessment of NAS algorithms (28]. Within NasBench201, architectures are constructed by stacking cells in sequence. Specifically, each cell is composed of 4 nodes interconnected by 6 edges. Nodes represent feature maps, while edges correspond to various operations that transform one node's output into the next node's 's input. Operations connecting nodes include zeroization, skip-connection, IxI convolution, 3x3 convolution, and 3x3 average pooling. To uniquely identify each architecture, we employ numeric encoding scheme where the five operations are represented by integers through Consequently, 6-element vector represents the specific architecture.
E.2.2 Overview of NasBench301
NasBench301 [93] serves as a surrogate benchmark for the NASNet [102] search space, which has over 10 possible architectures. NasBench301 employs surrogate model trained on approximately 60,000 sampled architectures to predict performance across entire DARTS 46 search space for the CIFAR-10 image classification task. This surrogate model within NasBench301 has been shown provide accurate regression outcomes, facilitating reliable evaluations within this vast search space. NasBench301 offers predicted accuracies from the surrogate model, while other fundamental metrics such as the number of parameters (#Params), floating-pois operations per second (#FLOPs), and inference time are readily ascertainable during the evaluation phase. In this multi-objective optimization context, our goal is to simultancously maximize inference accuracy and minimize #Params within the NasBench301 search space.
The NASNet search space includes operations such as 3x3 max pool, 3x3 convolutions, 5x5 depth- separable convolutions, and skip connections. The objective here is to identify optimal architectures for both reduction and normal cells, each comprising nodes, culminating in a search space of approximately 3.5 x 10 architectures [9] For the CIFAR-10 task, we adopt the same encoding strategy used in the NASNet search space, representing architectures as 16-digit vectors. The first four digits denote the operations in normal cell, digits 5-8 correspond to the concatenation patterns in the normal cell, digits 9-12 represent operations in a reduction cell, and the final four digits encode the concatenation patterns in the reduction cell.
21

---

E.2.3 Baselines
We chose three types of baselines according to different GPU allocation strategies and NAS evaluation algorithmms During the search process, all search methods employ the state-of-the-art multi-objective optimizer, LaMOO 198). Specifically, one-shot LaMOO is method that utilizes one-shot evaluations throughout the search process. The vanilla LaMOO relies on actual training for architecture evaluation throughout the search.
We also provide a heuristic strategy that automatically allocates GPU resources between the sampling and evaluation processes given the carbon emissions C₁ at time as our baseline. This allocation is based on the energy characteristies of the processes: architecture sampling is often energy-efficient because it does not involve actual training of architectures, while architecture evaluation is often energy- mergyyconsumiing because does. We assume that the maximum and minimum carbon intensities Cmaz and Cmin for a future time window are known. G2 denotes the total number of available GPUs. de and X, represent the ratio of GPU numbers allocated to the evaluation and sampling processes at given moment, and Xe + 1. We calculate A as Cmas "Cma=-Cmin" where Ceur is the current carbon intensity. The GPU allocations for the sampling and evaluation processes are, therefore, G. X and G. Ae This simple heuristic allocation allows the NAS system to prioritize more energy-efficient sampling tasks during periods of higher carbon intensity, whereas, during low-carbon periods, the system will allocate more resources for energy -ntegy-intensiveee evaluation tasks.
Canc- Cuuin.
E.2.4 Metrics in HW-NAS-Bench
We use two main metrics to evaluate the carbon and search efficiency of CE-NAS. First, we use relative carbon emission to quantify the amount of CO2 each NAS method is responsible for. The relative carbon emission is calculated by summing the average carbon intensity (in gCO2/KwHI) over the search process. We assume that all NAS methods use the same type of GPU whose power consumption remains the same throughout the search process. Second, we use the metric hypervolume (HV) to measure the "goodness" of searched samples. HV is a commonly used multi- objective optimization quality indicator 124 25 98 that considers all dimensions of the search objective. Given reference point R E MM the HV of finite approximate Pareto set P is the M-dimensional Lebesgue measure Xas of the space dominated by P and bounded from below by
R. That is, HV,P, Rt) = Aas (U/ [R, ytl). where [R. vi] denotes the hyper-rectangle bounded by the reference point Rt and yi. A higher hypervolume denotes better multi-objective results. In this experiment, we calculate the hypervolume (HV) using the accuracy and inference energy of the searched architectures. For this dataset, we use the log hypervolume difference, the same as in 22|22 18), as our evaluation criterion for HW-NASBench, since the hypervolume difference may be minimal over the search process. Therefore, using log hypervolume allows us to visualize the sample efficiency of different search methods. We define H/ Vo diff log( /V max HV where HVmax represents the maximum hypervolume calculated from all points in the search space, and HVeur denotes the hypervolume of the current samples, which are obtained by the algorithm within specified budget. The V/aa in this problem is 4150.7236.
For our simulation, we use the training and evaluation time costs for the architectures derived from NasBench201 (28), and inference energy costs measured on the NVIDIA Edge GPU Jetson TX2 from HW NASBechhhh 40 We ran the simulation 10 times with each method.
E.2.5 Metrics in NasBench301
Given the expansive nature of the NasBench301 search space, the maximal hypervolume remains undetermined. Consequently, rather than employing the hypervolume difference as a performance metric, we utilize the absolute hypervolume value represent the efficacy of our search strategy.
E.3
Experimental Details on Open-Domain NAS Tasks
E.3.1 Details on Cifar10 Image Classification Task
Search space overview. Our search space consistent with the NASNet search space [102] comprises eight searchable operations: 3x3 max pooling, 3x3 average pooling, 3x3, 5x5, and 7x7 depthwise convolutions, 3x3 and 5x5 dilated convolutions, and skip connection. The architecture consists of normal cells, which retain the size of the feature map, and reduction cells, which double
22

---

the channel size and halve the feature map resolution. Each cell integrates four nodes connected by eight different operations. The search space encompasses a total of 10 architectures. We employ the same encoding strategy prior studies (78]
Search setup We implement low-fidelity estimation methods 102 66 79 to expedite the energy- consuming evaluation part in the neural architecture search process. This approach effectively accelerates the evaluation process using cost-effective approximations while largely preserving the true relative ranking of the searched architectures. Initially, we utilize early stopping for training sampled architectures for 200 epochs, in contrast to 600 epochs required for the final training phase. To further hasten training, we reduce the initial channel size from 36 to 18 and increase the batch size 320. Additionally, the number of layers is scaled down from 24 to 16 during the search phase. To speed up the evaluation process, we use the NVIDIA Automatic Mixed Precision (AMP) library with FP16 for training during the search.
We utilize a pre-trained reinforcement learning model based on data from NasBench201 and continu- ally tune this model during the NAS process according task-specific data. The carbon budget for the search is set at 100 lbs.
Training setup The final selected architectures are trained for 600 epochs, using a batch size of 128 and momentum SGD optimizer with an initial learning rate of 0.025. This rate is adjusted following cosine learning rate schedule throughout the training. Weight decay is applied for regularization.
E.3.2 Details on ImageNet
Search space overview. Our ImageNet search space is modeled after EfficientNet 722). Specifically, includes 5-8 stages for each architecture, with the number of stages being determined by NAS. From Inverse-Residus -Block (Fused-IRB) or
stages 3 to 8, the type of stage, either
Block (IRB), is searchable. Within each stage, searchable parameters include activation type (c.g.. ReLU, Swish), kernel size (e.g.. 1, 3, 5, 7), number of layers ([I, 10]), expansion rate ((2, 7J), and number of channels (varies based on the stage). Additionally, our search space considers input image resolutions (e.g., 224, 288, 320, 384, 456, 528). The total search space size is approximately 1 Our NAS search focuses on two objectives: top-1 accuracy and TensorRT latency with FP16 on an NVIDIA V100. For TensorRT latency, we fixed the workspace at 10GB for all runs and benchmarked latency using a batch size of with explicit shape configuration, reporting the average latency from 1000 runs.
Search setup As with CIFAR-10, we implement fow-fidelity estimation methods [102 accelerate the energy-consuming evaluation part in the neural architecture search process. Sampled architectures are trained for 150 epochs using early stopping, as opposed the 450 epochs required in the final training phase. During the evaluation in the search process, we also reduce the channel size by a factor of 4. To further speed up the evaluation process, we leverage AMP with FP16 for training searched architectures.
66 79 to
Consistent with our CIFAR-10 approach, we employ a pre-trained reinforcement leamning model based on data from NasBench201 and continually fine-tune this model during the NAS process using ImageNet task data. The carbon budget for the search is set at 1000 lbs.
Training setup For each architecture in the Pareto frontier, we train on 8 Tesla V100 GPUs with a resolution of 320x320 in our two-objective accuracy and TensorRT latency. We utilize standard SGD optimizer with Nesterov momentum of 0.9 and set the weight decay at 3 x 10 33 Each architecture undergoes training for total of 450 epochs, with the initial 10 epochs serving as the warm-up period. During these warm-up epochs, we apply a constant learning rate of 0.01. The remaining epochs are trained with initial learning rate of 0.1, using cosine learning rate decay schedule [51], and batch size of 1024 (i.e., 128 images per GPU). The model parameters undergo decay at a factor of 0.9997 to further enhance models' training performance.
F
Ablation Studies
F1
Effectiveness of LaMOO for NAS
We conducted ten runs of LaMOO (i.e., search space split) with a random search on the HW- NASBench dataset [40]. In addition, we performed random sampling for both the LaMOO-sclected
23

---

80
40
EdgeGpu Energy
Hypervolume
3900
Accuracy
80
3700
3600
10
20
3500
LaMO0 selected space Whole space
LaMOO selected spece Whole space
LaMOO selected space Whole space Region
Region
Region
Hypervolume
(b) Accuracy
(c) Inference energy
Comparisons of architecture qualities between LaMOO-selccted region and the entire search space of HW-Nasbench. We ran LaMOO 10 times. For each run, we randomly sampled 50 architectures from the LaMOO-seleche space and the whole search space.
Figure 7:
energy--cneeumigg
Ca fbon
0.8
rallo
0.4
0.2
10 20
70
80
Time(hours)
Figure An overview CE-NAS. This carbon trace and GPU ratio.
region and the entire search space, conducting 50 trials for each. The distribution of accuracy and edge GPU energy consumption of the architectures in both the LaMOO selected region and the entire search space can be seen in Figure 7
Specifically, results show that the architectures in the region selected by LaMOO have higher average accuracy and lower average edge GPU energy consumption compared to those in the entire search space. On average, the accuracy of the architectures in the LaMOO selected region is 72.12, while the accuracy in the entire search space 68.28. The average edge GPU energy for the LaMOO selected region is 16.59 mJ, opposed to 22.84 mJ for the entire space.
Furthermore, as illustrated Figure 7(a) we observe that searching within the LaMOO-selected region yielded a tighter distribution, and the median hypervolume demonstrated improvement compared searching across entire search space. These results suggest the efficacy of using LaMOO to partition the search space for NAS.

---

11333
10,53353
Lh.15080g
626644
400%%
(P/eat en
Wriee
Chantel
COMMoNi
Veboc
(a) Search budget:10 hrs
(b) Search budget:20 hrs
(c) Search budget:40 hrs (d) Search budget:80 hrs
Figure 9: Search efficiency under time constraints. These results are based on NasBench201 with carbon trace showing in fig. 2 and ran each method seven times.
Table 5: Search efficiency under carbon emission constraints in terms of Hypervolume. We use the transformer- based carbon predictor in sec. 3.5.2 to forecast the carbon trace and use the actual carbon trace as the baseline. We ran each method ten times and made the average.
Carbon Constrain
CO2 Cost: 5000g
CO2 Cost: 10000g
CO2 Cost: 30000g
CO2 Cost: 50000g
Prediction Actual
3724.84
3774.21
3891.70
3966.70
3742.36
3796.93
3898.43
3971.16
E2 CE-NAS framework with Time Budget
Our framework also supports neural architecture search (NAS) within a specified time budget, in addition the option of a carbon budget. We adapt the time budget replace the carbon budget in the state and reward functions of reinforcement learning. Figure 9 presents the results of our Carbon-Efficienn NAS (CE-NAS) under various time budgets. As illustrated, CE-NAS significantly outperforms both heuristic and one-shot methods in terms of hypervolume across different time budgets. While there is minor decline in performance compared to vanilla NAS measured by hypervolume, it is important to note that the carbon cost of our CE-NAS is on with that of the heuristic method. This represents a substantial reduction compared to vanilla NAS. For instance, as depicted in Figure P(d) vanilla NAS incurs three times the carbon cost of CE-NAS but only achieves a marginal improvement in hypervolume.
F33 CE-NAS framework with actual carbon trace
As detailed in Section 3.5.2 we implemented transformer-basee carbon predictor to forecast the carbon intensity for the upcoming hour, integrating this predicted value into our framework. The accuracy of this forecast is demonstrated in 2 In the same section, we compare the performance of NAS searches using both the predicted carbon intensity and the actual carbon intensity. Table5) displays the results of our system operating under various carbon budget constraints with both predicted and actual carbon intensities. The data in this table indicates that, across different carbon budgets, employing the predicted carbon intensity trace yields results that are only marginally less effective than using the actual carbon trace, as measured by hypervolume.
Table 6: Search efficiency under carbon emission constraints in terms of Hypervolume. We use the continuous action space of the reinforcement learning method introduced in sec. 3.5.1 We ran each method ten times and made the average.
RL Action Space
CO2 Cost: 5000g
CO2 Cost: 10000g
CO2 Cost: 30000g
CO2 Cost: 50000g
Discrete
3724.84
3774.21
3891.70
3966.70
Continuous
3645.21
3748.83
3853,86
3920.47
25

---

E4 CE-NAS framework with continuous RL Action Space
Recall that our RL design includes both version with discrete action space and another with a continuous action space. We compare their performance in Table O and observe that the two versions deliver comparable performance, with the continuous version performing slightly worse. experiencing a degradation within 3%. Such degradation can be attributed to that the continuous version complicates the action space and increases the learning difficulty. Despite this, we anticipate that the continuous version will outperform its discrete counterpart as the number of available GPUs grows, owing to its more fine-grained GPU allocation decision granularity that gives it the potential to generating better-optimized strategies. We leave the exploration of how GPU cluster sizes impact the relative performance of the two version for future study.
26

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
27

---

Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer: [Yes]
Justification:
See supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cpippi.ccppulli//uuuii wwwiicccguie//eeeeeeeeeis CoodeSubeiiie ssi ssionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Appendix. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Appendix. and Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
29

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix.
E
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: haapas:///uripps/ http///uris.ccpppupppp/e http:///.cccpuccccttttcttttteeea
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
30

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworee.wtchhe....c.. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
31

---

Answer:
[Yes]
Justification:
See our supplementary material.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]-
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "4uiAUzkpT0", "reviewer": "Reviewer_CjFX", "summary": "The paper introduces CE-NAS, a novel framework for neural architecture search that prioritizes carbon efficiency in the model design process. It addresses the high carbon cost associated with NAS by dynamically adjusting GPU resources based on predicted carbon intensity and search results. The framework integrates a reinforcement-learning agent, a time-series transformer for carbon intensity prediction, and a multi-objective optimizer to balance energy-efficient sampling with energy-intensive evaluation tasks. Experiments demonstrate CE-NAS's ability to reduce carbon emissions significantly while achieving good results on various NAS datasets and tasks.", "questions": "Please describe the advantages of CE-NAS when compared to zero-shot NAS methods.", "limitations": "None", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* CE-NAS presents a unique and timely approach to NAS that considers carbon efficiency, aligning with broader environmental sustainability goals.\n* The integration of a reinforcement-learning agent, time-series transformer, and multi-objective optimizer provides a comprehensive solution for carbon-aware NAS.\n* The framework's effectiveness on various datasets suggests that it could be widely applicable to different NAS scenarios.", "weaknesses": "* Experimental comparisons are focused on traditional and one-shot NAS methods. However, zero-shot NAS methods are not mentioned in the paper. Some recent zero-shot NAS methods have achieved better results and higher efficiency. \n* The integration of multiple components introduces complexity in implementation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for your insightful comments and suggestions, please check our answers below.\n\n---\n\n**Comparison between CE-NAS and zero-shot NAS** \n\nWe first thank the reviewer for pointing out the missing related work on zero-shot NAS in our paper.  We will add the following discussion, comparison, and potential integration with zero-shot NAS in our final version. \n\nFirst, Zero-shot NAS leverages training-free score functions/proxies to efficiently evaluate the performance of neural architectures [15][16]. Compared to vanilla NAS and one/few-shot NAS, zero-shot NAS is cost-effective due to its training-free nature at the search stage. However, the main disadvantage of zero-shot NAS is its evaluation performance, which can be very inaccurate compared to one/few-shot NAS and vanilla NAS. According to [15], mainstream zero-shot NAS methods, such as gradient norm [17], SNIP [18], Synflow [19], GraSP [20], GradSign [21], Fisher [22], Jacobian_cov [23][24], Zen_score [25][26], and NTK [27][28] did not achieve high rank correlations between architectures’ predicted and true accuracy. For example, in the NasBench201 search space for CIFAR10, the highest Kendall’s tau value (a ranking correlation metric ranging from -1 to 1, where 1 indicates identical ranking and -1 indicates completely reversed ranking) from these zero-shot methods is below 0.6 [21], with most values ranging from 0 to 0.5. In contrast, one/few-shot NAS [29][30] can achieve Kendall’s tau value of 0.75. Importantly, for the top 5% of architectures ordered by true accuracy, most zero-shot methods [17,18,19,20,22,23,24,27,28] have Kendall’s tau value around or below 0, equivalent to random search. Zen-score [25, 26] and GradSign [21] perform slightly better than others but still below 0.3. Based on our experiments, few-shot NAS [29] achieves Kendall’s tau value of 0.56 for the top 5% accurate architectures, significantly higher than zero-shot NAS.\n\nSecond, Our CE-NAS integrates few-shot NAS and vanilla NAS, leveraging vanilla NAS in a low-carbon intensity period to provide accurate performance feedback, effectively guiding the search algorithms and identifying promising regions in the search space. Our CE-NAS effectively guides the exploration of optimal neural architectures with minimal carbon emissions. Zero-shot NAS, due to its high randomness and inaccurate performance proxies, struggles to find satisfactory neural architectures. Additional evidence from [15] shows that the top-1 accuracy of the best zero-shot NAS method [27, 28] on the ProxylessNAS search space is 73.63%, only slightly better than random search sorted by the highest FLOPs (73.08%). In comparison, one-shot-based ProxylessNAS[31] achieves 75.1% top-1 accuracy, and few-shot ProxylessNAS further improves this to 75.91%[29]. Our CE-NAS achieves a top-1 accuracy higher than 80% on ImageNet.\n\nIn summary, we acknowledge that zero-shot NAS can be very cost-effective and energy/carbon-efficient. However, relying solely on training-free proxies to evaluate neural architectures can be extremely inaccurate, compared to one/few-shot NAS. Nevertheless, we note that CE-NAS is complementary to zero-shot NAS. Specifically, CE-NAS can integrate zero-shot NAS methods for quick evaluations, replacing one/few-shot NAS. The vanilla NAS component is a valuable complement that helps zero-shot NAS search in promising regions, thereby improving the search effectiveness of zero-shot NAS.\n\n---\n**The integration of multiple components introduces complexity in implementation.** \n\nWe appreciate reviewer g77N’s attention to this matter and would like to highlight that we have conducted many ablation studies of CE-NAS components in the appendix (Sec. F). These ablation studies provide strong evidence that these components work well together and are indispensable to the framework. Additionally, compared to the entire NAS overhead, the carbon/energy cost of the remaining components, such as the carbon intensity predictor and reinforcement learner, is negligible. Lastly, we have included the source code as supplementary material,  and we invite the reviewer to inspect our research prototype, which we will open-source upon acceptance. \n\n---\n**Reference**\n\nDue to a limit of 6000 characters, the reference can be found in **Common Rebuttal for All Reviewers**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KgaHnzzhTG", "reviewer": "Reviewer_g77N", "summary": "This paper introduces a NAS framework, named CE-NAS, which focuses on reducing the carbon footprint of architecture search. To do this, CE-NAS decides when to use which NAS evaluation method depending the current carbon intensity, and how to allocate available GPU resources. Their carbon forecasting model also outperforms previous models for predicting carbon intensity. In experiments they show that CE-NAS outperforms other methods, like vanilla NAS and one-shot NAS, given a fixed carbon budget.\n\nSampling and evaluation are decoupled in the framework. Sampling is done using a BO algorithm which generates new architectures. These are then added to a queue of architectures that will be put on the available GPU resources for full training. A proxy evaluation is also done before this using the supernet constructed from the most promising region of the search space, according to a multi-objective optimiser. This approximate performance is used by the BO sampling algorithm.\n\nDuring periods of high carbon emissions, the sampling is the priority, and in periods of low carbon emissions, the full training is prioritised. In the latter, it also updates the promising search space region used to construct the supernet. Every hour, the priority is reassigned and GPU resources are reallocated.", "questions": "How easy is it in practice to access live information about carbon efficiency? This is key to the method and reducing carbon emissions from the NAS process, but as a practitioner I would not know how to get this information. If there is a straightforward way to get this data from, e.g. ElectricityMap, I would like to see that described in the paper or pointed to in the appendix. Additionally, it would be good to have a discussion on the availability of carbon emissions data across different countries/regions as I imagine it will be unavailable in many places.", "limitations": "There is no clear section that discusses limitations, and I think this is where my suggestion on discussing the availability of carbon emissions data could be put.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "**Originality**\nConstructing a framework which puts carbon efficiency at its core is an original idea that goes further than previous work. The paper also motivates this idea very well.\n\n**Quality**\nThe proposed solution is effective at providing models on the Pareto frontier of the optimised objectives. It consistently returns architectures that have either better performance or lower cost/parameter count and these choices are produced in the same search, allowing the user to choose between them at test time. Additionally, the carbon cost of the framework lies between vanilla and one-shot methods while being able to outperform both in terms of accuracy.\n\nOne of my concerns was that this paper presents a complex framework with many moving parts, including several trained networks. Fortunately there are good ablations of these components in the appendix, accounting for their contribution to the whole.\n\n**Clarity**\nThe paper is clear and well presented overall. There are some minor issues on descriptive clarity which I have listed in the weaknesses section below.\n\n**Significance**\nI think the contribution is significant, but have some minor concerns whether the scope of this paper fits best within the NeurIPS conference. Would this paper that focuses on a complex and practical framework perhaps be better suited to a venue that focused on the practical aspects on running NAS, such as the AutoML conference? On the other hand I think the paper tackles important questions for the NAS community which should be highlighted at a venue that gives it more impact. This concern does not overall impact my rating but I would like to hear the authors thoughts on it.", "weaknesses": "My main concern is that the paper does not make it clear enough how a practitioner would use the framework themselves. The critical point is that real-time carbon emission data needs to be available locally and there is no discussion of how realistic this is or any suggestion for where to find it. Without this aspect of carbon efficiency, there is still a decent contribution of a framework that seems to outperform other methods for multi-objective NAS but I would like more discussion on how I could realistically use the full framework myself.\n\nMinor comments:\n* Figure 1 needs a more descriptive caption that clarifies the diagram and how CE-NAS works on a high level. The diagram is complex and hard to parse at the moment.\n* The CIFAR10 performance differs between the abstract (97.35%) and the introduction (97.84%). In the end of the introduction, in the list of contributions, it is again 97.35%. This seems to be the two different models from the Pareto frontier. I suggest either explaining that you find both or be consistent with which you highlight in the text.\n* Is there a citation for the claim on lines 97-99? *A recent NAS work on transformer revealed that their comprehensive architecture search used 979 million training steps, totaling 274,120 GPU hours and resulting in 626,155 pounds of CO2 emissions.*\n* When reading, I was confused by both the relative carbon emission and hypervolume metrics. I think there should be a short explanation in the main paper along with the pointer to the full description in the appendix. This pointer is currently in section 4.2.1 but hypervolume is first mentioned in 3.5.1 which causes confusion for a while.\n* The caption for Figure 2 states that CE-NAS has the lowest relative carbon emission but this seems to be wrong. One-shot LaMOO has a lower CO_2 number in (a) and both One-shot LaMOO and Heuristic have lower numbers in (b).\n* On line 263: *“To date, there are three popular open-source NAS datasets, NasBench101 [77], HW-NAS-BENCH [25], and NasBench301 [80]”* There are other popular ones, e.g. DARTS, and I think a more appropriate term here is NAS benchmarks instead of datasets.\n* Table 2, row CE-Net-P1 says hybird instead of hybrid.\n* In Table 3, MnasNet achieves a TensorRT Latency of 0.53, which is lower than CE-Net-G1 and should be the bolded number (unless there was a typo).\n* Figure 9 of the appendix needs a caption that better explains it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for your insightful comments, please check our answers below.\n\n---\n**How easy is it in practice to access live information about carbon efficiency?**\n\nThere are third-party providers, such as ElectricityMap and WattTime, that allow users to query carbon intensity through APIs [APIs1] [APIs2]. As sustainability becomes an increasingly important issue in computing, we anticipate that more resources for obtaining carbon information will become available. We will include additional details about the accessibility of carbon information in the final paper.\n\n[APIs1] https://static.electricitymaps.com/api/docs/index.html\n\n[APIs2] https://docs.watttime.org/#tag/GET-Regions-and-Maps/operation/get_reg_loc_v3_region_from_loc_get\n\n---\n**Discussion on the availability of carbon emissions data across different countries/regions**\n\nYou are correct that carbon data signals are only available in certain parts of the world. For example, there is a lack of such information for Africa and large parts of Asia [CarbonAvailability]. However, we believe there will be a growing range of grid regions as commercial services flourish.\n\nCurrently, in our work, CE-NAS leverages the temporal variation of carbon signals to strategically allocate GPU resources for NAS methods. This means that CE-NAS can be used on GPU clusters that operate in regions with available carbon emission data. Users from regions that currently do not have these carbon signals can still contribute to sustainability goals by launching CE-NAS on remote GPU clusters.\n\n[CarbonAvailability] https://watttime.org/docs-dev/coverage-map/\n\n---\n**Make Figure 1 better**\n\nWe will simplify this figure and write a descriptive caption based on your suggestion.\n\n---\n**Inconsistency in the abstract and the introduction**\n\nIn abstract, we want to highlight we searched small model still has high accuracy. In introduction, we want to emphasis that we can find state-of-the-art models in terms of accuracy. Sorry for making this confusing inconsistency in the different context of the paper. We will make them consistent in our paper. \n\n---\n**Citation for the claim on lines 97-99**\n\nIt shares the same citation ([60] in the paper) with the last sentence. We will add this citation to the paper in this sentence.\n\n---\n**Definition of carbon emission and hypervolume metrics**\n\nThanks for catching this. We will move he definition of relative carbon emission and hypervolume metrics to a more appropriate place in the paper.\n\n---\n**Inaccurate caption statement in Figure 2**\n\nThanks for catching this. We will fix it in our final version. \n\n---\n**Appropriate term for NasBench**\n\nWe will rephrase the dataset to benchmarks in our final version.\n\n---\n**MnasNet should be bolded in Table 3**\n\nWe will bold it as the lowest TensorRT Latency in the table. \n\n---\n\n**Caption of Figure 9**\n\nFigure 9 describes the search efficiency among different NAS methods under time constraints. The total carbon cost of each method is shown above each method. We will also add more details in the figure caption.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "raWcNMz5w9", "reviewer": "Reviewer_d3FP", "summary": "Neural architecture search (NAS) is known to be extremely compute intensive, which also increases the corresponding energy consumption and the carbon costs due to energy production. In this work, the authors attempt to reduce the carbon cost of performing NAS. To this end, they propose a reinforcement learning (RL) agent that switches between energy intensive evaluation of architectures (standard NAS) and energy-efficient sampling (few-shot NAS), during high carbon intensive and low carbon intensity regimes, respectively. To predict the carbon intensity for efficiency scheduling of the two NAS regimes, this work uses a time series transformer which predicts the carbon intensity of energy production using historical information. Using a combination of RL agent for scheduling based on the time series transformer based carbon intensity prediction, the work shows considerable reduction in the overall carbon costs of NAS while still achieving architectures that are comparable to SOTA.", "questions": "See weaknesses.", "limitations": "The authors do no discuss limitations of the work. I would encourage them to discuss the effects of the carbon intensity predictor. It could be done at two steps: Firstly, dependent on the accuracy. What if the predicted carbon intensity is inaccurate? What is the impact of scheduling the energy intensive step during carbon intensive times? \n\nSecond, at a higher level. Let us assume everyone starts doing carbon efficient NAS (or even computations in general) then there is a possible risk of rebound effect which might then go on to increase the carbon footprint paradoxically.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Carbon costs of developing deep learning models is not sufficiently addressed in the literature. This work takes up a carbon efficient approach to NAS, and presents a clear motivation for looking at costs such as energy consumption and carbon emissions. \n\n* The proposed end-to-end carbon efficient NAS that performs considerably well across benchmark and open domain tasks. \n\n* The strategy of balancing between energy efficient sampling of architecture spaces (using few-shot NAS like approaches) and more energy intensive evaluations to explore search space better, during low and high carbon intensity times, respectively, is a strong contribution of this work. That a trade-off between these strategies is formulated using as a multi-objective optimization that aims to optimize for carbon costs is an elegant formulation of NAS than only one-shot or vanilla NAS.\n\n* The time series predictor, although is a straightforward use of an existing transformer model, achieves good predictive performance on carbon intensity predictions.\n\n* Experimental evaluation is sufficiently convincing. The use of real carbon intensity data from electricitymaps also provides useful insights into the real-world applications of this method.", "weaknesses": "* **Spatial carbon intensity variations**: This work mainly addresses the temporal variations of carbon intensity. While it can be argued that it is the first step, and also an easier mode of control, it would also be interesting to see how the scheduling can take different geographical locations into account. We can assume that large scale DL compute happens on cloud instances, and this could improve the broader usefulness of this proposed method. For instance, could the scheduler take the carbon intensity of the three regions considered in this work, and further optimize the carbon costs, as is commonly done in carbon-aware scheduling literature [4]? \n\n* **NasBench101**: In L.256 authors state that NasBench101 does not provide accuracy. I am not sure what they mean by this. NasBench101 does provide training and validation accuracy for all the 423k architectures considered in the search space. Clarification can be useful.\n\n* **Beyond CNNs**: This is a general critique of most NAS works, as we are exploring only CNN architectures that are by now very efficient. Why did the authors not consider other open domains? For instance, something that takes a hybrid space of CNNs and transformers [1] into account? \n\n* **Dominance number o(a)**: If _a_ is a unique architecture (L 175), what are the _number of samples that dominate_ a given architecture _a_ (L 178)? Or is _a_ a proxy for a sub-space that is being explored? This requires some clarification. And also, on where exactly is the dominance number being used? Is it within the MOO algorithms?  \n\n#### Other comments\n* L 47: Authors seem to allude that low latency is always corresponding to low energy/carbon costs. This is not always the case as has been shown in many works [5,3] \n* Authors are missing some key references pertaining to carbon-aware scheduling [4] and energy/carbon efficient NAS [2,3]. While [2,3] are not exactly efficient NAS they do grapple with some of the concerns expressed in this work. These suggestions for missing reviews are only samples and not comprehensive; I encourage the authors to explore further starting from these papers.\n\n#### References\n\n[1] Xu, Shuying, and Hongyan Quan. \"ECT-NAS: Searching efficient CNN-transformers architecture for medical image segmentation.\" In 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 1601-1604. IEEE, 2021.\n\n[2] Dou, S., Jiang, X., Zhao, C. R., & Li, D. (2023). EA-HAS-bench: Energy-aware hyperparameter and architecture search benchmark. In The Eleventh International Conference on Learning Representations.\n\n[3] Bakhtiarifard, P., Igel, C., & Selvan, R. (2024, April). EC-NAS: Energy consumption aware tabular benchmarks for neural architecture search. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 5660-5664). IEEE.\n\n[4] Radovanović, A., Koningstein, R., Schneider, I., Chen, B., Duarte, A., Roy, B., ... & Cirne, W. (2022). Carbon-aware computing for datacenters. IEEE Transactions on Power Systems, 38(2), 1270-1280. \n\n[5] Henderson, P., Hu, J., Romoff, J., Brunskill, E., Jurafsky, D., & Pineau, J. (2020). Towards the systematic reporting of the energy and carbon footprints of machine learning. Journal of Machine Learning Research, 21(248), 1-43.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer for a thorough review and valuable comments. Below are our responses to the reviewer's concerns.\n\n---\n**Spatial carbon intensity variations**\n\nThank you for the suggestion. Considering the spatial variation of carbon intensity is an intriguing idea, and it is indeed feasible to further extend our scheduler to include such information in its strategy. For example, inspired by the EuroSys paper[1], the most straightforward approach is for the scheduler to first pick the data center with the lowest carbon intensity and subsequently leverage temporal variation to optimize GPU allocation.\n\nHowever, to further enhance scheduler performance, such as enabling dynamic switching between different data centers during the training process, we anticipate a key challenge will be managing the carbon costs associated with spatial shifts. Transferring large volumes of data over the Internet incurs significant carbon costs, and estimating these costs can be complex. The data can traverse a random number of intermediate routers across a large geographic region, making it difficult to ensure that the benefits of performing a spatial shift outweigh its overhead. We will add this discussion to the final paper and cite related work in carbon-aware scheduling.\n\n---\n**NasBench101**\n\nWe apologize for any confusion caused. What we meant is that NasBench101 does not provide accuracy for all architectures in the defined search space.  While NasBench101 does provide accuracy for 423k architectures, **the actual search space is much larger**. This means that if we use the search space defined in NasBench101, we may encounter architectures that are not included in the 423k provided by NasBench101 and therefore lack accuracy information.\n\n---\n**Beyond CNN**\n\nWe agree that exploring transformer-based architecture search space is a very important and timely research topic. As mentioned in our paper, our transformer-based carbon intensity predictor is designed by the LaMOO NAS algorithm,  running inside our CE-NAS framework (Sec. 3.5.2 and Sec. D). The NAS-designed transformer-based predictor outperformed all existing state-of-the-art baselines, as shown in Table 1 of the paper.\n\nWe greatly appreciate the reviewer pointing out the good hybrid search space of CNN and transformer[32]. However, the search space described in this paper is not yet open-sourced, which makes it hard for us to investigate the hybrid search space of CNN and transformer. Despite it being a promising future direction. Moreover, for this work, due to our limited computing resources, we are unable to conduct large open-domain tasks for searching transformer-based architectures. The existing NAS benchmarks[33] are all based on CNN search spaces, so we currently lack transformer-based benchmarks to validate our CE-NAS. In future work, it will be promising to implement CE-NAS into transformer-based search spaces, such as for large language model fine-tuning, adapter (LoRA) design, and more.\n\n---\n**Dominance number o($\\alpha$)** \n\nSorry for causing the misunderstanding. $\\alpha$ refers to a unique architecture. In this paper/context, a sample means a neural architecture. To clarify, I would like to provide a toy example: we aim to search for good architectures with higher accuracy and lower inference latency. Consider we have sampled 5 architectures in the search space; one of them, referred to as $\\alpha$, has a dominance number O($\\alpha$). The O($\\alpha$) can be calculated using the remaining four sampled architectures. Specifically, if an architecture has a higher accuracy than $\\alpha$ and also a lower inference latency than $\\alpha$, that means this architecture dominates $\\alpha$. The number of architectures among these 5 that dominate $\\alpha$ is the dominance number of $\\alpha$ (O($\\alpha$)). As such, an architecture with a smaller dominance number is more promising than an architecture with a higher dominance number. Now, back to CE-NAS, we use the dominance number to determine the order for sampling and evaluation. Specifically, during the sampling phase, we will sample many architectures and sort them based on their dominance number.  When CE-NAS executes the evaluation component, the evaluation order of architectures is based on the dominance number; i.e., the architecture with the lowest dominance number has the highest priority to be evaluated.\n\n---\n**L47**\n\nOur intention was to convey that we are not only focusing on performance metrics like accuracy. We can consider more metrics such as inference latency, #PARAMs, #FLOPs, etc. We will rewrite this part to clarify our meaning and remove any misleading information.\n\n---\n**More related works**\n\nThanks for providing useful resources. We carefully read the suggested papers and further explore starting from these sources. We have organized the papers into three categories as detailed below:\n\n+ Category one[2,3,4,5,6]: This category focuses on measuring carbon emissions and the impact of carbon intensity on AI training or NAS tasks but does not include any designs to reduce carbon emissions.\n\n+ Category two[7,8,9]: This category proposes energy-aware scheduling for NAS tasks but omits the effect of carbon intensity variations on carbon costs.\n\n+ Category three[10,11,12,13,14]: This category considers the effect of either temporal or spatial carbon intensity variations on the carbon emissions of data center tasks, relying on inter-job execution order scheduling and location selection to reduce total emissions. However, the scheduling techniques proposed by these works to exploit temporal carbon intensity variation do not apply when optimizing for a single NAS task.\n\nWe will include them in the final paper. \n\n---\n**Limitations**\n\nThank you for your suggestion. We will add the discussion of the carbon intensity predictor in the final paper. \n\n---\n**Reference**\n\nDue to the characters limit, the references can be found in **Common Rebuttal for All Reviewers**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jKp6YIqofk", "reviewer": "Reviewer_QYii", "summary": "The paper proposes a carbon-efficient neural architecture search (NAS), CE-NAS, to reduce the carbon emitted during the NAS process by dynamically allocate GPU resources based on the predicted future carbon intensity. CE-NAS leverages reinforcement learning for generating the probability of choosing which model of NAS (one-shot or vanilla) based on the predicted carbon intensity by a time-series transformer. One-shot model would be chosen to run in high carbon intensity periods and vanilla NAS would be chosen otherwise. Alongside emitting less carbon, CE-NAS achieves a competitive accuracy against state-of-the art models for CIFAR-10 and ImageNet datasets", "questions": "1. How would the CE-NAS perform if we have a deadline for used GPU hours? e.g., we can only use 2k hours of GPU and we can't exceed that amount.\n2. How robust is CE-NAS in regions that we have high carbon intensity prediction errors?", "limitations": "1. The authors can discuss the scalability limitations of their model when the search space becomes more complex", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. Focusing on reducing the carbon emitted by ML tasks that has great environmental impact.\n2. Designed a high-accuracy carbon intensity prediction model compared to previous models.\n3. Using various baseline models in the experiments.\n4. Leveraging real data for the experiments makes it more applicable to real-world scenarios.", "weaknesses": "1. The experiments are only conducted in California, it would be good if authors show their results for multiple locations.\n2. The baseline models for comparing the proposed time-series carbon intensity predictor could use one more model proposed by Zhang et al. (Title: A GNN-based Day Ahead Carbon Intensity Forecasting Model for Cross-Border Power Grids) which is also a strong model for prediction.\n3. The overhead of switching between the models has not been considered considering carbon intensity variations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the positive comments and insightful suggestions from the reviewer. Below are our answers to the reviewer's questions. \n\n---\n**Experiments are California only**\n\nWe will demonstrate more results for different locations in our future work. \n\n---\n**Compared to the work proposed by Zhang et al [1]**\n\nThanks for providing this useful related work. We will add this baseline in the final version of our paper. \n\n---\n**The overhead of switching between the models has not been considered considering carbon intensity variations.**\n\nCompared to the entire NAS overhead, the carbon/energy cost of the remaining components, such as the carbon intensity predictor and reinforcement learner, is negligible. \n\n---\n**How would the CE-NAS perform if we have a deadline for used GPU hours?**\n\nWe provided an ablation study based on the time constraint in the appendix (Section F2 and Figure 9). We will move this result to the main paper if we have more space in the final version.\n\n---\n**How robust is CE-NAS in regions that we have high carbon intensity prediction errors?**\n\nIf the carbon intensity prediction is incorrect, we might allocate GPU resources inappropriately among vanilla and one-shot NAS methods. This could impact the accuracy of the searched architectures and the carbon emissions. However, the total carbon emissions should still be less than directly using vanilla NAS, and the searched architectures should still be better than directly using one-shot NAS. This is partly because of our novel use of LAMOO to refine the search space.\n\nFurthermore, we anticipate that our RL-based GPU allocation strategy will exhibit tolerance towards carbon intensity prediction errors. The adverse impact of these prediction errors will be perceived by the RL agent through its continuously updated input states, enabling it to adjust its decision-making accordingly. For instance, if the predictor underestimates carbon emission intensity, the agent may initially adopt aggressive decisions to schedule carbon-intensive tasks. This results in a rapid decrease in the remaining carbon budget, which is reflected in the input states. The RL agent perceives this change and switches to more conservative decisions in the subsequent process.\n\n---\n**Reference**\n\n[1] Zhang, Xiaoyang, and Dan Wang. \"A GNN-based Day Ahead Carbon Intensity Forecasting Model for Cross-Border Power Grids.\" Proceedings of the 14th ACM International Conference on Future Energy Systems. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Reference**

[1] On the Limitations of Carbon-Aware Temporal and Spatial Workload Shifting in the Cloud, Sukprasert et al. EuroSys 2024. 

[2] Anthony L F W, Kanding B, Selvan R. "Carbontracker: Tracking and predicting the carbon footprint of training deep learning models". arXiv preprint arXiv:2007.03051, 2020.

[3] Selvan R, Bhagwat N, Wolff Anthony L F, et al. "Carbon footprint of selecting and training deep learning models for medical image analysis". International Conference on Medical Image Computing and Computer-Assisted Intervention, 2022.

[4] Naidu R, Diddee H, Mulay A, et al. "Towards quantifying the carbon emissions of differentially private machine learning". arXiv preprint arXiv:2107.06946, 2021.

[5] Lacoste A, Luccioni A, Schmidt V, et al. "Quantifying the carbon emissions of machine learning". arXiv preprint arXiv:1910.09700, 2019.

[6] Dodge J, Prewitt T, Tachet des Combes R, et al. "Measuring the carbon intensity of ai in cloud instances". Proceedings of the 2022 ACM conference on fairness, accountability, and transparency, 2022.

[7] Frey N C, Zhao D, Axelrod S, et al. "Energy-aware neural architecture selection and hyperparameter optimization". IPDPSW, 2022.

[8] Dou S, Jiang X, Zhao C R, et al. "EA-HAS-bench: Energy-aware hyperparameter and architecture search benchmark". The Eleventh ICLR. 2023.

[9] Bakhtiarifard P, Igel C, Selvan R. "EC-NAS: Energy consumption aware tabular benchmarks for neural architecture search". ICASSP, 2024.

[10] Radovanović A, Koningstein R, Schneider I, et al. "Carbon-aware computing for datacenters". IEEE Transactions on Power Systems, 2022

[11] Souza A, Jasoria S, Chakrabarty B, et al. "CASPER: Carbon-Aware Scheduling and Provisioning for Distributed Web Services". Proceedings of the 14th International Green and Sustainable Computing Conference, 2023.

[12] Wiesner P, Behnke I, Scheinert D, et al. "Let's wait awhile: How temporal workload shifting can reduce carbon emissions in the cloud." Proceedings of the 22nd International Middleware Conference. 2021

[13] Bahreini T, Tantawi A, Youssef A. "A Carbon-aware Workload Dispatcher in Cloud Computing Systems." 16th CLOUD, 2023.

[14] Tripathi S, Kumar P, Gupta P, et al. "Workload Shifting Based on Low Carbon Intensity Periods: A Framework for Reducing Carbon Emissions in Cloud Computing." BigData, 2023.

[15] Li, Guihong, et al. "Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunities." IEEE Transactions on Pattern Analysis and Machine Intelligence (2024).

[16] Meng-Ting Wu, Chun-Wei Tsai, Training-free neural architecture search: A review, ICT Express, Volume 10, Issue 1, 2024, Pages 213-231, ISSN 2405-9595,

[17] M. S. Abdelfattah, A. Mehrotra, Ł. Dudziak, and N. D. Lane, “Zerocost proxies for lightweight nas,” in International Conference on Learning Representations, 2021.

[18] N. Lee, T. Ajanthan, and P. Torr, “SNIP: SINGLE-SHOT NETWORK PRUNING BASED ON CONNECTION SENSITIVITY,” in International Conference on Learning Representations, 2019.

[19] H. Tanaka, D. Kunin, D. L. Yamins, and S. Ganguli, “Pruning neural networks without any data by iteratively conserving synaptic flow,” in Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 6377–6389.

[20] C. Wang, G. Zhang, and R. B. Grosse, “Picking winning tickets before training by preserving gradient flow,” in International Conference on Learning Representations. OpenReview.net.

[21] Z. Zhang and Z. Jia, “Gradsign: Model performance inference with theoretical insights,” in International Conference on Learning Representations, 2022.

[22] L. Theis, I. Korshunova, A. Tejani, and F. Huszar, “Faster gaze ” prediction with dense networks and fisher pruning,” CoRR, vol. abs/1801.05787, 2018.

[23] V. Lopes, S. Alirezazadeh, and L. A. Alexandre, “Epe-nas: Efficient performance estimation without training for neural architecture search,” in International Conference on Artificial Neural Networks. Springer, 2021, pp. 552–563.

[24] J. Mellor, J. Turner, A. Storkey, and E. J. Crowley, “Neural architecture search without training,” in International Conference on Machine Learning. PMLR, 2021, pp. 7588–7598.

[25] M. Lin, P. Wang, Z. Sun, H. Chen, X. Sun, Q. Qian, H. Li, and R. Jin, “Zen-nas: A zero-shot nas for high-performance image recognition,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 347–356.

[26] Z. Sun, M. Lin, X. Sun, Z. Tan, H. Li, and R. Jin, “MAE-DET: revisiting maximum entropy principle in zero-shot NAS for efficient object detection,” in International Conference on Machine Learning, ICML 2022, 17-23, Proceedings of Machine Learning Research, vol. 162. PMLR, 2022.

[27] W. Chen, X. Gong, and Z. Wang, “Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective,” in International Conference on Learning Representations, 2021.

[28] W. Chen, W. Huang, X. Du, X. Song, Z. Wang, and D. Zhou, “Autoscaling vision transformers without training,” in International Conference on Learning Representations, 2022.

[29] Zhao, Yiyang, et al. "Few-shot neural architecture search." International Conference on Machine Learning. PMLR, 2021.

[30] Su, Xiu, et al. "K-shot nas: Learnable weight-sharing for nas with k-shot supernets." International Conference on Machine Learning. PMLR, 2021.

[31] Cai, H., Zhu, L., and Han, S. ProxylessNAS: Direct neural architecture search on target task and hardware. In International Conference on Learning Representations, 2019.

[32]  Xu, Shuying, and Hongyan Quan. "ECT-NAS: Searching efficient CNN-transformers architecture for medical image segmentation." In 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 1601-1604. IEEE, 2021.

[33] Chitty-Venkata, Krishna Teja, et al. "Neural architecture search benchmarks: Insights and survey." IEEE Access 11 (2023): 25217-25236.

Meta-review: This paper introduces CE-NAS, a NAS (Neural Architecture Search) framework designed to minimize the carbon footprint of architecture searches. CE-NAS optimizes the use of GPU resources and NAS evaluation methods based on real-time carbon intensity forecasts, which their model predicts more accurately than existing methods. All the reviewers are satisfied with the authors' response. The authors should consider including a discussion on zero-shot NAS in the related work section to ensure the completeness of the paper.
Decision: Accept (poster)