CE-NAS: An End-to-End Carbon-Efficient Neural Architecture Search Framework

Authors: yiyang  zhao
yunzhuo  liu
bo  jiang
tian  guo
#  new
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 05:41:12

Abstract: This work presents a novel approach to neural architecture search (NAS) that aims to increase carbon efficiency for the model design process. The proposed framework CE-NAS addresses the key challenge of high carbon cost associated with NAS by exploring the carbon emission variations of energy and energy differences of different NAS algorithms. At the high level, CE-NAS leverages a reinforcement-learning agent to dynamically adjust GPU resources based on carbon intensity, predicted by a time-series transformer, to balance energy-efficient sampling and energy-intensive evaluation tasks. Furthermore, CE-NAS leverages a recently proposed multi-objective optimizer to effectively reduce the NAS search space. We demonstrate the efficacy of CE-NAS in lowering carbon emissions while achieving SOTA results for both NAS datasets and open-domain NAS tasks. For example, on the HW-NasBench dataset, CE-NAS reduces carbon emissions by up to 7.22X while maintaining a search efficiency comparable to vanilla NAS. For open-domain NAS tasks, CE-NAS achieves SOTA results with 97.35% top-1 accuracy on CIFAR-10 with only 1.68M parameters and a carbon consumption of 38.53 lbs of CO2. On ImageNet, our searched model achieves 80.6% top-1 accuracy with a 0.78 ms TensorRT latency using FP16 on NVIDIA V100, consuming only 909.86 lbs of CO2, making it comparable to other one-shot-based NAS baselines. Our code is available at https://github.com/cake-lab/CE-NAS.

Main paper: 
 introduction: 
 Deep Learning (DL) has become an increasingly important field in computer science, with wide applications such as healthcare and finance [81, 6, 60, 30, 33, 63, 35, 38] . Neural architecture search (NAS) has emerged as a means to automate the design of DL models, which involves training many DL models from a massive architecture design space with hundreds of millions to trillions of candidates [102, 66, 79, 97, 98, 46] . Consequently, NAS can be energy-intensive and significantly contributes to today's carbon emissions [68, 102] . Many NAS works have reported using thousands of GPU-hours [102, 66, 79, 78, 98] . For example, Zoph et al. [102] used 800 GPUs for 28 days, resulting in 22,400 GPU-hours, to obtain the final architectures. Strubell et al. [68] found that a single NAS solution can emit as much carbon as five cars during its lifetime.The environmental impact of NAS, if left untamed, can be substantial. While recent works have significantly improved the search efficiency of NAS [102, 66, 79, 78, 97] , e.g., reducing the GPUhours to tens of hours without sacrificing the architecture quality, there still lacks conscious efforts in reducing carbon emissions. As noted in a recent vision paper by Bashir et al. [9] , energy efficiency can help reduce carbon emissions but is not equivalent to carbon efficiency. This paper aims to Figure 1 : An overview of CE-NAS. The sampling and evaluation tasks will be allocated different GPU resources based on carbon emission intensity during the neural architecture search. Specifically, when the CO 2 intensity is low, we allocate more resources to energy-intensive evaluation, which focuses on assessing the performance of sampled architectures generated through energy-efficient sampling. Conversely, when CO 2 intensity is high, more GPUs are dedicated to energy-efficient sampling, which utilizes various NAS algorithms to sample new architectures from a search space partitioned based on previously evaluated architectures. The specific GPU allocation strategy is learned through a reinforcement learning agent, as detailed in the middle section of the figure . bridge the gap between carbon and energy efficiency with a new NAS framework designed to be carbon-aware from the outset.The proposed framework, termed CE-NAS, tackles the high carbon emission problem from two main aspects. First, CE-NAS regulates the model design process by deciding when to use different NAS evaluation strategies based on the carbon intensity, which varies geographically and temporally with the mix of active generators as observed in [9] . To elaborate, given a fixed amount of GPU resources, CE-NAS will allocate more resources to energy-efficient NAS evaluation strategies, e.g., one-shot NAS [65, 10, 46, 97, 14, 13] , during periods of high carbon intensity. Conversely, during periods of low carbon intensity, CE-NAS will shift the focus to running energy-intensive but more effective NAS evaluation strategies, e.g., vanilla NAS [102, 66, 79, 78] . We design and train a reinforcement learning (RL) agent ( §3.5) based on historical and predicted carbon emissions ( §3.5.2) and observed search results to make such allocation decisions. Second, the CE-NAS framework will support searching for DL models satisfying multiple metrics beyond the commonly used metric of inference accuracy. CE-NAS can consider more metrics such as inference latency, #PARAMs, #FLOPs, etc. For example, CE-NAS can search for models with high accuracy and low inference latency. Specifically, both the search and deployment efficiency are achieved by integrating a recent learning-based multi-objective optimizer LaMOO [98] to CE-NAS. Figure 1 depicts the overall workflow of CE-NAS.To demonstrate CE-NAS's efficacy in addressing the energy and carbon issues, we conduct a comprehensive evaluation using both commonly used NAS benchmarks and real-world NAS tasks. For example, utilizing carbon emission data from ElectricityMap [58] , CE-NAS achieves better performance with reduced carbon costs compared to existing methods on various NAS benchmarks, including HW-NASBench [40] and NasBench301 [93] . In open-domain tasks, CE-NAS achieves the state-of-the-art (SOTA) top-1 accuracy of 97.35% with only 1.68M parameters on the CIFAR-10 image classification task, with CO 2 costs similar to those of one-shot-based NAS algorithms. For the ImageNet dataset, CE-NAS achieves the SOTA top-1 accuracy of 80.6% under the same inference latency of 0.78 ms with TensorRT on NVIDIA V100, while maintaining a carbon cost comparable to SOTA NAS methods [13] . We also evaluate our carbon forecasting model and show that it outperforms existing models [11, 56, 55] , including the SOTA method [55] .In summary, we make the following key contributions.• We introduce a carbon-efficient NAS framework named CE-NAS that can dynamically allocate GPU resources among different NAS evaluation methods-namely, vanilla NAS and one/few-shot NAS-based on carbon emissions data and current search results. CE-NAS is an end-to-end framework that synergistically integrates an RL-based agent for GPU resource allocation, a time-series transformer [2] for carbon intensity prediction, and a multi-objective optimizer [98] for reduced search space.• We implement and evaluate CE-NAS on different NAS benchmarks, including HW-NasBench [40] and Nasbench301 [93] . We show that CE-NAS can achieve the best search performance compared to only using vanilla NAS, one-shot NAS, and a recently proposed heuristic GPU allocation strategy [4] given the same carbon budget.• CE-NAS delivers SOTA architectural performance in open-domain NAS tasks while maintaining carbon costs comparable to one-shot NAS methods. For example, on CIFAR-10, CE-NAS achieves a top-1 accuracy of 97.35% with only 1.68M parameters and a carbon cost of 38.53 lbs CO 2 . On ImageNet, CE-NAS reaches a top-1 accuracy of 80.6% with a 0.78 ms TensorRT latency using FP16 on NVIDIA V100. 
related work: 
 Efficient neural architecture search (NAS) often focuses on improving the evaluation phase, e.g., via weight-sharing (one-shot) [65, 46, 88, 18, 13] , zero-shot proxy [41, 85, 3, 37, 73, 76, 95, 74, 50, 59, 44, 70, 16, 17] , performance predictor [45, 13, 79, 93] , low-fidelity NAS evaluation [102, 66, 79, 78, 46, 36] , and gradient proxy NAS [87] . A comparison of these methods can be found in Table 4 in Appendix. Weight-sharing leverages the accuracy estimated by a supernet as a proxy for the true architecture performance, while gradient proxy NAS uses the gradient as a proxy. These proxy-based methods, although incurring smaller search costs in terms of energy, can have lower search efficiency because their estimated architecture accuracy may have poor rank correlation [97] . Zero-shot proxy NAS eliminates the need for supernet training entirely, serving as a fully training-free proxy for network evaluation. However, as noted in [41] , mainstream zero-shot NAS methods still struggle to achieve high rank correlations between predicted and true accuracies, even when compared to weight-sharing-based NAS evaluation. Performance predictors provide a more accurate performance prediction than weight-sharing and gradient proxy NAS. Still, their accuracy heavily relies on the volume and quality of the training data, which can be very expensive to create [90, 93] . Low-fidelity evaluation still requires training each searched architecture, leading to limited energy savings. In practice, simply employing these methods without modification might not effectively balance carbon emissions and performance during the search process. CE-NAS successfully achieves good search efficiency, search quality, and carbon efficiency by integrating the weight-sharing NAS method into vanilla NAS. We also leverage the low-fidelity evaluation method in the open-domain NAS search.The high carbon emission of NAS is a pressing issue. A study by [68] found that a single NAS solution can emit as much carbon as five cars over its lifetime. For example, if we design a transformerbased model, training a base Transformer model can use 96 GPU hours on NVIDIA's P100 [68] ,while training a larger model can take 28 GPU days. A recent NAS work [68] on transformer revealed that their comprehensive architecture search used 979 million training steps, totaling 274,120 GPU hours and resulting in 626,155 pounds of CO 2 emissions. Although several NAS studies [87, 13, 12] have attempted to address carbon issues during the search process, they often treat carbon emissions uniformly, regardless of time and location variances. These carbon-agnostic NAS methods therefore miss the opportunity to explore the inherent carbon variations [9] . Concurrently, systems researchers have explored the temporal and spatial variations exhibited in electricity's carbon emission for popular data center applications like training [4, 9] and inference [39] . This work targets a special form of training workload, i.e., NAS, by leveraging the domain knowledge to optimize the end-to-end NAS process to explore carbon variations. Specifically, we concentrate on implementing the temporal variations in carbon intensity rather than spatial variations, as managing the carbon costs associated with spatial shifts poses significant challenges. Transferring large volumes of data over the Internet can incur substantial carbon costs, and accurately estimating these costs is complex. Data often passes through numerous intermediate routers across large geographic regions, making it difficult to ensure that the benefits of a spatial shift outweigh its associated overhead.3 The Design of a Carbon-Efficient NAS FrameworkIn this section, we present an overview of the proposed CE-NAS framework ( §3.1). We first introduce the initial setup of our CE-NAS in §3.2. We then propose a sampling strategy based on the oneshot/few-shot NAS evaluation method in §3.3. Finally, we discuss the architecture evaluation part in §3.4, which is both time and computing resource-intensive. In §3.5, we describe how to use an RL-based agent to automatically allocate GPU resources based on predicted hourly carbon intensity. 
ce-nas overview: 
 As observed in [9] , grid carbon emissions vary geographically and temporally based on the mix of active generators. Consequently, systems can emit different amounts of carbon even when consuming the same electricity at different locations or times. Currently, neither one (few)-shot nor vanilla NAS methods account for these variations in carbon emissions, which can lead to inefficiencies in terms of NAS carbon usage. Furthermore, one (few)-shot and vanilla NAS exhibit different search and carbon efficiency trade-offs. Vanilla NAS is carbon-intensive but effective, while one (few)-shot NAS can be carbon-friendly but sample-inefficient.CE-NAS is a NAS framework that directly addresses the high-carbon issues by balancing energy consumption across high-carbon and low-carbon periods. The primary objective of this framework is to efficiently search for optimal neural architectures while minimizing the carbon footprint associated with the NAS process. Specifically, CE-NAS decouples the two parts of a NAS search processenergy-efficient sampling and energy-intensive evaluation-and handles them independently across different carbon periods. For each carbon period, CE-NAS uses an RL-based agent to automatically allocate GPU resources for the NAS search process to achieve good carbon efficiency.CE-NAS considers more than one search objective(e.g., accuracy, mAP, latency, #FLops, etc), so we formulate the search problem as a multi-objective optimization (MOO). Similar to other NAS and optimization problems [24, 77, 98, 66] , CE-NAS initializes the search process by randomly selecting several architectures, a, from the search space, Ω, and evaluating their accuracy, E(a), and other deployment metrics (#Params, #FLops, inference latency, inference energy, etc) T (a). Compared to the accuracy E(a), T (a) can be quickly measured without much cost. The resulting samples are then added to the observed samples set, P.We distinguish two types of methods for evaluating the accuracy of an architecture. One is actual training, which trains an architecture a from scratch until convergence and evaluates it to obtain its true accuracy, E(a). To optimize computing resources during this process, we implement the low-fidelity training strategy described in [102, 66, 46, 79, 36] . This strategy reduces computing costs by simplifying the parameters of neural architectures (e.g., #depth, #channels, #resolutions) or shortening the training process under controlled conditions. The other method is using one-shot evaluation [10, 65, 46] , which leverages a trained supernet, or a zero-shot-based NAS evaluator [95, 74, 50, 59, 44, 70, 16, 17] as a performance proxy to estimate the accuracy of the architecture, denoted as E ′ (a). Note that obtaining E ′ (a) is very cheap and carbon-efficient; however, due to the co-adaption among operations [97] , E ′ (a) is often not as accurate as E(a). We train all the sampled architectures in the initialization stage to obtain their true accuracy for further search.Multi-objective search space partition. We leverage the recently proposed multi-objective optimizer called LaMOO [98] that learns to partition the search space from observed samples to focus on promising regions likely to contain the Pareto frontier. LaMOO is an optimizer for general black-box optimization problems; we can apply it to NAS as follows.We utilize LaMOO [98] to partition the search space Ω into several subspaces, and find the optimal subspace denoted by Ω best . Next, we construct and train a supernet [10, 97] , S best , for Ω best . We then use a NAS search algorithm to identify new architectures that will be used to refine the search space. In this work, we employ the state-of-the-art multi-objective Bayesian optimization algorithm qNEHVI [25] . This algorithm will generate new architectures a n from Ω best , and estimate their approximate accuracy E ′ (a n ) using S best . At the same time, these architectures a n are added to a ready-to-train set R, consisting of candidates for further actual training as described in §3.4.We define the maximum capacity of R as Cap(R), hyperparameter in CE-NAS. Note that the architectures in R are removed once they are actual trained as described in §3.4. When the capacity is reached, i.e., when there are more architectures to train than we have resources for, the sampling process blocks until spaces free up in R. The accuracy of architectures estimated by S best will be fed back into the search algorithm as shown in Figure 1 to repeat the process described above.As mentioned in §3.2, obtaining estimated accuracy through supernet is energy-efficient because these architectures can be evaluated without the time-consuming training. Therefore, during high carbon emission periods, CE-NAS will try to perform this process to save energy and produce as little carbon as possible, as shown in the energy-efficient sampling part of Fig. 1 .If we perform the entire NAS only using the process described in §3.3, CE-NAS will be essentially performing one/few-shot NAS only within the subspace S best . Although these methods are efficient, they typically underperform compared to vanilla NAS. As Zhao et al. showed, it is possible to improve LaMOO's space partition with more observed samples through actual training [98] . This section describes the process to evolve S best during low carbon emission periods.At the high level, we will pick new architectures from the ready-to-train set R to train to convergence and use them to refine the search space partition. That is, the architecture a, with its true accuracy, E(a), will be added to the observed sample set P to help identify a more advantageous subspace, Ω best , for the architecture sampling process as described in [98] . In this work, we sort the architectures in the ready-to-train set R by their dominance number. The dominance number o(a) of an architecture a is defined as the number of samples that dominate a in search space Ω:EQUATIONwhere I[•] is the indicator function. With the decreasing of the o(a), a would be approaching the Pareto frontier; o(a) = 0 when the sample architecture a is located in the Pareto frontier. The use of dominance number allows us to rank an architecture by considering both the estimated accuracy E ′ (a) and other metrics T (a) at the same time. CE-NAS will first train the architectures with lower dominance number values when GPU resources are available. Once an architecture is trained, it is removed from R.This process is depicted in the energy-consuming evaluation component of Figure 1 . Note that this process includes actual time-consuming DL model training, which is time-consuming and highly energy-intensive, leading to significant CO 2 emissions. Hence, CE-NAS will try to prioritize this process during periods of low carbon intensity.The carbon impact of the above two processes (i.e., §3. Although the heuristic-based strategy proposed by [4] can adjust the tendency for sampling and evaluation based on carbon intensity, it is still far from making the most carbon-efficient decision. It overlooks critical influencing factors such as the varying gains in accuracy from performing sampling and evaluation at different stages of the search. These factors are specific to the NAS algorithms and are challenging to model, which complicates its inclusion in heuristic strategy design. To address this issue, we introduce a method based on reinforcement learning that automatically accounts for these diverse factors to develop allocation strategies. We propose a Reinforcement Learning (RL)-based method to automatically customize a GPU allocation strategy for specific NAS algorithms. In this paper, we choose LaMOO [98] with qNEHVI [25] .The key to applying RL lies in the design of effective state inputs, actions, and rewards according to the specific task. Our CE-NAS focuses on tailoring these elements for carbon-efficient NAS tasks, with the guiding principle of simplifying the state and action spaces to expedite efficient and rapid training. Specifically, Our RL agent, which is a simple-structured neural network with only four fully connected layers, takes the remaining carbon budget, the number of already searched architectures and their hypervolume (The definition of the hypervolume refers to Appendix E.2.4.), as well as the future carbon traces predicted by the time transformer as input. Then it outputs the probabilities of different GPU allocation ratios for architecture evaluation as action. These possible allocation ratios are discretized to boost learning efficiency. After the action is executed, CE-NAS generates a reward based on the improvement of the performance of the already searched architectures and then uses the reward to refine the policies of the agent by updating its network parameters with the actor-critic [61] policy gradient algorithm. The specifics of our RL design can be found in Appendix C and the middle part of Fig. 1 .We utilize a machine learning model to predict future carbon intensity, which serves as the input of the RL agent. We leverage a time series transformer-based model, as shown in the top left part of Fig. 1 , to forecast future carbon intensity because of their efficacy [62, 84, 99] . Specifically, we leverage the architecture design by employing a standard encoder-decoder Transformer for time series forecasting [2] . This method incorporates temporal features that act as positional encodings within the Transformer's encoder/decoder framework. Past values are input into the encoders, and future values are input into the decoders in the training stage. For instance, as described in [2] , if a time series data point corresponds to the 11th of August, the temporal feature vector would be (11, 8) , where 11 denotes the day of the month, and 8 signifies the month of the year. We leverage the SOTA NAS algorithm LaMOO [98] to design the architecture of the time series transformer. The details of search space and founded architectures are in Appendix D. 
ce-nas experimental evaluation: 
 We develop a prototype of the CE-NAS framework as outlined in Section 3. This section analyzes CE-NAS's carbon efficiency and search efficacy through trace-driven simulations and emulation. We first demonstrate the performance of our designed time-series transformer for carbon forecasting in § 4.1. We then assess CE-NAS across two distinct NAS scenarios: the first leveraging two commonly used NAS benchmarks, HW-NAS-Bench [40] and NasBench301 [93] , and the second including real-world computer vision applications, such as image classification.Our experiments utilize carbon trace data sourced from ElectricityMap [58] , an independent carbon information service. We selected the CISO2 trace beginning on July 2, 2021, due to its variable carbon intensity, which provides an opportunity to evaluate CE-NAS's performance over time and its adaptability to fluctuating carbon levels. Figure 2 illustrates the initial 80 hours of the carbon trace, demonstrating the actual data against the forecasted trace predicted by our designed time series transformer, as detailed in Section 3.5.2. Throughout the CE-NAS search phase, predicted carbon intensity is utilized for GPU allocation, while the actual carbon trace informs the calculation of carbon costs in the evaluation phase.We also conduct several ablation studies detailed in Appendix F. These studies include variations of hyperparameters in the RL agent settings, validation of LaMOO's effectiveness, and a comparison of search performance using predicted versus actual carbon traces, among others. Our findings reveal only a minimal difference in search performance between using predicted and actual carbon traces.We assess the performance of our time series forecasting transformer across three distinct regions: CISO, DE 3 , and PJM 4 . We compare our approach against other SOTA models referenced in [11, 56, 55]. The studies in [11, 56] employ a Feed-Forward Neural Network (FFNN) for their forecasts, while [55] utilizes a combination of FFNN, Convolutional Neural Network, and Long Short-Term Memory networks for multi-day carbon intensity predictions. We use the Mean Absolute Percentage Error (MAPE) to evaluate forecasting accuracy. Table 1 demonstrates a day-to-day comparison of all methods, with the most effective technique in bold. Our model demonstrates the best performance for both single-day and multi-day carbon intensity forecasting scenarios. More experimental details can be found in Appendix E.1.To date, there are three popular open-source NAS benchmarks, NasBench101 [90] , HW-NAS-BENCH [28] , and NasBench301 [93] . For our evaluation, we focus on the latter two, as the network architectures within these benchmarks cover the entirety of the search space. In contrast, the NasBench101 benchmark comprises only a limited subset of potential architectures. Evaluating using NasBench101 is challenging because we will not have access to important information, such as accuracy, during the search for the architecture not in the NasBench101 benchmark. We choose three types of baselines according to different GPU allocation strategies and NAS evaluation algorithms, including vanilla LaMOO [98] , one-shot LaMOO [98] and a heuristic GPU allocation strategy [4] , which is the SOTA result so far. We also define a carbon budget as the termination condition. Once the NAS exceeds this CO 2 budget, it will be stopped. More baseline details are in Appendix E.2.3. We evaluate our experimental results using two metrics: relative carbon emission and log hypervolume difference. Note that log hypervolume difference measures the quality of the search result and the smaller the better. Detailed definitions of these metrics can be found in Appendix E.2.4. As depicted in Figure 3 (a), as the search time progresses, vanilla LaMOO demonstrates the highest performance in terms of HV log_diff . Vanilla LaMOO's superior performance can be attributed to its approach of training all sampled architectures to obtain their true accuracy, which effectively steers the search direction. However, when considering the relative carbon emission, vanilla LaMOO consumes 3.24X-7.22X carbon compared to other approaches. This is expected because vanilla LaMOO is an energy-intensive approach and is not designed to be aware of carbon emissions.We show that CE-NAS's search efficiency is only marginally lower than that of vanilla LaMOO while having the least relative carbon emission. Note that we are plotting the HV log_diff in the Y-axis of Figure 3 (a); the actual HV values achieved by CE-NAS and Vanilla LaMOO are about 4100 and 4117, differing only by 0.034%, even though the two lines look far apart. This result also suggests that only relying on energy-efficient approaches (e.g., one-shot LaMOO in this case) is insufficient to achieve good search performance. CE-NAS only takes 15 hours to get the comparable HV results with the final results of One-shot LaMOO.Figure 4 presents a comparative analysis of CE-NAS's performance against various baselines under differing carbon budgets. It is evident that CE-NAS surpasses all baselines in search efficiency under different CO 2 budget constraints 5 . Below a consumption level of 10,000g CO 2 , Notably, the search outcomes associated with CE-NAS, as depicted in Figure 4 (a), and those of vanilla/one-shot LaMOO, as shown in Figure 4 (d), are comparable in terms of the median log hypervolume difference. This indicates that integrating CE-NAS with the NAS strategy can achieve a carbon cost saving of up to 5X compared to the conventional vanilla/one-shot NAS methodologies.On NasBench301, we use two metrics: relative carbon emission and hypervolume(HV ). Note that higher HV means better search results. Given the expansive nature of the NasBench301 search space, the maximal hypervolume remains undetermined. Consequently, rather than employing the hypervolume difference as a performance metric, we utilize the absolute hypervolume value to represent the efficacy of our search strategy. As depicted in Figure 3 (b), our findings are consistent with those from HW-NasBench. One-shot LaMOO incurs the lowest carbon footprint but yields the least impressive performance in terms of hypervolume. Conversely, Vanilla LaMOO achieves the highest HV but at a carbon cost that exceeds CE-NAS by more than 3.76 times. CE-NAS markedly surpasses the SoTA heuristic GPU allocation strategy regarding HV , while maintaining a comparable carbon expenditure. At equivalent levels of carbon cost, as illustrated in Figure 5 , CE-NAS demonstrates significantly superior search performance compared to other baselines. Figures 5(a ) and 5(d) show that architectures identified through CE-NAS not only yield better results but also do so with a carbon cost that is four times lower than that of one-shot and vanilla NAS methods. Within the NasBench301 framework, CE-NAS also substantially outperforms the heuristic GPU allocation approach for NAS.  
conclusion: 
 In this work, we described the design of a carbon-efficient NAS framework CE-NAS by leveraging the temporal variations in carbon intensity. To search for energy-efficient architectures, CE-NAS integrates a SOTA multi-objective optimizer, LaMOO [98] , with the one/few-shot and vanilla NAS algorithms. These two NAS evaluation strategies have different energy requirements, which CE-NAS leverages an RL-based agent to schedule when to use each based on average carbon intensity. CE-NAS has demonstrated very promising results across various NAS tasks. For example, on CIFAR-10, CE-NAS successfully identified an architecture that achieves 97.35% top-1 accuracy with just 1.68M parameters and emitted only 38.53 lbs of CO 2 . These compelling results suggest the efficacy and potential of CE-NAS as an effective framework in carbon-aware NAS. Additionally, on ImageNet, CE-NAS discovered state-of-the-art models achieving a top-1 accuracy of 80.6% with a TensorRT latency of 0.78 ms using FP16 on NVIDIA V100, and a top-1 accuracy of 79.0% with a latency of 0.56 ms on the same hardware. The carbon cost for this search was only 909.86 lbs. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] . Justification: Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: See [1]. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: See our supplementary material. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Multi-objective NAS Multi-objective NAS has gained popularity in the NAS community, as deep neural networks need to address not only traditional metrics like accuracy but also practical efficiency metrics. There are two main approaches within Multi-objective NAS. The first category [83, 71, 13, 23, 27] leverages linear scalarization of various metrics (e.g., accuracy #F LOP s ) into a single metric, followed by the application of standard single-objective NAS algorithms for architecture search. The second category [98, 53, 52] approaches MONAS as a multi-objective black-box optimization problem. According to existing results [98, 52] , this latter approach generally outperforms the former in terms of effectiveness and performance. For multi-objective optimization, mathematically, we optimize M objectivesf (x) = [f 1 (x), f 2 (x), . . . , f M (x)] ∈ r M : min f 1 (x), f 2 (x), ..., f M (x) (2) s.t.x ∈ Ω, where x represents the neural architecture from the search space, f i (x) denotes the function of objective i. Modern MOO methods aim to find the problem's entire Pareto frontier, the set of solutions that are not dominated by any other feasible solutions. Here we define dominance y ≺ f x as f i (x) ≤ f i (y) for all metrics f i , and there exists at least one i s.t.f i (x) < f i (y), 1 ≤ i ≤ M . Ifthe condition holds, a solution x is always better than solution y. A.2 One (few)-shot NAS One (few)-shot NAS utilizes a weight-sharing technique, avoiding the need to retrain sampled networks from scratch. Specifically, one-shot NAS initially trains an over-parameterized supernet, including all possible operations and connections within the entire search space, and employs this supernet as a performance estimator to predict an architecture's efficacy [10, 92, 97, 65] . The performance of any architecture within the search space can be approximated by leveraging the welltrained supernet. Few-shot NAS [97, 34, 86, 32, 69] further improves the accuracy of architecture evaluations conducted by supernets. It achieves this by using multiple sub-supernets, thereby reducing the co-adaptation impact among operations [10] . In this work, we employ this few-shot supernet as an efficient proxy for architecture evaluation, with more details provided in Section 3.3.Zero-shot NAS leverages training-free score functions/proxies to efficiently evaluate the performance of neural architectures [41, 85] . Compared to vanilla NAS and one/few-shot NAS, zero-shot NAS is cost-effective due to its training-free nature at the search stage. However, the main disadvantage of zero-shot NAS is its evaluation performance, which can be very inaccurate compared to one/few-shot NAS and vanilla NAS [41] .RL has been widely used for designing system strategies for tasks such as scheduling and resource allocation. For instance, RL has been deployed to develop resource allocation strategies in wireless networks [22, 89, 15, 48] and cloud platforms [19, 47, 94, 20] , as well as for making bitrate decisions in adaptive video streaming systems [57, 49, 42, 82] , and also generating task scheduling strategies in data centers [64, 7, 96, 8] . The key advantage of RL-based methods is their capability to learn strategies and easily adapt to system changes automatically. An RL agent begins by taking the system state as input and proceeds to generate an action, i.e. a scheduling or allocation decision, based on its learned strategies, which is typically randomized at the start. Upon execution of the action, the system provides feedback in the form of a reward, enabling the RL agent to refine its strategies. The key to applying RL lies in the design of effective state inputs, actions, and rewards according to the specific task. Our CE-NAS focuses on tailoring these elements for carbon-efficient NAS tasks, with the guiding principle of simplifying the state and action spaces to expedite efficient and rapid training.We utilize LaMOO [98] to partition the search space, Ω, into several sub-search spaces. This partitioning will be based on the architectures' accuracy E(a) and their evaluation metrics T (a) as observed in the sample set, P. Specifically, LaMOO recursively divides the search space into promising and non-promising regions. Each partitioned region can then be mapped to a node in a search tree. Using Monte-Carlo Tree Search (MCTS), LaMOO selects the most promising sub-space (i.e., tree node) for further exploration based on their UCB values [5] . This optimal sub-space selected by MCTS is denoted as Ω best .State Input. We design the RL state input at time step t as s t = (b t , n t , h t , c t ), where b t is the remaining carbon budget of the current searching task, n t is the number of already searched architectures, h t is the hypervolume of these architectures, and c t is the future carbon traces in the upcoming hour (e.g., predicted by our time series transformer from §3.5.2). The general idea is that being aware of b t helps the RL algorithm plan the overall amount of carbon emission to spend for architecture evaluation, while n t , h t and c t help it decide the best timing to execute the evaluation that generates the highest hypervolume improvements per carbon cost.Network architecture. Our network consists of an input layer, 2 hidden layers, and an output layer. All of the layers are fully connected layers, with hidden sizes of 100, 150, 200 and 100 respectively, which are determined by performing a simple grid search. The final output of the network specifies the probability of distribution of actions.Action. We design the output of the RL agent as K discrete actions, i.e. the output is a vectoro t =< o 1 t , o 2 t , ..., o K t >, where o k t denotes the probability of allocating a ratio of (k-1)GPUs for architecture evaluation. We use K = 8 for our evaluation. We also implement a version of the agent with continuous action space, i.e., the network outputs the mean µ and standard σ of a Gaussian distribution to indicate the probability distribution of the GPU allocation ratio. Our evaluation in §F. 4 shows that the two versions achieve comparable performance.Rewards. As the overall goal is to maximize the cumulative reward that represents the final accuracy obtained by the search task, we design the reward in each time step as the improvement of the best accuracy of the already searched architectures: r t = (co t , hi t , nn t ), where co t denotes the carbon cost in this iteration, hi t represents the hypervolume increase in this iteration, and nn t means the number of new samples in this step.Policy gradient. We adopt the popular actor-critic algorithm to calculate the policy gradient and update the network. Actor-critic has been proved to achieve good performance while enabling fast training in tasks of similar scales [57, 49, 75, 67, 101] . Note that our design is not coupled with any specific DRL algorithm. Thus, replacing actor-critic with other algorithms is easy if needed. The key gradient equation of actor-critic is as follows:EQUATIONwhere ∞ t=0 γ t r t is the cumulative rewards and γ is a discount factor. Actor-critic algorithm includes two networks referred to as actor and critic respectively. In our design, they use the same network structure. The actor is responsible for outputting π θ (s, a), which is the output probability for action a with input state s. The policy parameter θ is the network parameter of the actor. A π θ (s, a) is the advantage function that indicates the performance difference between the current policy and the average performance of policies learned by actor. A π θ (s, a) is obtained with the temporal difference method based on the output of the critic. The calculation is as follows,EQUATIONwhere A(s t , a t ) is an unbiased estimation of A π θ (s, a), θ v is the parameter of the critic. V π θ (s t ; θ v ) is the output of critic that estimates the cumulative reward follow actor policy π θ starting from state s t .Building upon the foundational design of the time series Transformer, we employ LaMOO [98] as our NAS strategy to identify the optimal architectural parameters for our carbon dataset [58] . Our goal is to devise an architecture that not only achieves high forecasting accuracy but also maintains efficient training and latency times. Given that we are working with a standard encoder-decoder Transformer, our search space is straightforward. We focus on searching for: i) the number of encoder layers, ii) the number of decoder layers, iii) the embedding size, and iv) the maximum context length of the time series that the model is capable of considering. Following our NAS search process, we configured a model with 64 encoder layers, 96 decoder layers, with an embedding size of 48. Additionally, we set the context length to 8760.We align our training and testing data with the approach used in [55] by utilizing ElectricityMap [58] , a third-party carbon information service. We select three regions (namely, CISO6 , DE 7 , and PJM8 ) from ElectricityMap [58] to represent distinct carbon trace scenarios for our experiments. The dataset spans from January 1, 2020, to December 31, 2021, with an hourly resolution. We employ a train(validation)-test split of 75%-25%. We trained a single transformer model using data from three different regions simultaneously. Additionally, we conducted separate training sessions for the model using distinct traces from each location. The experimental results from both approaches were similar.In our training configuration, the context length is set to 8760, as determined by our NAS search. We utilize the AdamW optimizer, setting the weight decay to 1 × 10 -2 and selecting beta1 as 0.9 for the exponential decay rate of the first moment estimates, and beta2 as 0.95 for the second moment estimates. The model is trained over 150 epochs with an initial learning rate of 6 × 10 -4 . The batch size is configured at 512 on a single NVIDIA A100 GPU.For testing, to predict carbon intensity for the forthcoming 24 hours, we use all historical data up to the last complete 24-hour block. This block is then replaced with predictions from our model to forecast the carbon intensity for the 25th hour. Subsequent forecasts are generated by updating the most recent hour with its actual value and using the model's prediction for the next hour. We measure forecasting performance using the Mean Absolute Percentage Error (MAPE). For multi-day forecasts, we similarly update and replace values for the 24-, 48-, and 72-hour marks.Figure 6 presents the hourly time series, averaged over a week, demonstrating both actual and predicted carbon intensities for the electricity grids in these areas. The comparison suggests that our model accurately forecasts the actual carbon trace in each region, aligning closely with the observed data.HW-NAS-Bench [40] enhances the original NasBench201 dataset by including additional metrics such as inference latency, parameter size, and number of FLOPs, thereby expanding the evaluative dimensions available for NAS research. NasBench201 itself is a comprehensive open-source benchmark designed for the systematic assessment of NAS algorithms [28] . Within NasBench201, architectures are constructed by stacking cells in sequence. Specifically, each cell is composed of 4 nodes interconnected by 6 edges. Nodes represent feature maps, while edges correspond to various operations that transform one node's output into the next node's input. Operations connecting nodes include zeroization, skip-connection, 1x1 convolution, 3x3 convolution, and 3x3 average pooling. To uniquely identify each architecture, we employ a numeric encoding scheme where the five operations are represented by integers 0 through 4. Consequently, a 6-element vector represents the specific architecture.NasBench301 [93] serves as a surrogate benchmark for the NASNet [102] search space, which has over 10 21 possible architectures. NasBench301 employs a surrogate model trained on approximately 60,000 sampled architectures to predict performance across the entire DARTS [46] search space for the CIFAR-10 image classification task. This surrogate model within NasBench301 has been shown to provide accurate regression outcomes, facilitating reliable evaluations within this vast search space. NasBench301 offers predicted accuracies from the surrogate model, while other fundamental metrics such as the number of parameters (#Params), floating-point operations per second (#FLOPs), and inference time are readily ascertainable during the evaluation phase. In this multi-objective optimization context, our goal is to simultaneously maximize inference accuracy and minimize #Params within the NasBench301 search space.The NASNet search space includes operations such as 3x3 max pool, 3x3 convolutions, 5x5 depthseparable convolutions, and skip connections. The objective here is to identify optimal architectures for both reduction and normal cells, each comprising 4 nodes, culminating in a search space of approximately 3.5 × 10 21 architectures [93, 78] . For the CIFAR-10 task, we adopt the same encoding strategy used in the NASNet search space, representing architectures as 16-digit vectors. The first four digits denote the operations in a normal cell, digits 5-8 correspond to the concatenation patterns in the normal cell, digits 9-12 represent operations in a reduction cell, and the final four digits encode the concatenation patterns in the reduction cell.We chose three types of baselines according to different GPU allocation strategies and NAS evaluation algorithms. During the search process, all search methods employ the state-of-the-art multi-objective optimizer, LaMOO [98] . Specifically, one-shot LaMOO is a method that utilizes one-shot evaluations throughout the search process. The vanilla LaMOO relies on actual training for architecture evaluation throughout the search.We also provide a heuristic strategy that automatically allocates GPU resources between the sampling and evaluation processes given the carbon emissions C t at time t as our baseline. This allocation is based on the energy characteristics of the processes: architecture sampling is often energy-efficient because it does not involve actual training of architectures, while architecture evaluation is often energy-consuming because it does. We assume that the maximum and minimum carbon intensities C max and C min for a future time window are known. G t denotes the total number of available GPUs. λ e and λ s represent the ratio of GPU numbers allocated to the evaluation and sampling processes at a given moment, and λ e + λ s = 1. We calculate λ s as Ccur-Cmin Cmax-Cmin , where C cur is the current carbon intensity. The GPU allocations for the sampling and evaluation processes are, therefore, G t * λ s and G t * λ e . This simple heuristic allocation allows the NAS system to prioritize more energy-efficient sampling tasks during periods of higher carbon intensity, whereas, during low-carbon periods, the system will allocate more resources for energy-intensive evaluation tasks.We use two main metrics to evaluate the carbon and search efficiency of CE-NAS. First, we use relative carbon emission to quantify the amount of CO 2 each NAS method is responsible for. The relative carbon emission is calculated by summing the average carbon intensity (in gCO2/KwH) over the search process. We assume that all NAS methods use the same type of GPU whose power consumption remains the same throughout the search process. Second, we use the metric hypervolume (HV) to measure the "goodness" of searched samples. HV is a commonly used multiobjective optimization quality indicator [24, 25, 98] that considers all dimensions of the search objective. Given a reference point R ∈ r M , the HV of a finite approximate Pareto set P is the M-dimensional Lebesgue measure λ M of the space dominated by P and bounded from below by R. That is,HV (P, R) = λ M (∪ |P| i=1 [R, y i ]), where [R, y i ] denotes the hyper-rectangle bounded by the reference point R and y i . A higher hypervolume denotes better multi-objective results. In this experiment, we calculate the hypervolume (HV) using the accuracy and inference energy of the searched architectures. For this dataset, we use the log hypervolume difference, the same as in [24, 25, 98] , as our evaluation criterion for HW-NASBench, since the hypervolume difference may be minimal over the search process. Therefore, using log hypervolume allows us to visualize the sample efficiency of different search methods. We define HV log_diff := log(HV max -HV cur ) where HV max represents the maximum hypervolume calculated from all points in the search space, and HV cur denotes the hypervolume of the current samples, which are obtained by the algorithm within a specified budget. The HV max in this problem is 4150.7236.For our simulation, we use the training and evaluation time costs for the architectures derived from NasBench201 [28] , and inference energy costs measured on the NVIDIA Edge GPU Jetson TX2 from HW-NASBench [40] . We ran the simulation 10 times with each method.Given the expansive nature of the NasBench301 search space, the maximal hypervolume remains undetermined. Consequently, rather than employing the hypervolume difference as a performance metric, we utilize the absolute hypervolume value to represent the efficacy of our search strategy.Search space overview. Our search space is consistent with the NASNet search space [102] . It comprises eight searchable operations: 3x3 max pooling, 3x3 average pooling, 3x3, 5x5, and 7x7 depthwise convolutions, 3x3 and 5x5 dilated convolutions, and a skip connection. The architecture consists of normal cells, which retain the size of the feature map, and reduction cells, which double the channel size and halve the feature map resolution. Each cell integrates four nodes connected by eight different operations. The search space encompasses a total of 10 21 architectures. We employ the same encoding strategy as prior studies [78] .Search setup We implement low-fidelity estimation methods [102, 66, 79] to expedite the energyconsuming evaluation part in the neural architecture search process. This approach effectively accelerates the evaluation process using cost-effective approximations while largely preserving the true relative ranking of the searched architectures. Initially, we utilize early stopping for training sampled architectures for 200 epochs, in contrast to 600 epochs required for the final training phase. To further hasten training, we reduce the initial channel size from 36 to 18 and increase the batch size to 320. Additionally, the number of layers is scaled down from 24 to 16 during the search phase. To speed up the evaluation process, we use the NVIDIA Automatic Mixed Precision (AMP) library with FP16 for training during the search.We utilize a pre-trained reinforcement learning model based on data from NasBench201 and continually tune this model during the NAS process according to task-specific data. The carbon budget for the search is set at 100 lbs.Training setup The final selected architectures are trained for 600 epochs, using a batch size of 128 and a momentum SGD optimizer with an initial learning rate of 0.025. This rate is adjusted following a cosine learning rate schedule throughout the training. Weight decay is applied for regularization.Search space overview. Our ImageNet search space is modeled after EfficientNet [72] . Specifically, it includes 5-8 stages for each architecture, with the number of stages being determined by NAS. From stages 3 to 8, the type of stage, either Fused-Inverse-Residual-Block (Fused-IRB) or Inverse-Residual-Block (IRB), is searchable. Within each stage, searchable parameters include activation type (e.g., ReLU, Swish), kernel size (e.g., 1, 3, 5, 7), number of layers ([1, 10]), expansion rate ( [2, 7] ), and number of channels (varies based on the stage). Additionally, our search space considers input image resolutions (e.g., 224, 288, 320, 384, 456, 528). The total search space size is approximately 10 31 . Our NAS search focuses on two objectives: top-1 accuracy and TensorRT latency with FP16 on an NVIDIA V100. For TensorRT latency, we fixed the workspace at 10GB for all runs and benchmarked latency using a batch size of 1 with explicit shape configuration, reporting the average latency from 1000 runs.Search setup As with CIFAR-10, we implement low-fidelity estimation methods [102, 66, 79] to accelerate the energy-consuming evaluation part in the neural architecture search process. Sampled architectures are trained for 150 epochs using early stopping, as opposed to the 450 epochs required in the final training phase. During the evaluation in the search process, we also reduce the channel size by a factor of 4. To further speed up the evaluation process, we leverage AMP with FP16 for training searched architectures.Consistent with our CIFAR-10 approach, we employ a pre-trained reinforcement learning model based on data from NasBench201 and continually fine-tune this model during the NAS process using ImageNet task data. The carbon budget for the search is set at 1000 lbs.Training setup For each architecture in the Pareto frontier, we train it on 8 Tesla V100 GPUs with a resolution of 320x320 in our two-objective accuracy and TensorRT latency. We utilize a standard SGD optimizer with Nesterov momentum of 0.9 and set the weight decay at 3 × 10 -5 . Each architecture undergoes training for a total of 450 epochs, with the initial 10 epochs serving as the warm-up period. During these warm-up epochs, we apply a constant learning rate of 0.01. The remaining epochs are trained with an initial learning rate of 0.1, using a cosine learning rate decay schedule [51] , and a batch size of 1024 (i.e., 128 images per GPU). The model parameters undergo decay at a factor of 0.9997 to further enhance our models' training performance.We conducted ten runs of LaMOO (i.e., search space split) with a random search on the HW-NASBench dataset [40] . In addition, we performed random sampling for both the LaMOO-selected region and the entire search space, conducting 50 trials for each. The distribution of accuracy and edge GPU energy consumption of the architectures in both the LaMOO selected region and the entire search space can be seen in Figure 7 .Specifically, our results show that the architectures in the region selected by LaMOO have higher average accuracy and lower average edge GPU energy consumption compared to those in the entire search space. On average, the accuracy of the architectures in the LaMOO selected region is 72.12, while the accuracy in the entire search space is 68.28. The average edge GPU energy for the LaMOO selected region is 16.59 mJ, as opposed to 22.84 mJ for the entire space.Furthermore, as illustrated in Figure 7 (a), we observe that searching within the LaMOO-selected region yielded a tighter distribution, and the median hypervolume demonstrated an improvement compared to searching across the entire search space. These results suggest the efficacy of using LaMOO to partition the search space for NAS. Our framework also supports neural architecture search (NAS) within a specified time budget, in addition to the option of a carbon budget. We adapt the time budget to replace the carbon budget in the state and reward functions of reinforcement learning. Figure 9 presents the results of our Carbon-Efficient NAS (CE-NAS) under various time budgets. As illustrated, CE-NAS significantly outperforms both heuristic and one-shot methods in terms of hypervolume across different time budgets. While there is a minor decline in performance compared to vanilla NAS as measured by hypervolume, it is important to note that the carbon cost of our CE-NAS is on par with that of the heuristic method. This represents a substantial reduction compared to vanilla NAS. For instance, as depicted in Figure 9 (d), vanilla NAS incurs three times the carbon cost of CE-NAS but only achieves a marginal improvement in hypervolume.As detailed in Section 3.5.2, we implemented a transformer-based carbon predictor to forecast the carbon intensity for the upcoming hour, integrating this predicted value into our framework. The accuracy of this forecast is demonstrated in Fig. 2 . In the same section, we compare the performance of NAS searches using both the predicted carbon intensity and the actual carbon intensity. Table5 displays the results of our system operating under various carbon budget constraints with both predicted and actual carbon intensities. The data in this table indicates that, across different carbon budgets, employing the predicted carbon intensity trace yields results that are only marginally less effective than using the actual carbon trace, as measured by hypervolume. Recall that our RL design includes both a version with a discrete action space and another with a continuous action space. We compare their performance in Table 6 and observe that the two versions deliver comparable performance, with the continuous version performing slightly worse, experiencing a degradation within 3%. Such degradation can be attributed to that the continuous version complicates the action space and increases the learning difficulty. Despite this, we anticipate that the continuous version will outperform its discrete counterpart as the number of available GPUs grows, owing to its more fine-grained GPU allocation decision granularity that gives it the potential to generating better-optimized strategies. We leave the exploration of how GPU cluster sizes impact the relative performance of the two version for future study.Answer Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).We define dominance y ≺ f x as fi(x) ≤ fi(y) for all functions fi, and exists at least one i s.t. fi(x) < fi(y), 1 ≤ i ≤ M , where M is the number of objectives.CISO: California Independent System OperatorDE: GermanyPJM: Pennsylvania-Jersey-Maryland InterconnectionWe did not account for the carbon cost of training and inference for the RL models and the time-series transformer, as they consume negligible CO2 compared to the NAS process.CISO: California Independent System OperatorDE: GermanyPJM: Pennsylvania-Jersey-Maryland Interconnection 


Decision: This paper introduces CE-NAS, a NAS (Neural Architecture Search) framework designed to minimize the carbon footprint of architecture searches. CE-NAS optimizes the use of GPU resources and NAS evaluation methods based on real-time carbon intensity forecasts, which their model predicts more accurately than existing methods. All the reviewers are satisfied with the authors' response. The authors should consider including a discussion on zero-shot NAS in the related work section to ensure the completeness of the paper.

Reviews: review: paper_summary: paper_summary: The paper introduces CE-NAS, a novel framework for neural architecture search that prioritizes carbon efficiency in the model design process. It addresses the high carbon cost associated with NAS by dynamically adjusting GPU resources based on predicted carbon intensity and search results. The framework integrates a reinforcement-learning agent, a time-series transformer for carbon intensity prediction, and a multi-objective optimizer to balance energy-efficient sampling with energy-intensive evaluation tasks. Experiments demonstrate CE-NAS's ability to reduce carbon emissions significantly while achieving good results on various NAS datasets and tasks.
strength_weakness: strength_weakness: * Experimental comparisons are focused on traditional and one-shot NAS methods. However, zero-shot NAS methods are not mentioned in the paper. Some recent zero-shot NAS methods have achieved better results and higher efficiency. 
* The integration of multiple components introduces complexity in implementation.
questions: questions: Please describe the advantages of CE-NAS when compared to zero-shot NAS methods.
limitations: limitations: None
paper_summary: paper_summary: This paper introduces a NAS framework, named CE-NAS, which focuses on reducing the carbon footprint of architecture search. To do this, CE-NAS decides when to use which NAS evaluation method depending the current carbon intensity, and how to allocate available GPU resources. Their carbon forecasting model also outperforms previous models for predicting carbon intensity. In experiments they show that CE-NAS outperforms other methods, like vanilla NAS and one-shot NAS, given a fixed carbon budget.

Sampling and evaluation are decoupled in the framework. Sampling is done using a BO algorithm which generates new architectures. These are then added to a queue of architectures that will be put on the available GPU resources for full training. A proxy evaluation is also done before this using the supernet constructed from the most promising region of the search space, according to a multi-objective optimiser. This approximate performance is used by the BO sampling algorithm.

During periods of high carbon emissions, the sampling is the priority, and in periods of low carbon emissions, the full training is prioritised. In the latter, it also updates the promising search space region used to construct the supernet. Every hour, the priority is reassigned and GPU resources are reallocated.
strength_weakness: strength_weakness: My main concern is that the paper does not make it clear enough how a practitioner would use the framework themselves. The critical point is that real-time carbon emission data needs to be available locally and there is no discussion of how realistic this is or any suggestion for where to find it. Without this aspect of carbon efficiency, there is still a decent contribution of a framework that seems to outperform other methods for multi-objective NAS but I would like more discussion on how I could realistically use the full framework myself.

Minor comments:
* Figure 1 needs a more descriptive caption that clarifies the diagram and how CE-NAS works on a high level. The diagram is complex and hard to parse at the moment.
* The CIFAR10 performance differs between the abstract (97.35%) and the introduction (97.84%). In the end of the introduction, in the list of contributions, it is again 97.35%. This seems to be the two different models from the Pareto frontier. I suggest either explaining that you find both or be consistent with which you highlight in the text.
* Is there a citation for the claim on lines 97-99? *A recent NAS work on transformer revealed that their comprehensive architecture search used 979 million training steps, totaling 274,120 GPU hours and resulting in 626,155 pounds of CO2 emissions.*
* When reading, I was confused by both the relative carbon emission and hypervolume metrics. I think there should be a short explanation in the main paper along with the pointer to the full description in the appendix. This pointer is currently in section 4.2.1 but hypervolume is first mentioned in 3.5.1 which causes confusion for a while.
* The caption for Figure 2 states that CE-NAS has the lowest relative carbon emission but this seems to be wrong. One-shot LaMOO has a lower CO_2 number in (a) and both One-shot LaMOO and Heuristic have lower numbers in (b).
* On line 263: *“To date, there are three popular open-source NAS datasets, NasBench101 [77], HW-NAS-BENCH [25], and NasBench301 [80]”* There are other popular ones, e.g. DARTS, and I think a more appropriate term here is NAS benchmarks instead of datasets.
* Table 2, row CE-Net-P1 says hybird instead of hybrid.
* In Table 3, MnasNet achieves a TensorRT Latency of 0.53, which is lower than CE-Net-G1 and should be the bolded number (unless there was a typo).
* Figure 9 of the appendix needs a caption that better explains it.
questions: questions: How easy is it in practice to access live information about carbon efficiency? This is key to the method and reducing carbon emissions from the NAS process, but as a practitioner I would not know how to get this information. If there is a straightforward way to get this data from, e.g. ElectricityMap, I would like to see that described in the paper or pointed to in the appendix. Additionally, it would be good to have a discussion on the availability of carbon emissions data across different countries/regions as I imagine it will be unavailable in many places.
limitations: limitations: There is no clear section that discusses limitations, and I think this is where my suggestion on discussing the availability of carbon emissions data could be put.
paper_summary: paper_summary: Neural architecture search (NAS) is known to be extremely compute intensive, which also increases the corresponding energy consumption and the carbon costs due to energy production. In this work, the authors attempt to reduce the carbon cost of performing NAS. To this end, they propose a reinforcement learning (RL) agent that switches between energy intensive evaluation of architectures (standard NAS) and energy-efficient sampling (few-shot NAS), during high carbon intensive and low carbon intensity regimes, respectively. To predict the carbon intensity for efficiency scheduling of the two NAS regimes, this work uses a time series transformer which predicts the carbon intensity of energy production using historical information. Using a combination of RL agent for scheduling based on the time series transformer based carbon intensity prediction, the work shows considerable reduction in the overall carbon costs of NAS while still achieving architectures that are comparable to SOTA.
strength_weakness: strength_weakness: * **Spatial carbon intensity variations**: This work mainly addresses the temporal variations of carbon intensity. While it can be argued that it is the first step, and also an easier mode of control, it would also be interesting to see how the scheduling can take different geographical locations into account. We can assume that large scale DL compute happens on cloud instances, and this could improve the broader usefulness of this proposed method. For instance, could the scheduler take the carbon intensity of the three regions considered in this work, and further optimize the carbon costs, as is commonly done in carbon-aware scheduling literature [4]? 

* **NasBench101**: In L.256 authors state that NasBench101 does not provide accuracy. I am not sure what they mean by this. NasBench101 does provide training and validation accuracy for all the 423k architectures considered in the search space. Clarification can be useful.

* **Beyond CNNs**: This is a general critique of most NAS works, as we are exploring only CNN architectures that are by now very efficient. Why did the authors not consider other open domains? For instance, something that takes a hybrid space of CNNs and transformers [1] into account? 

* **Dominance number o(a)**: If _a_ is a unique architecture (L 175), what are the _number of samples that dominate_ a given architecture _a_ (L 178)? Or is _a_ a proxy for a sub-space that is being explored? This requires some clarification. And also, on where exactly is the dominance number being used? Is it within the MOO algorithms?  

#### Other comments
* L 47: Authors seem to allude that low latency is always corresponding to low energy/carbon costs. This is not always the case as has been shown in many works [5,3] 
* Authors are missing some key references pertaining to carbon-aware scheduling [4] and energy/carbon efficient NAS [2,3]. While [2,3] are not exactly efficient NAS they do grapple with some of the concerns expressed in this work. These suggestions for missing reviews are only samples and not comprehensive; I encourage the authors to explore further starting from these papers.

#### References

[1] Xu, Shuying, and Hongyan Quan. "ECT-NAS: Searching efficient CNN-transformers architecture for medical image segmentation." In 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 1601-1604. IEEE, 2021.

[2] Dou, S., Jiang, X., Zhao, C. R., & Li, D. (2023). EA-HAS-bench: Energy-aware hyperparameter and architecture search benchmark. In The Eleventh International Conference on Learning Representations.

[3] Bakhtiarifard, P., Igel, C., & Selvan, R. (2024, April). EC-NAS: Energy consumption aware tabular benchmarks for neural architecture search. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 5660-5664). IEEE.

[4] Radovanović, A., Koningstein, R., Schneider, I., Chen, B., Duarte, A., Roy, B., ... & Cirne, W. (2022). Carbon-aware computing for datacenters. IEEE Transactions on Power Systems, 38(2), 1270-1280. 

[5] Henderson, P., Hu, J., Romoff, J., Brunskill, E., Jurafsky, D., & Pineau, J. (2020). Towards the systematic reporting of the energy and carbon footprints of machine learning. Journal of Machine Learning Research, 21(248), 1-43.
questions: questions: See weaknesses.
limitations: limitations: The authors do no discuss limitations of the work. I would encourage them to discuss the effects of the carbon intensity predictor. It could be done at two steps: Firstly, dependent on the accuracy. What if the predicted carbon intensity is inaccurate? What is the impact of scheduling the energy intensive step during carbon intensive times? 

Second, at a higher level. Let us assume everyone starts doing carbon efficient NAS (or even computations in general) then there is a possible risk of rebound effect which might then go on to increase the carbon footprint paradoxically.
paper_summary: paper_summary: The paper proposes a carbon-efficient neural architecture search (NAS), CE-NAS, to reduce the carbon emitted during the NAS process by dynamically allocate GPU resources based on the predicted future carbon intensity. CE-NAS leverages reinforcement learning for generating the probability of choosing which model of NAS (one-shot or vanilla) based on the predicted carbon intensity by a time-series transformer. One-shot model would be chosen to run in high carbon intensity periods and vanilla NAS would be chosen otherwise. Alongside emitting less carbon, CE-NAS achieves a competitive accuracy against state-of-the art models for CIFAR-10 and ImageNet datasets
strength_weakness: strength_weakness: 1. The experiments are only conducted in California, it would be good if authors show their results for multiple locations.
2. The baseline models for comparing the proposed time-series carbon intensity predictor could use one more model proposed by Zhang et al. (Title: A GNN-based Day Ahead Carbon Intensity Forecasting Model for Cross-Border Power Grids) which is also a strong model for prediction.
3. The overhead of switching between the models has not been considered considering carbon intensity variations.
questions: questions: 1. How would the CE-NAS perform if we have a deadline for used GPU hours? e.g., we can only use 2k hours of GPU and we can't exceed that amount.
2. How robust is CE-NAS in regions that we have high carbon intensity prediction errors?
limitations: limitations: 1. The authors can discuss the scalability limitations of their model when the search space becomes more complex

Comments: title: Thanks for the efforts.
Response to authors
Response to author rebuttal
comment: Thank you for your kind feedback. We apologize for the code being inaccessible earlier. We have now made it public, and we will organize and release the code on GitHub once the paper is accepted.
Thanks for the response. I acknowledge that zero-shot NAS has its limitations and agree that integrating it into the CE-NAS framework is a promising direction. The work's novel combination with carbon efficiency indeed offers insights, and as such, I will increase my score.

I suggest the authors to include a discussion on zero-shot NAS in the related work section to ensure the completeness of the paper. Additionally, the provided code in the Supp. is currently inaccessible, requiring an application for access. I kindly urge the authors to make the code publicly available.
Thanks to the authors for their detailed response. After reading this and the other reviews and responses, I think the paper will be improved. On the assumption that these changes will be made for the final version, have decided to upgrade my score.
The authors have addressed most of my concerns in their rebuttal with appropriate clarifications. I urge them include these clarifications in the final version of the paper. 

I will raise my score to Accept.
Thank you authors for addressing the mentioned concerns.