Title: LayerNAS: Neural Architecture Search in Polynomial Complexity
Venue: NeurIPS 2023
Accepted: no
Keywords: AutoML

Neural Architecture Search

Model Optimization
Abstract: 
Neural Architecture Search (NAS) has become a popular method for discovering effective model architectures, especially for target hardware. As such, NAS methods that find optimal architectures under constraints are essential. In our paper, we propose LayerNAS to address the challenge of multi-objective NAS by transforming it into a combinatorial optimization problem, which effectively constrains the search complexity to be polynomial. 

LayerNAS rigorously derives its method from the fundamental assumption that modifications to previous layers have no impact on the subsequent layers. When dealing with search spaces containing $L$ layers that meet this requirement, the method performs layerwise-search for each layer, selecting from a set of search options $\mathbb{S}$. LayerNAS groups model candidates based on one objective, such as model size or latency, and searches for the optimal model based on another objective, thereby splitting the cost and reward elements of the search. This approach limits the search complexity to $ O(H \cdot |\mathbb{S}| \cdot L) $, where $H$ is a constant set in LayerNAS.

Our experiments show that LayerNAS is able to consistently discover superior models across a variety of search spaces in comparison to strong baselines, including search spaces derived from NATS-Bench, MobileNetV2 and MobileNetV3. 


1

Abstract

Neural Architecture Search (NAS) has become a popular method for discovering effective model architectures, especially for target hardware. As such, NAS methods that find optimal architectures under constraints are essential. In our paper, we propose LayerNAS to address the challenge of multi-objective NAS by transforming it into a combinatorial optimization problem, which effectively constrains the search complexity to be polynomial.

LayerNAS rigorously derives its method from the fundamental assumption that modifications to previous layers have no impact on the subsequent layers. When dealing with search spaces containing L layers that meet this requirement, the method performs layerwise-search for each layer, selecting from a set of search options S . LayerNAS groups model candidates based on one objective, such as model size or latency, and searches for the optimal model based on another objective, thereby splitting the cost and reward elements of the search. This approach limits the search complexity to O ( H · | S | · L ) , where H is a constant set in LayerNAS.

Our experiments show that LayerNAS is able to consistently discover superior models across a variety of search spaces in comparison to strong baselines, including search spaces derived from NATS-Bench, MobileNetV2 and MobileNetV3.

1 Introduction

With the surge of ever-growing neural models used across all ML-based disciplines, the efficiency

of neural networks is becoming a fundamental factor in their success and applicability. A carefully

crafted architecture can achieve good quality while maintaining efficiency during inference. However, 22

23

designing optimized architectures is a complex and time-consuming process - this is especially

true when multiple objectives are involved, including the model's performance and one or more 24

cost factors reflecting the model's size, Multiply-Adds (MAdds) and inference latency. Neural 25

Architecture Search (NAS), is a highly effective paradigm for dealing with such complexities. NAS 26

automates the task and discovers more intricate and complex architectures than those that can be 27

found by humans. Additionally, recent literature shows that NAS allows to search for optimal models 28

under specific constraints (e.g., latency), with remarkable applications on architectures such as 29

MobileNetV3 [15], EfficientNet [34] and FBNet [36]. 30

Most NAS algorithms encode model architectures using a list of integers, where each integer 31

represents a selected search option for the corresponding layer. In particular, notice that for a 32

given model with L layers, where each layer is selected from a set of search options S , the search 33

space contains O ( | S | L ) candidates with different architectures. This exponential complexity presents 34

a significant efficiency challenge for NAS algorithms. 35

78.6

ImageNet (%)

78.0

77.5

77.1

77.0

EfficientNet

Magic-AT

on

FairNAS

76.5

ShapleyNAS

GreedyNAS

76.0

Top-1

75.6

75.5

★

200

250

300

350

400

450

500

550

600

MillionofMAdds

In this paper, we present LayerNAS, an algorithm that addresses the problem of Neural Architecture 36

Search (NAS) through the framework of combinatorial optimization. The proposed approach decou37

38

ples the constraints of the model and the evaluation of its quality, and explores the factorized search

39

space more effectively in a layerwise manner, reducing the search complexity from exponential to

42

43

44

searching for layer

i

can be constructed from one of the models in layer

i

-

1

. For search spaces that satisfy this assumption, LayerNAS enforces a directional search process from the first layer to the

last layer. The directional layerwise search makes the search complexity

46

47

48

49

50

51

52

53

54

55

56

For multi-objective NAS problems, LayerNAS treats model constraints and model quality as separate metrics. Rather than utilizing a single objective function that combines multi-objectives, LayerNAS stores model candidates by their constraint metric value. Let M i,h be the best model candidate for layer i with cost = h . LayerNAS searches for optimal models under different constraints in the next layer by adding the cost of the selected search option for next layer to the current layer, i.e., M i,h . This transforms the problem into the following combinatorial optimization problem: for a model with L layers, what is the optimal combination of options for all layers needed to achieve the best quality under the cost constraint? If we bucketize the potential model candidates by their cost, the search space is limited to O ( H · | S | · L ) , where H is number of buckets per layer. In practice, capping the search at 100 buckets achieves reasonable performance. Since this holds H constant, it makes the search complexity polynomial.

57

58

59

60

61

62

63

64

65

Our contributions can be summarized as follows:

We propose LayerNAS, an algorithm that transforms the multi-objective NAS problem to a Combinatorial Optimization problem. This is a novel formulation of NAS.

LayerNAS is directly designed to tackle the search complexity of NAS, and reduce the search complexity from O ( | S | L ) to O ( H · | S | · L ) , where H is a constant defined in the algorithm.

We demonstrate the effectiveness of LayerNAS by identifying high-performing model architectures under various Multiply-Adds (MAdds) constraints, by searching through search spaces derived from MobileNetV2 [30] and MobileNetV3 [15].

2 Related Work 66

The survey by [12] categorizes methods for Neural Architecture Search into three dimensions: search 67

space, search strategy, and performance estimation strategy. The formulation of NAS as different 68

O

(

C

· |

S

| ·

L

), where

C

is

1.Selectparentmodel

3.Updatethebucket

andgeneratechildren

withbettermodel

2.Filteroutmodels

outofcostrange

problems has led to the development of a diverse array of search algorithms. Bayesian Optimization 69

is first adopted for hyper-parameter tuning [3, 9, 13, 20]. Reinforcement Learning is utilized for 70

training an agent to interact with a search space [45, 27, 46, 19]. Evolutionary algorithms [24, 29] 71

have been employed by encoding model architectures to DNA and evolving the candidate pool. 72

73

ProgressiveNAS [22] uses heuristic search to gradually build models by starting from simple and

74

75

76

shallow model architectures and incrementally adding more operations to arrive at deep and complex final architectures. This is in contrast to LayerNAS, which iterates over changes in the layers of a full

complex model.

Recent advancements in mobile image models, such as MobileNetV3 [15], EfficientNet [34], FB77 Net [36], are optimized by NAS. The search for these models is often constrained by metrics such 78 as FLOPs, model size, latency, and others. To solve this multi-objective problem, most NAS al79 gorithms [33, 5] design an objective function that combines these metrics into a single objective. 80 LEMONADE [11] proposes a method to split two metrics, and searches for a Pareto front of a family 81 of models. Once-for-All [4] proposes progressive shrinking algorithm to efficiently find optimal 82 model architectures under different constraints. 83

84

85

86

87

88

89

90

Larger models tend to have better performance compared to smaller models. However, the increased size of models also means increased computational resource requirement. As a result, the optimization of neural architectures within constrained resources is an important and meaningful aspect of NAS problems, which can be solved as multi-objective optimization [16]. There is increasing interest in treating NAS as a compression problem [43, 42] from an over-sized model. These works indicate that compressing with different configurations on each layer leads to a model better than uniform compression. Here, NAS can be used to search for optimal configurations [14, 25, 35].

91

92

93

94

95

96

The applicability of NAS is significantly influenced by the efficiency of its search process. One-shot algorithms [23, 5, 1, 2] provide a novel approach by constructing a supernet from the search space to

perform more efficient NAS. However, this approach has limit on number of branches in supernet due to the constraints of supernet size. The search cost is not only bounded by the complexity of search

space, but also the cost of training under "train-and-eval" paradigm. Training-free NAS [26, 6, 44, 32]

breaks this paradigm by estimating the model quality with other metrics that are fast to compute.

However, the search quality heavily relies on the effectiveness of the metrics. 97

3 Problem Definition 98

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

4 Method

We propose LayerNAS as an algorithm that leverages layerwise attributes. When searching models M i on layer i , we are searching for architectures in the form of ( s 1 ..i -1 , x i , o i +1 ..L ) . s 1 ..i -1 are the selected options for layer 1 ..i -1 , and o i +1 ..L are the default, predefined options. x i is the search option selected for layer i , which is the current layer in the search. In this formulation, only layer i can be changed, all preceding layers are fixed, and all succeeding layers are using the default option. In topology search, the default option is usually no-op; in size search, the default option can be the option with most computation.

LayerNAS operates on a search space that meets the following assumption, which has been implicitly utilized by past chain-structured NAS techniques [22, 33, 15].

Assumption 4.1. The optimal model M i on layer i can be constructed from a model m ∈ M i -1 , where M i -1 is a set of model candidates on layer i -1 .

This assumption implies:

Enforcing a sequential search process is possible when exploring layer i because improvements to the model cannot be achieved by modifying layer i -1 .

The information for finding an optimal model architecture on layer i was collected when searching for model architectures on layer i -1 .

Search spaces that are constructed in a layerwise manner, such as those in size search problems discussed in Section 3, can usually meet this assumption. Each search option can completely define how to construct a succeeding layer, and does not depend on the search options in previous layers.

It's worth noting that not all search spaces can meet the assumption. Succeeding layers may be coupled with or affect preceding layers in some cases. In practice, we transform the search space in Section 5 to ensure that it meets the assumption.

4.1 LayerNAS for Topology Search

The LayerNAS algorithm is described by the pseudo code in Algorithm 1. M l is a set of model candidates on layer l . M l,h is the model on layer l mapped to h ∈ H , a lower dimensional representation.

H is usually a finite integer set, so that we can index and store models. 151

Topology search: the search space defines a graph with multiple nodes. The objective is to identify an optimal topology for connecting nodes with different operations. This task allows for the exploration of novel architectures.

Size search or compression search: the search occurs on a predefined model architecture with multiple layers. Each layer can be selected from as a set of search options. Empirically, the best-performing model is normally the one with the most parameters per layer. Therefore, in practice, we aim to search for the optimal model under certain constraints. NATSBench size search [10] provides a public dataset for this type of task. MobilNetV3 [15], EfficientNet [34], FBNet [36] also establish the search space in this manner. This problem can also be viewed as a compression problem [14], as reducing the layer size serves as a means of compression by decreasing the model size, FLOPs and latency.

Scale search: model architectures are uniformly configured by hyper-parameters, such as number of layers or size of fully-connected layers. This task views the model as a holistic entity and uniformly scales it up or down, rather than adjusting individual layers or components.

This taxonomy illustrates the significant variation among NAS problems. Rather than proposing a general solution to address all of them, we propose to tackle with search spaces in a layerwise manner. Specifically, we aim to find a model with L layers. For each layer i , we select from a set of search options S i . A model candidate M can be represented as a tuple with size L : ( s 1 , s 2 , ..., s L ) . s i ∈ S i is a selected search option on layer i . The objective is to find an optimal model architecture M = ( s 1 , s 2 , ..., s L ) with the highest accuracy:

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

Algorithm 1 LayerNAS algorithm

Some functions can be customized for different NAS problems with a-priori knowledge:

select: samples a model from a set of model candidates in the previous layer M l . It could be a mechanism of evolutionary algorithm [29] or a trained predictor [22] to select most promising candidates. The method will also filter out model architectures that we do not need to search, usually a model architecture that is invalid or known not to generate better candidates. This can significantly reduce the number of candidates to search.

apply_search_option: applies a search option from S l +1 on M l to generate M l +1 . We currently use random selection in the implementation, though, other methods could lead to improved search option.

ϕ : M → H maps model architecture in M to a lower dimensional representation H . ϕ could be an encoded index of model architecture, or other identifiers that group similar model architectures. We discuss this further in 4.2. When there is a unique id for each model, LayerNAS will store all model candidates in M .

In this algorithm, the total number of model candidates we need to search is ∑ L i =1 | M i | · | S i | . It has a polynomial form, but | M L | = O ( | S | L ) if we set ϕ ( M ) as unique id of models. This does not limit the order of | M L | to search. For topology search, we can design a sophisticated ϕ to group similar model candidates. In the following discussion, we will demonstrate how to lower the order of | M L | in multi-objective NAS.

4.2 LayerNAS for Multi-objective NAS

LayerNAS is aimed at designing an efficient algorithm for size search or compression search problems. 171 As discussed in Section 3, such problems satisfy Assumption 4.1 by nature. Multi-objective NAS 172 usually searches for an optimal model under some constraints, such as model size, inference latency, 173 hardware-specific FLOPs or energy consumption. We use 'cost' as a general term to refer to these 174 constraints. These 'cost' metrics are easy to calculate and can be determined when the model 175 architecture is fixed. This is in contrast to calculating accuracy, which requires completing the model 176 training. Because the model is constructed in a layer-wise manner, the cost of the model can be 177 estimated by summing the costs of all layers. 178

Hence, we can express the multi-objective NAS problem as, 179

where Cost ( s i ) is the cost of applying option s i on layer i .

We introduce an additional assumption by considering the cost in Assumption 4.1:

Assumption 4.2. The optimal model M i with cost = C when searching for layer i can be constructed from the optimal model M i -1 with cost = C -Cost ( s i ) from M i -1 .

In this assumption, we only keep one optimal model out of a set of models with similar costs. Suppose we have two models with the same cost, but M i has better quality than M ′ i . The assumption will be satisfied if any changes on following layers to M i will generate a better model than making the same change to M ′ i .

By applying Assumption 4.2 to Equation (2), we can formulate the problem as combinatorial optimization:

This formulation decouples cost from reward, so there is no need to manually design an objective function to combine these metrics into a single value, and we can avoid tuning hyper-parameters of such an objective. Formulating the problem as combinatorial optimization allows solving it efficiently using dynamic programming. M l,h can be considered as a memorial table to record best models on layer l at cost h . For layer l , M l generates the M l +1 by applying different options selected from S l +1 on layer l +1 . The search complexity is O ( H · | S | · L ) .

We do not need to store all M l,h candidates, but rather group them with the following transformation:

where H is the desired number of buckets to keep. Each bucket contains model candidates with costs in a specific range. In practice, we can set H = 100 , meaning we store optimal model candidates within 1% of the cost range.

Equation (4) limits | M i | to be a constant value since H is a constant. min Cost ( M i ) and max Cost ( M i ) can be easily calculated when we know how to select the search option from S i +1 .. S L in order to maximize or minimize the model cost. This can be achieved by defining the order within S i . Let s i = 1 represent the option with the maximal cost on layer i , and s i = | S | represent the option with the minimal cost on layer i . This approach for constructing the search space facilitates an efficient calculation of maximal and minimal costs.

The optimization applied above leads to achieving polynomial search complexity O ( H · | S | · L ) . O ( | M | ) = H is upper bound of the number of model candidates in each layer, and becomes a constant after applying Equation (4). | S | is the number of search options on each layer.

LayerNAS for Multi-objective NAS does not change the implementation of Algorithm 1. Instead, we configure methods to perform dynamic programming with the same framework:

ϕ : groups M i by their costs with Equation (4)

select: filters out M l if all M l +1 constructed from it are out of the range of target cost. This significantly reduces the number of candidates to search.

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

In practice, Assumption 4.2 is not always true because accuracy may vary in each training trial. The algorithm may store a lucky model candidate that happens to get a better accuracy due to variation. We store multiple candidates for each h to reduce the problem from training accuracy variation.

5 Experiments

5.1 Search on ImageNet

Search Space: we construct several search spaces based on MobileNetV2, MobileNetV2 (width multiplier=1.4), MobileNetV3-Small and MobileNetV3-Large. For each search space, we set similar backbone of the base model. For each layer, we consider kernel sizes from {3, 5, 7}, base filters and expanded filters from a set of integers, and a fixed strides. The objective is to find better models with similar MAdds of the base model.

To avoid coupling between preceding and succeeding layers, we first search the shared base filters in each block to create residual shortcuts, and search for kernel sizes and expanded filters subsequently. This ensures the search space satisfy Assumption 4.1.

We estimate and compare the number of unique model candidates defined by the search space and the maximal number of searches in Table 1. In the experiments, we set H = 100 , and store 3 best models with same h -value. Note that the maximal number of searches does not mean actual searches conducted in the experiments, but rather an upper bound defined by the algorithm.

A comprehensive description of the search spaces and discovered model architectures in this experiment can be found in the Appendix for further reference.

Search, train and evaluation:

During the search process, we train the model candidates for 5 epochs, and use the top-1 accuracy on ImageNet as a proxy metrics. Following the search process, we select several model architectures with best accuracy on 5 epochs, train and evaluate them on 4x4 TPU with 4096 batch size (128 images per core). We use RMSPropOptimizer with 0.9 momentum, train for 500 epochs. Initial learning rate is 2.64, with 12.5 warmup epochs, then decay with cosine schedule.

Results

We list the best models discovered by LayerNAS, and compare them with baseline models and results from recent NAS works in Table 2. For all targeted MAdds, the models discovered by LayerNAS achieve better performance: 69.0% top-1 accuracy on ImageNet for 61M MAdds, a 1.6% improvement over MobileNetV3-Small; 75.6% for 229M MAdds, a 0.4% improvement over MobileNetV3-Large; 77.1% accuracy for 322M MAdds, a 5.1% improvement over MobileNetV2; and finally, 78.6% accuracy for 627M MAdds, a 3.9% improvement over MobileNetV2 1.4x.

Note that for all of these models, we include squeeze-and-excitation blocks [17] and use Swish activation [28], in order to to achieve the best performance. Some recent works on NAS algorithms, as well as the original MobileNetV2, do not use these techniques. For a fair comparison, we also list the model performance after removing squeeze-and-excitation and replacing Swish activation with ReLU. The results show that the relative improvement from LayerNAS is present even after removing these components.

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

5.2 NATS-Bench

The following experiments compare LayerNAS with others NAS algorithms on NATS-Bench [10]. We evaluate NAS algorithms from these three perspectives:

Candidate quality: the quality of the best candidate found by the algorithm, as can be indicated by the peak value in the chart.

Stability: the ability to find the best candidate, after running multiple searches and analyzing the average value and range of variation.

Efficiency: The training time required to find the best candidate. The sooner the peak accuracy candidate is reached, the more efficient the algorithm.

NATS-Bench topology search

NATS-Bench topology search defines a search space on 6 ops that connect 4 tensors, each op has 5 options (conv1x1, conv3x3, maxpool3x3, no-op, skip). It contains 15625 candidates with their number of parameters, FLOPs, accuracy on Cifar-10, Cifar-100 [21], ImageNet16-120 [7].

In Table 3, we compare with recent state-of-the-art methods. Although training-free NAS has advantage of lower search cost, LayerNAS can achieve much better results.

NATS-Bench size search

NATS-Bench size search defines a search space on a 5-layer CNN model, each layer has 8 options on different number of channels, from 8 to 64. The search space contains 32768 model candidates. The one with the highest accuracy has 64 channels for all layers, we can refer this candidate as 'the largest model'. Instead of searching for the best model, we set the goal to search for the optimal model with 50% FLOPs of the largest model.

Under this constraints for size search, we implement popular NAS algorithms for comparison, which are also used in the original benchmark papers [40, 10]: random search, proximal policy optimization (PPO) [31] and regularized evolution (RE) [29]. We conduct 5 runs for each algorithm, and record the best accuracy at different training costs.

LayerNAS treats this as a compression problem. The base model, which is the largest model, has 64 channels on all layers. By applying search options with fewer channels, the model becomes smaller,

faster and less accurate. The search process is to find the optimal model with expected FLOPs. By 281 filtering out candidates that do not produce architectures falling within the expected FLOPs range, 282 we can significantly reduce the number of candidates that need to be searched. 283

6 Conclusion and Future Work

In this research, we propose LayerNAS that formulates Multi-objective Neural Architecture Search to Combinatorial Optimization. By decoupling multi-objectives into cost and accuracy, and leverages layerwise attributes, we are able to reduce the search complexity from O ( | S | L ) to O ( H · | S | · L ) .

Our experiment results demonstrate the effectiveness of LayerNAS in discovering models that achieve superior performance compared to both baseline models and models discovered by other NAS algorithms under various constraints of MAdds. Specifically, models discovered through LayerNAS achieve top-1 accuracy on ImageNet of 69% for 61M MAdds, 75.6% for 229M MAdds, 77.1% for 322M MAdds, 78.6% for 627M MAdds. Furthermore, our analysis reveals that LayerNAS outperforms other NAS algorithms on NATS-Bench in all aspects including best model quality, stability and efficiency.

While the current implementation of LayerNAS has shown promising results, several current limitations that can be addressed by future work:

LayerNAS is not designed to solve scale search problems mentioned in Section 3, because many hyper-parameters of model architecture are interdependent in scale search problem, which contradicts the statement in Assumption 4.1.

One-shot NAS algorithms have been shown to be more efficient. We aim to investigate the potential of applying LayerNAS to One-shot NAS algorithms.

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

References

Bender, G., Kindermans, P.-J., Zoph, B., Vasudevan, V., and Le, Q. (2018). Understanding and simplifying one-shot architecture search. In International conference on machine learning , pages 550-559. PMLR.

Bender, G., Liu, H., Chen, B., Chu, G., Cheng, S., Kindermans, P.-J., and Le, Q. V. (2020). Can weight sharing outperform random architecture search? an investigation with tunas. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14323-14332.

Bergstra, J., Yamins, D., and Cox, D. (2013). Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In International conference on machine learning , pages 115-123. PMLR.

Cai, H., Gan, C., Wang, T., Zhang, Z., and Han, S. (2020). Once for all: Train one network and specialize it for efficient deployment. In International Conference on Learning Representations .

Cai, H., Zhu, L., and Han, S. (2018). Proxylessnas: Direct neural architecture search on target task and hardware. In International Conference on Learning Representations .

Chen, W., Gong, X., and Wang, Z. (2021). Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. arXiv preprint arXiv:2102.11535 .

Chrabaszcz, P., Loshchilov, I., and Hutter, F. (2017). A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv:1707.08819 .

Chu, X., Zhang, B., and Xu, R. (2021). Fairnas: Rethinking evaluation fairness of weight sharing neural architecture search. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 12239-12248.

Domhan, T., Springenberg, J. T., and Hutter, F. (2015). Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In Twenty-fourth international joint conference on artificial intelligence .

Dong, X., Liu, L., Musial, K., and Gabrys, B. (2021). Nats-bench: Benchmarking nas algorithms for architecture topology and size. IEEE transactions on pattern analysis and machine intelligence .

Elsken, T., Metzen, J. H., and Hutter, F. (2018). Efficient multi-objective neural architecture search via lamarckian evolution. In International Conference on Learning Representations .

Elsken, T., Metzen, J. H., and Hutter, F. (2019). Neural architecture search: A survey. The Journal of Machine Learning Research , 20(1):1997-2017.

Falkner, S., Klein, A., and Hutter, F. (2018). Bohb: Robust and efficient hyperparameter optimization at scale. In International Conference on Machine Learning , pages 1437-1446. PMLR.

He, Y., Lin, J., Liu, Z., Wang, H., Li, L.-J., and Han, S. (2018). Amc: Automl for model compression and acceleration on mobile devices. In Proceedings of the European conference on computer vision (ECCV) , pages 784-800.

Howard, A., Sandler, M., Chu, G., Chen, L.-C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., Vasudevan, V., et al. (2019). Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1314-1324.

Hsu, C.-H., Chang, S.-H., Liang, J.-H., Chou, H.-P., Liu, C.-H., Chang, S.-C., Pan, J.-Y., Chen, Y.-T., Wei, W., and Juan, D.-C. (2018). Monas: Multi-objective neural architecture search using reinforcement learning. arXiv preprint arXiv:1806.10332 .

Hu, J., Shen, L., and Sun, G. (2018). Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7132-7141.

Huang, S.-Y. and Chu, W.-T. (2021). Searching by generating: Flexible and efficient one-shot nas with architecture generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 983-992.

Jaafra, Y., Laurent, J. L., Deruyver, A., and Naceur, M. S. (2019). Reinforcement learning for neural architecture search: A review. Image and Vision Computing , 89:57-66.

Kandasamy, K., Neiswanger, W., Schneider, J., Poczos, B., and Xing, E. P. (2018). Neural architecture 349 search with bayesian optimisation and optimal transport. Advances in neural information processing systems , 350 31. 351

Krizhevsky, A., Hinton, G., et al. (2009). Learning multiple layers of features from tiny images. 352

Liu, C., Zoph, B., Neumann, M., Shlens, J., Hua, W., Li, L.-J., Fei-Fei, L., Yuille, A., Huang, J., and 353 Murphy, K. (2018a). Progressive neural architecture search. In Proceedings of the European conference on 354 computer vision (ECCV) , pages 19-34. 355

Liu, H., Simonyan, K., and Yang, Y. (2018b). Darts: Differentiable architecture search. In International 356 Conference on Learning Representations . 357

Liu, Y., Sun, Y., Xue, B., Zhang, M., Yen, G. G., and Tan, K. C. (2021). A survey on evolutionary neural 358 architecture search. IEEE transactions on neural networks and learning systems . 359

360

361

362

Liu, Z., Mu, H., Zhang, X., Guo, Z., Yang, X., Cheng, K.-T., and Sun, J. (2019). Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEEE/CVF international conference on computer vision , pages 3296-3305.

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

Mellor, J., Turner, J., Storkey, A., and Crowley, E. J. (2021). Neural architecture search without training. In International Conference on Machine Learning , pages 7588-7598. PMLR.

Pham, H., Guan, M., Zoph, B., Le, Q., and Dean, J. (2018). Efficient neural architecture search via parameters sharing. In International conference on machine learning , pages 4095-4104. PMLR.

Ramachandran, P., Zoph, B., and Le, Q. V. (2017). Searching for activation functions. arXiv preprint arXiv:1710.05941 .

Real, E., Aggarwal, A., Huang, Y., and Le, Q. V. (2019). Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence , volume 33, pages 4780-4789.

Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. (2018). Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4510-4520.

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 .

Shu, Y., Cai, S., Dai, Z., Ooi, B. C., and Low, B. K. H. (2021). Nasi: Label-and data-agnostic neural architecture search at initialization. arXiv preprint arXiv:2109.00817 .

Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., and Le, Q. V. (2019). Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2820-2828.

Tan, M. and Le, Q. (2019). Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning , pages 6105-6114. PMLR.

Wang, K., Liu, Z., Lin, Y., Lin, J., and Han, S. (2019). Haq: Hardware-aware automated quantization with mixed precision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8612-8620.

Wu, B., Dai, X., Zhang, P., Wang, Y., Sun, F., Wu, Y., Tian, Y., Vajda, P., Jia, Y ., and Keutzer, K. (2019). Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10734-10742.

Xiao, H., Wang, Z., Zhu, Z., Zhou, J., and Lu, J. (2022). Shapley-nas: Discovering operation contribution for neural architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11892-11901.

Xu, J., Tan, X., Song, K., Luo, R., Leng, Y., Qin, T., Liu, T.-Y., and Li, J. (2022). Analyzing and mitigating interference in neural architecture search. In International Conference on Machine Learning , pages 24646-24662. PMLR.

Xu, J., Zhao, L., Lin, J., Gao, R., Sun, X., and Yang, H. (2021). Knas: green neural architecture search. In International Conference on Machine Learning , pages 11613-11625. PMLR.

Ying, C., Klein, A., Christiansen, E., Real, E., Murphy, K., and Hutter, F. (2019). Nas-bench-101: Towards reproducible neural architecture search. In International Conference on Machine Learning , pages 7105-7114. PMLR.

You, S., Huang, T., Yang, M., Wang, F., Qian, C., and Zhang, C. (2020). Greedynas: Towards fast one-shot nas with greedy supernet. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1999-2008.

Yu, J. and Huang, T. (2019). Autoslim: Towards one-shot architecture search for channel numbers. arXiv preprint arXiv:1903.11728 .

Zhou, H., Yang, M., Wang, J., and Pan, W. (2019). Bayesnas: A bayesian approach for neural architecture search. In International conference on machine learning , pages 7603-7613. PMLR.

Zhu, Z., Liu, F., Chrysos, G. G., and Cevher, V. (2022). Generalization properties of nas under activation and skip connection search. arXiv preprint arXiv:2209.07238 .

Zoph, B. and Le, Q. V. (2017). Neural architecture search with reinforcement learning. In International Conference on Learning Representations .

Zoph, B., Vasudevan, V., Shlens, J., and Le, Q. V. (2018). Learning transferable architectures for scalable image recognition. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8697-8710. IEEE Computer Society.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces LayerNAS, which is a method for neural architecture search (NAS). The idea is to reduce the computational cost of NAS, which is exponential in the number of layers. As such, the paper introduces a layerwise search option with the idea being that the current layer can be directly determined based on the results of the previous layers. This enables polynomial computational complexity. The paper goes one step further and establishes the inclusion of the cost-constraints, e.g. in eq. 4. LayerNAS is then empirically validated in ImageNet and in a standard benchmark of NAS, which includes cifar10, cifar100 and Imagenet16 datasets. \n\n\n\n**Post rebuttal**: I appreciate the effort by the authors and their numerical evidence. I would urge the authors to include the references and the discussion in the camera-ready version. I would strongly encourage the authors to also include the numerical result on the cost per layer (see answer 2 in the original responses) and the additional results in transfer learning. One thing that I believe should be explained better in the main paper is the reasoning for the separation from larger architectures, e.g. the somewhat arbitrary 600 MAdds.\n\nSTRENGTHS:\nThe reduction of the computational cost of NAS is an important aspect that makes the paper relevant for the NeurIPS community. Besides, a number of papers on NAS are published in NeurIPS and related conferences (**relevance**).  The solution proposed for the layerwise search has appeared before, but the paper makes a complete framework to support the idea and empirically validates the framework. In addition, the paper makes a clear statement of the limitations and the assumptions that led to those (**clarity**). This enables the research community to extend this paper further.\n\nWEAKNESSES:\nI am not sure what the novelty of the proposed method is, this is not clearly stated at the moment. \n\nI find that the train-free methods of NAS are more important for the story and as alternative methods than the two lines devoted to the end of the related work. Having said that, I do understand that they do not cover the contribution of this work.\n\nQUESTIONS:\nOverall, the overall writing is clear to me, but I have few questions: \n\n1) What is the difference in the obtained architectures depending on the cost set as threshold? I think this insight is very important and should be included in the main paper. \n\n2) How is the cost distributed over the layers? Are the first layers more costly in the end or is it uniformly distributed? This might be one of the drawbacks of this heuristic approach of selecting/focusing on one layer at a time. \n\n3) How would the proposed architectures perform on transfer learning? I believe this is an important experiment, since NAS is not sota in ImageNet (current methods surpass the 85% accuracy), BUT NAS can be tremendously useful in transferring to tasks with less annotated data or different tasks.\n\nLIMITATIONS:\nThe limitations are explicitly identified in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-16"}

comments: SUMMARY:
The paper tries to overcome a drawback of Neural Architecture Search (NAS), an enormous search space that hard to traverse whole space to design a well-optimized network.
From an assumption that a previous layer in a network doesn’t affect the subsequent layers, the paper converts multi-objective NAS to a combinatorial optimization problem.
With the proposed method, the paper designs optimized networks layer by layer, unlike other works that design the whole network simultaneously. It leads to reducing the search complexity of NAS to polynomial complexity.

STRENGTHS:
- Writing is easy to follow.

- The paper compares the proposed method with other methods fairly with NATS-Bench.

WEAKNESSES:
- The proposed method that removes networks from search space by their costs is not novel.

- The search cost of the proposed method is still worse than one-shot NAS.

- The paper naively analyzes the cost of searched networks with MAdds, omitting other metrics such as energy consumption or latency, etc.

QUESTIONS:
- Is it sure that searched networks with the proposed method aren’t suboptimal solutions? For example, if the proposed method is applied to every candidate of the first layer, is the network searched with the proposed method from scratch the best solution?

- According to section 5.1, searching the shared base filters first, and sequentially searching kernel sizes and expanded filters leads to decoupling preceding and succeeding layers. Can supports for this statement be provided?

LIMITATIONS:
The paper refers to several limitations of the proposed method in Conclusion and Future Work. However, a thorough analysis is needed in terms of cost, too.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-05

comments: SUMMARY:
The authors propose LayerNAS with polynomial complexity. Namely, this work transforms the multi-objective NAS problem into a Combinatorial Optimization problem with proper assumptions. 
LayerNAS is benchmarked against recent NAS arts on ImageNet classification task, as well as on dedicated NATS-Bench in terms of quality, stability, and efficiency. 
Algorithm details and searched architectures are provided, which might benefit the community.

STRENGTHS:
1. The paper is clearly written and easy to follow. The detailed algorithm and searched architectures are provided, which makes the results replicable. 

2. Extensive results are reported on NASBench-101, and the results are promising. 

3. The limitation of the assumptions are properly discussed. 

4. Performing NAS in polynomial complexity is of good research impact and real-world applications.

WEAKNESSES:
1. There are some format flaws, e.g., the abstract should be a single paragraph. 

2. Table 2 is not informative enough. It would be better to include more details such as training epochs, augmentations, whether distillation is utilized or not, etc. for comparison. Plus, sometimes it is hard to tell whether LayerNAS is better than previous arts when MAdds or Params are not aligned (e.g., LayerNAS has 50% more params than MobileNetV3-Small). I wonder if it is possible to strictly align both params and MAdds and compare accuracy, as anyway, LayerNAS is a multi-objective search. 

3. I wonder if there are latency-driven search results.

QUESTIONS:
Please refer to weaknesses.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
This paper propose a novel approach of breaking down NAS problem into a Combinatorial Optimization problem.

STRENGTHS:
The paper is well-written and easy to follow. The authors provide clear explanations and examples throughout the paper. 

Breaking down the search problem into a Combinatorial Optimization problem seems novel and interesting, and reducing the search cost to polynomial time, which is clearly a breakthrough to the research community. 

LayerNAS can be applied to operation, topology and multi-objective NAS search

Results on ImageNet seems to surpass state-of-the-art methods by a clear margin, evidencing their effectiveness of LayerNAS.

WEAKNESSES:
I do not particular have a question, this paper seems to be easy enough to follow.

QUESTIONS:
No

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08