Title: DensEMANN: How to Automatically Generate an Efficient while Compact DenseNet
Venue: NeurIPS 2023
Accepted: no
Keywords: neural network

convolutional neural network

neural architecture search

NAS

neural architecture optimization

constructive algorithms

growing

pruning

DenseNet

EMANN

in-supervised

self-structuring

connection strength
Abstract: We present a new and improved version of DensEMANN, an algorithm that grows small DenseNet architectures virtually from scratch while simultaneously training them on target data. Following a finite-state machine based on the network's accuracy and the evolution of its weight values, the algorithm adds and prunes dense layers and convolution filters during training only when this leads to significant accuracy improvement. We show that our improved version of DensEMANN can quickly and efficiently search for small and competitive DenseNet architectures for well-known image classification benchmarks. In half a GPU day or less, this method generates networks with under 500k parameters and between 93% and 95% accuracy on various benchmarks (CIFAR-10, Fashion-MNIST, SVHN). For CIFAR-10, we show that it comes very close to the state-of-the-art Pareto front between accuracy and size, finding networks with 98.84% of the accuracy and 98.08% of the size of the closest Pareto-optimal competitor, in only 0.70% of the search time it took to find that competitor. We also show that DensEMANN generates its networks with optimal weight values, and identify a simple mechanism that allows it to generate such optimal weights. All in all, we show this "in-supervised" essentially incremental approach to be promising for a fast design of competitive while compact convolution networks.

Abstract

We present a new and improved version of DensEMANN, an algorithm that grows small DenseNet architectures virtually from scratch while simultaneously training them on target data. Following a finite-state machine based on the network's accuracy and the evolution of its weight values, the algorithm adds and prunes dense layers and convolution filters during training only when this leads to significant accuracy improvement. We show that our improved version of DensEMANN can quickly and efficiently search for small and competitive DenseNet architectures for well-known image classification benchmarks. In half a GPU day or less, this method generates networks with under 500k parameters and between 93% and 95% accuracy on various benchmarks (CIFAR-10, Fashion-MNIST, SVHN). For CIFAR10, we show that it comes very close to the state-of-the-art Pareto front between accuracy and size, finding networks with 98.84% of the accuracy and 98.08% of the size of the closest Pareto-optimal competitor, in only 0.70% of the search time it took to find that competitor. We also show that DensEMANN generates its networks with optimal weight values, and identify a simple mechanism that allows it to generate such optimal weights. All in all, we show this 'in-supervised' essentially incremental approach to be promising for a fast design of competitive while compact convolution networks.

1 Introduction

The architecture of a neural network (NN) is known to have a great impact on its performance on a target task-on par with that of the training process through which the NN learns the task [1, 2, 3, 4, 5]. The main motivation behind neural architecture search (NAS) is precisely to find the most adequate architecture for a task, in the sense of achieving the highest accuracy [6, 7, 8], but also of using resources as efficiently as possible [7, 9, 8]. To this aim, NAS algorithms must compare the performance a great number of candidate architecture designs. Since naïvely training all of them from scratch would be very inefficient, the NAS community has put much effort into developping reliable and ressource-efficient performance estimation strategies [6, 7, 2, 3, 8].

In this respect, so-called 'growing', 'constructive' or 'incremental' algorithms provide an interesting approach to NAS and performance estimation. They simultaneously build and train candidate architectures, by adding elements such as weights, neurons, layers etc. during the training process [10, 11, 12, 13]. Since new candidate networks are evaluated on basis of the weights learned by previous candidates [14, 15], the time and computation resources consumed by the entire NAS process are equivalent to those required for training a single NN [12, 14]. Furthermore, the search space is not bounded, as new elements may be added ad infinitum [13, 16].

This paper presents our research on DensEMANN [1], an algorithm that simultaneously grows and 35 trains small and efficient DenseNets [17] virtually from scratch. Encouraged by previous positive 36

results found by its authors [1, 18], and by the great success of other similar methods [13, 19], we 37 created a new version of this algorithm with the aim of approaching state-of-the-art performance 38 for well-known benchmarks, or at least the state-of-the-art Pareto front between performance and 39 model size. As a secondary goal, we tested the authors' claim that DensEMANN-generated networks 40 perform equally or better than similar NN even when these are trained from scratch [1]. 41

Section 2 provides some background on growing-based NAS and related research. Section 3 contains 42 a presentation of DensEMANN's inner workings 3.1 and lists our modifications with regards to [1] 43 3.2. Section 4 presents our experiments and their results, and Section 5 includes our conclusions and 44 suggestions for future research. 45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

2 Rediscovery of an incremental approach

Most likely [18, 13, 20], the first growing-based NAS algorithm was Dynamic Node Creation (DNC) [21] or the cascade-correlation algorithm (CC-Alg) [22]. During the 1990's, the research field of 'constructive' algorithms (as they were called) was so active that at least two contemporary surveys exist of this field [23, 24]. To our knowledge, research on growing-based NAS for convolution neural networks (CNN) only took off after the introduction of Net2Net operators [25] and network morphisms [26], which can instantly make a CNN wider or deeper while not changing its behaviour.

We have observed a recurring pattern of rediscovery , or convergence , between early growing techniques and more recent ones. Parallels can be made, for instance, between some network morphisms [25, 26] and early node-splitting techniques [27], or between the 'backwards steps' in some modern algorithms [19, 28] and early growing-pruning hybridations [24, 23]. We believe that this convergence is helped by a direct correspondence (described in [25, 29]) between pioneering neural architectures such as multi-layer perceptrons, and more recent ones like CNN: perceptron layers correspond to convolutional layers, and perceptron neurons correspond to 3D convolution filters.

The most serious competitor to growing-based NAS algorithms are trainless or zero-cost algorithms [30, 31, 2]. These evaluate candidate NN on basis of their performance with random weights. Such methods can explore large search spaces in a matter of minutes or even seconds [31, 2]. However, extra time is still needed for training the final candidate architecture in order to use it.

3 DensEMANN: building a DenseNet one filter at a time

DensEMANN [1] is a growing algorithm that simultaneously builds and trains DenseNets [17] virtually from scratch. It is based on EMANN [16], an algorithm that grows multi-layer perceptrons with an analogous connection scheme to that of DenseNet, and on previous research on DenseNetgrowing techniques by the same authors [18]. Based on an introspective 'self-structuring' or 'in-supervised' approach, much more in line with real neurology than purely performance-based NAS, it grows and prunes the candidate NN on basis of the evolution of its internal weight values.

3.1 General presentation of DensEMANN

By default, DensEMANN's seed architecture is a DenseNet containing a single dense block, inside which there is a single dense layer producing k = 12 feature maps. 'Dense layers' may either be DenseNet or DenseNet-BC composite functions, with the same characteristics as in [17]. Also like in [17], the DenseNet's inputs are pre-processed by an initial convolution layer with 2 ∗ k = 24 filters, and its final outputs are generated through 2D batch normalization [32] and a fully connected layer.

Paralleling EMANN's 'double level adaptation' [16], DensEMANN consists of two components, the macro-algorithm and the micro-algorithm [1]. Each of them builds the seed network's dense block at a different granularity level. Afterwards, the dense block may be replicated a certain user-set number of times to produce an N -block DenseNet (see Section 3.1.3).

3.1.1 The macro-algorithm

The macro-algorithm works at the level of dense layers, and is reminiscent of CC-Alg [22]. It 82 iteratively stacks up dense layers in the block until there is no significant change in the accuracy (see 83 Algorithm 1 for its pseudocode). 84

Algorithm 1 DensEMANN macro-algorithm

1: procedure MACROALGORITHM( model ) 2: accuracy last ← 0 3: model last ← model 4: model, accuracy ← MICROALGORITHM ( model ) 5: while | accuracy -accuracy last | ≥ IT do 6: accuracy last ← accuracy 7: model last ← model 8: model ← ADDNEWLAYER ( model ) 9: model, accuracy ← MICROALGORITHM ( model 10: end while 11: return model last

12: end procedure

Each new layer is created with the same initial number of 3D convolution filters, set by a growth rate 85 parameter (by default k = 12 ). In the case of DenseNet-BC, the dense layer's first convolution is 86 created with 4 ∗ k filters, and its second convolution with k filters. 87

88

89

90

91

92

93

Before each layer addition, the macro-algorithm saves the current NN model (architecture and weights) and its accuracy. It then adds the new layer and calls the micro-algorithm to build it. Once the micro-algorithm finishes, the macro-algorithm compares the current accuracy to the one before the new layer was added. If the absolute difference between the two accuracies surpasses an improvement threshold (by default IT = 0 . 01 ), the macro-algorithm loops and creates a new layer. Otherwise, the algorithm undoes the layer's addition by loading back the last saved model, and stops there.

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

3.1.2 The micro-algorithm

The micro-algorithm works at the level of convolution filters. It operates only in the dense block's last layer (for DenseNet-BC, the second convolution in the last dense layer), and follows a finite-state machine with states for growing, pruning, and performance recovery.

While the network is trained through standard backpropagation, the micro-algorithm establishes different categories of filters on basis of their kernel connection strength (kCS). For filter λ , its kCS is the arithmetic mean of its absolute weight values w 1 , ..., w n .

A filter is declared 'settled' if its kCS remains near-constant for the last 40 training epochs. 1 After at least k / 2 filters have settled, settled filters can be declared 'useful' if their average kCS over the last 10 epochs falls above a usefulness threshold (UFT), and 'useless' if that same value falls below a uselessness threshold (ULT). During the micro-algorithm's improvement stage (1), the UFT and ULT are recalculated after each training epoch on basis of the maximum and minimum kCS among settled filters, and of user-settable parameters UFT auto and ULT auto (by default respectively 0 . 8 and 0 . 2 ):

The micro-algorithm uses these three filter categories-settled, useful and useless-as references for building the network's last layer. To do this, it alternates between three stages (see Figure 1):

Improvement : the network starts training with initial learning rate LR 0 = 0 . 1 , and filters are progressively declared settled, then useful or useless. Meanwhile, a countdown begins with a fixed length in training epochs: the patience parameter (by default PP = 40 epochs).

)

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

All the filters are

settled

AND

At least one filter is

useless

?

Yes

No

LR

= 0.001 *

LR 0

AND

accuracy

pre-pruning_accuracy

All or none of the

filters are

useless

Pruning

PP

countdown is over

AND

All the filters are

settled

Recovery

Pruning operation

was successful

End

New layer is created

Undo

pruning

and end

Recovery stage's duration

exceeds

PP re

epochs

Improvement

The learning rate (LR) is divided by 10 after 50% and 75% of this countdown has elapsed. If at any point during this stage the number of useful filters exceeds its last maximum value, a new filter is added and the countdown and LR are reset. The stage ends when (1) the countdown is over and (2) all filters have settled. The next stage is always 2 (pruning).

Pruning : if all or none of the layer's filters are useless, the micro-algorithm ends here. Otherwise, the micro-algorithm saves the NN model and its accuracy (like the macroalgorithm does before creating a new layer), deletes all useless filters, and moves to stage 3 (recovery). From the first pruning stage onwards, the UFT and ULT values are frozen.

Recovery : the network is trained again, with the same PP -epoch countdown and the same initial and scheduled LR values as in stage 1 (improvement), but without filter additions. There are two additional countdowns, one with the same length as the last improvement stage ( PP + the number of epochs it took for all filters to settle), and another one with a length of PP re > PP epochs (by default PP re = 130 ). Three things may happen:

(a) If (1) the learning rate has already reached its lowest scheduled value (i.e. in practice after 0 . 75 ∗ PP epochs) and (2) the current accuracy has reached or surpassed its pre-pruning value, the stage ends. If at this point (3) all the filters have settled and (4) there is at least one useless filter, the next stage is 2 (pruning). Otherwise, the micro-algorithm ends.

(b) If the stage's duration exceeds PP re epochs, the previous pruning operation is considered 'fruitless' and undone. The pre-pruning model is loaded back, and the microalgorithm ends.

(c) If the stage's duration exceeds that of the previous improvement stage, the filters' kCS values are considered 'frozen'. In practice, this means that all filters are declared settled at most 40 epochs after this point, and that the frozen kCS values will be used as the reference for any subsequent pruning.

DensEMANN's weight initialization mechanisms are also worth commenting:

The weights for new layers are initialized along a truncated normal distribution, similar to that of TensorFlow v1's [33] 'variance scaling initializer'. For this initializer, the distribution's standard deviation (SD) is usually inversely proportional to each layer's number of input features and to its filters' dimensions. However, for layers that the microalgorithm can act upon, the distribution's SD only depends on the filters' dimensions, resulting in bigger initial weights in these layers.

The weights of new filters are initialized using a special complementarity mechanism borrowed from EMANN [16]. The weights' absolute values are random, and follow the same truncated normal distribution as weights in new modifiable layers. Their sign configuration, however, is not random: it is the inverted sign configuration of the filter with the lowest kCS. This is done to ensure the mutual complementarity and co-adaptation of new and old filters.

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

3.1.3 Building more than one dense block

DensEMANN can also be set to build DenseNets with a user-set number of dense blocks N . To do this, DensEMANN first uses the macro- and micro-algorithms to build a one-block DenseNet, and then replicates the generated dense block N -1 times to create a N -block DenseNet.

Between blocks, transition layers are created with a similar architecture to that in [17], i.e. with a batch normalisation [32], a ReLU function [34], a 1x1 convolution and a 2x2 average pooling layer [35]. The number of filters in the 1x1 convolution depends on whether the network is a DenseNet or a DenseNet-BC: for DenseNet it is the same as the previous block's number of output features, while for DenseNet-BC it is multiplied by a reduction factor, by default θ = 0 . 5 .

The weights for the N -1 new blocks are initialized using the same method that DensEMANN uses for new layers. 2 After the new blocks are added, the NN is trained for 300 extra epochs. The LR recovers its initial value LR 0 at the beginning of these last 300 epochs, and is divided by 10 on epochs 150 and 255 (i.e., 50% and 75% through the extra training epochs). During these epochs, DensEMANN adopts a 'best model saving' approach: the NN's weights are saved whenever its loss reaches a new minimum value, and after the 300 epochs, these 'best' weights are loaded back to allow the NN to reach optimal performance.

Although it is in direct contrast with DensEMANN's incremental philosophy, this method for replicating the generated block N -1 times is activated by default, with N = 3 . Its development was motivated by previous experimental results with mechanisms that copy DensEMANN-generated layers a predefined number of times, and by the good performance of cell-based NAS approaches [13, 14, 36] that first search for a small neural pattern (the cell) and then replicate it N times. In Appendix A, we give the results of an ablation study that compares, among others, DensEMANN's performance with and without this dense block replication mechanism.

3.2 Differences with the original DensEMANN

Below are the differences between our version of the algorithm and the one described in [1]:

Changes to the macro-algorithm:

(a) The last layer addition is always undone, as it does not fulfil the accuracy improvement criterion. This was suggested in [1], and EMANN uses a similar mechanism [16].

(b) The improvement threshold's default value was changed to IT = 0 . 01 . Observations in [1] suggest that, with the previous default value ( 0 . 005 ), the last few layer additions do not have a big impact in the NN's final accuracy.

Changes to the micro-algorithm:

(a) Only settled filters may be declared useful or useless. This was proposed in [1] as a means to avoid quick cascades of often superfluous filter additions.

(b) The patience parameter's default value was changed to PP = 40 epochs. Observations in [1] suggest that it takes approximately 40 epochs for all the filters in a layer to settle.

(c) If at any point during the improvement stage the number of useful filters exceeds its last highest value, a new filter is added and the patience countdown is reset. In v1.0, this can only happen if the countdown has not yet ended.

(d) The pruning and recovery stages have been heavily modified to avoid long recovery stages and their effects. We indeed observed that the kCS of settled filters is not constant but actually decreases very slowly over time. If the recovery stage is too long, this causes a very harsh pruning after which the accuracy cannot be recovered.

We added a method that replicates the generated dense block N times.

4 Experiments

We re-implemented DensEMANN from scratch using the PyTorch (v1.8 or higher 3 ) [37] and Fastai 194 (v2.5.3) [38] Python libraries. Our code is based on the DenseNet implementation for PyTorch by 195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

Pleiss et al. [39] and on the original DensEMANN implementation by García-Díaz [40] (both under MIT license). We initially replicated the latter as faithfully as possible, including the unusual weight initialization described in Section 3.1.2. Then, we made the modifications described in this paper.

In our experiments we use three well-known image classification benchmarks: CIFAR-10 [41], Fashion-MNIST [42] and SVHN [43]. For each training process (either when running DensEMANN or training NN from scratch), we split the target data into three parts:

Training set: a random set of examples, different for each training process. It is used for training the NN. For CIFAR-10 and Fashion-MNIST it contains 45,000 'training' images, and for SVHN it contains 6,000 images from the 'train' and 'extra' sets.

Validation set: another random set of examples, different for each training process but separate from the training set (there is no overlap between the two sets). It is used for estimating the NN's accuracy and loss during training (and in the case of DensEMANN, during growing). For CIFAR-10 and Fashion-MNIST it contains 5,000 'training' images, and for SVHN it contains 6,000 images from the 'train' and 'extra' sets.

Test set: a predefined set of examples that the NN never 'sees' during training. It is used for evaluating the NN's final performance (accuracy and cross-entropy loss), and to compare it against the state of the art. It is the entire set of 'test' images provided by each dataset's authors: 10,000 images for CIFAR-10 and Fashion-MNIST, and 26,032 images for SVHN.

A batch size of 64 images is used for all datasets, and for all three of the above splits.

Our data pre-processing workflow is as follows:

Random crop with 4-pixel padding + random horizontal flip (as in [17]), only for CIFAR-10's training and validation data.

Normalization. For CIFAR-10 we use the dataset's channel-wise mean and SD values as in [17]. For the other two datasets we assume mean and SD values of 0.5 for all channels.

Cutout regularization [44], only for the training and validation data.

DensEMANN's parameters were set to their default values: IT = 0 . 01 , PP = 40 , PP re = 130 , LR 0 = 0 . 1 , k = 12 for the first layer, N = 3 blocks in the final network. All other default values are the same as in [1]. We opted to generate DenseNet-BC architectures, as in past research they provided better results than standard DenseNet [1, 18, 17].

For our experiments, we used the following computation environments:

MSi GT76 Titan DT laptop: Windows 10 Pro (64-bit) OS, Intel Core i9-10900K CPU (3.70 GHz), NVIDIA GeForce RTX 2080 Super GPU, 64.0 GB RAM (63.9 GB usable). Python is v3.9.8, PyTorch is v1.10.0+cu113.

Internal cluster: Linux Ubuntu 20.04.4 LTS (x86-64) OS, 16 AMD EPYC-Rome Processor CPUs (2.35 GHz), NVIDIA GeForce RTX 3090 GPU, 64 GB RAM. Python is v3.8.6, PyTorch is v1.8.1.

In Table 1, GPU times in black were obtained with the MSi GT76, while GPU times in italized purple were obtained on the internal cluster. We consider the times obtained on the MSi GT76 to be more reliable, as on the internal cluster we have let up to four tests run at a time, whereas on the MSi GT76 we have only run one test at a time.

The total computation time for all the experiments in this paper (excluding the appendices) was 6.49 GPU days (3.61 days on the MSi GT76 and 2.88 days on the internal cluster). Below are the computation times for each experiment:

4.1: 4.88 GPU days (2.81 on the MSi GT76, 2.07 on the cluster).

4.2: 1.61 GPU days (0.80 on the MSi GT76, 0.80 on the cluster).

4.1 DensEMANN's full potential unlocked

We began by running DensEMANN 5 times for each dataset, in order to get an idea of the algorithm's performance. The results of this experiment correspond to the two first lines for each dataset in

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

Table 1: the 'average performance' lines contains the mean and SD over the 5 runs for each variable, whereas the 'best network' line corresponds to the DensEMANN-generated NN that obtained the lowest (cross-entropy) loss on the validation set. In the latter case, we indicate two execution times: the execution time for the run that generated this 'best' NN, and the total execution time for all 5 runs of DensEMANN (i.e. the total GPU time that we consumed to search for this optimal candidate).

All in all, DensEMANN performs very well on all three benchmarks. The generated architectures are always under 0.5 million parameters (in the case of Fashion-MNIST they are even under 0.1 million parameters), yet the average test set accuracies are all between 93% and 95%. The current state of the art test set accuracies on CIFAR-10 [45] and SVHN [46] are at 99% or higher, while that on Fashion-MNIST [47] is at just under 97%. This said, the top-performing models for these benchmarks are very large, containing several millions of parameters.

Concerning DensEMANN's execution times, they range from around 3 hours (SVHN) to just over half a day (CIFAR-10). Consequently, it always took us less than 3 days to run DensEMANN 5 times, and find our best candidate network for all benchmarks.

It is noteworthy that DensEMANN does seem to build minimal architectures that adapt to each dataset's peculiarities. For Fashion-MNIST, a grayscale dataset with smaller images than the other two datasets, DensEMANN generated very small and shallow architectures-an order of magnitude smaller than those for the two other datasets. Meanwhile, the biggest and deepest NN were generated for SVHN, an RGB dataset whose images contain distractors around the main data to classify.

4.2 Retraining our best networks from scratch: DensEMANN vs. 'perfect' NAS

For our second experiment, we erased the weights in DensEMANN's best network for each dataset, replaced them with randomly initialized weights (using an exact copy of TensorFlow's 'variance scaling initializer') and trained the network from scratch 5 times for 300 epochs. We used the same workflow as for the block replication mechanism at the end of DensEMANN: beginning with a LR value of LR 0 = 0 . 1 , we divide it by 10 on epochs 150 and 255. We also use the same 'best model saving' approach, where we save the weight values that produce the lowest validation set loss, and load them back at the end of the training process. The results of this experiment correspond to the 'best network retrained' lines in Table 1.

This experiment is useful for two reasons. On one hand, it allows us to test the claims in [1] that DensEMANN generates its networks with optimal weights. If that were the case, the network's performance (accuracy and loss) with the original and retrained weights would not be very different. On the other hand, it can be used for comparing DensEMANN's execution times to those of a hypothetical 'perfect' NAS algorithm. As explained in Section 2, even if we imagine a 'perfect' zero-cost NAS algorithm that can instantly find an optimal network in DensEMANN's search space, one still needs to train the candidate network before using it. For this reason, we consider the GPU time cost of retraining DensEMANN's final architecture to be a good proxy for the GPU time cost of using such a 'perfect' NAS algorithm to explore DensEMANN's search space.

Concerning the network's performance, the difference before and after being retraining is not very big, and this is true on both the validation and test set. In the case of CIFAR-10, there is an improvement for the accuracy, but the loss does not seem significantly different. One-sample T-tests confirm this observation: the only statistically significant differences are found for CIFAR-10's validation and test set accuracies (respectively P = 0 . 039 and P = 0 . 010 ), and for SVHN's test set accuracy ( P = 0 . 044 ). In the former case the accuracy improved after retraining, while in the latter case it

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

worsened after retraining. Nevertheless, in both cases the test loss does not change significantly, which leads us to conclude that DensEMANN does optimally train the architectures that it grows.

Concerning the GPU times, for all datasets there is a significant difference between DensEMANN's execution times (both the average time and that of the best run) and the average time cost for retraining the best run's final candidate NN. The mean GPU time cost of DensEMANN is 3.49 times longer than that of retraining for CIFAR-10, 2.33 times longer for Fashion-MNIST, and 3.25 times longer for SVHN. Part of this difference most certainly comes from the 300 extra training epochs required by DensEMANN's block replication mechanism-exactly the same number of epochs that we use to retrain the best generated NN from scratch. Since for two similar architectures 300 training epochs will represent a similar GPU time, it is mathematically impossible for DensEMANN to outperform 'perfect' NAS' time cost when using block replication. Nevertheless, the GPU time costs of DensEMANN and SGD training remain of the same order of magnitude (hours).

4.3 Comparison against the state of the art

In Table 2 and Figure 2, we take advantage of the widespread use of CIFAR-10 as a benchmark task to map the state of the art for the compromise between model size and error rate on this dataset, and see where DensEMANN fits with regards to that state of the art. Concretely, we use Figure 2 to visualize the current Pareto front for the size vs. error rate compromise. We compare DensEMANN to this Pareto front by identifying the NN models and NAS algorithms that make up this front, and by focusing on the ones that are closest to DensEMANN's performance.

The closest Pareto-optimal competitor to DensEMANN is LEMONADE S-I [19], another growing-based algorithm that is designed specifically to explore the Pareto front between model acccuracy and size. After 80 GPU days LEMONADE S-I generated a final candidate architecture with 190 thousand parameters that obtained 94.5% accuracy on CIFAR-10. DensEMANN's average performance is very close to this: with 98.08% of its size, we reach 98.84% of the LEMONADE S-I network's accuracy, in 0.70% of the GPU time that LEMONADE took to find that network.

Concerning the execution time, a closer competitor to DensEMANN is NASH Random, by

0

2

4

6

8

10

12

0.01

0.1

1

10

Error rate on CIFAR-10 (%)

Trainable parameters (millions)

Human-designed

Reinforcement learning (RL)

Evolutionary and genetic algorithms (EA and GA)

Gradient-based optimization (GO)

Growing / Forward NAS

Random Search

Ours (DensEMANN, average performance)

Ours (DensEMANN, best network)

Ours (DensEMANN, best network retrained)

the same authors as LEMONADE [14]. On average, it takes 0.19 GPU days to produce its final candidate networks, their average size is 4.4 million parameters, and their average accuracy on CIFAR-10 is 93.50%. DensEMANN reaches a similar accuracy with 4.24% of the size, but its average execution time is 2.96 times longer. It is likely that network morphisms [25, 26], which NASH uses for preserving the NN's behaviour after growing, are in part responsible for its quick execution time.

5 Conclusions and future work

We present DensEMANN, an 'in-supervised' growing-based NAS algorithm that simultaneously 335 builds and trains DenseNet architectures for target tasks. We show that, in half a GPU day or less, 336 DensEMANN can generate very small networks (under 500 thousand trainable parameters) for 337 various benchmark image classification tasks, training them with optimal weights that allow them 338 to reach around 94% accuracy on these benchmarks. For one of them (CIFAR-10), we show that 339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

DensEMANN's performance is very close to state-of-the-art Pareto-optimality for the compromise between accuracy and neural architecture size. This said, by studying DensEMANN in detail and comparing it to other algorithms in the literature, we find methodologies-such as network morphisms and best model saving-that could make this approach even quicker and more optimal.

Future research on improving DensEMANN could follow some of the following research lines:

Developing a zero-cost NAS algorithm that quickly explores DensEMANN's search space. This should become the baseline for evaluating future DensEMANN-inspired algorithms.

Further incorporating best model saving into DensEMANN: during the micro-algorithm's improvement and recovery stages, save the weights-and the model-that correspond to the best validation loss since the start of the stage, then reload them at the end of the stage. This could double as a method for controlling the true usefulness of filter additions.

Comparing different ways to initialize new filters or layers, such as network morphisms [25, 26] and GradMax [10].

Replacing the block replication method with an improved macro-algorithm, that can decide when it is more convenient to start a new block or to just add a new layer in the current one.

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

References

Antonio García-Díaz and Hugues Bersini. DensEMANN: building a DenseNet from scratch, layer by layer and kernel by kernel. In 2021 International Joint Conference on Neural Networks (IJCNN) , pages 1-10, 2021. doi: 10.1109/IJCNN52387.2021.9533783. URL https://difusion.ulb.ac.be/vufind/Record/ULB-DIPOT:oai:dipot.ulb.ac. be:2013/335727/Details .

Joe Mellor, Jack Turner, Amos Storkey, and Elliot J Crowley. Neural architecture search without training. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 7588-7598. PMLR, 2021. URL https://proceedings.mlr.press/v139/ mellor21a.html .

Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. A comprehensive survey of neural architecture search: Challenges and solutions. ACM Comput. Surv. , 54(4), 2021. ISSN 0360-0300. doi: 10.1145/3447582. URL https: //doi.org/10.1145/3447582 .

Yanan Sun, Bing Xue, Mengjie Zhang, Gary G. Yen, and Jiancheng Lv. Automatically designing CNN architectures using the genetic algorithm for image classification. IEEE Transactions on Cybernetics , 50(9):3840-3854, 2020. doi: 10.1109/TCYB.2020.2983860. URL https: //ieeexplore.ieee.org/document/9075201 .

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL https://arxiv.org/abs/1409.1556v6 .

Jeon-Seong Kang, JinKyu Kang, Jung-Jun Kim, Kwang-Woo Jeon, Hyun-Joon Chung, and Byung-Hoon Park. Neural architecture search survey: A computer vision perspective. Sensors , 23(3), 2023. ISSN 1424-8220. doi: 10.3390/s23031713. URL https://www.mdpi.com/ 1424-8220/23/3/1713 .

Dilyara Baymurzina, Eugene Golikov, and Mikhail Burtsev. A review of neural architecture search. Neurocomputing , 474:82-93, 2022. ISSN 0925-2312. doi: https://doi.org/10.1016/j. neucom.2021.12.014. URL https://www.sciencedirect.com/science/article/pii/ S0925231221018439 .

Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research , 20(55):1-21, 2019. URL http://jmlr.org/papers/ v20/18-598.html .

Hadjer Benmeziane, Kaoutar El Maghraoui, Hamza Ouarnoughi, Smail Niar, Martin Wistuba, and Naigang Wang. Hardware-aware neural architecture search: Survey and taxonomy. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21 , pages 4322-4329. International Joint Conferences on Artificial Intelligence Organization, 2021. doi: 10.24963/ijcai.2021/592. URL https://doi.org/10.24963/ijcai.2021/592 . Survey Track.

Utku Evci, Bart van Merrienboer, Thomas Unterthiner, Fabian Pedregosa, and Max Vladymyrov. GradMax: growing neural networks using gradient information. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=qjN4h_wwUO .

Xin Yuan, Pedro Henrique Pamplona Savarese, and Michael Maire. Growing efficient deep networks by structured continuous sparsification. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=wb3wxCObbRT .

Wei Wen, Feng Yan, Yiran Chen, and Hai Li. AutoGrow: automatic layer growing in deep convolutional networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , KDD '20, page 833-841, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450379984. doi: 10.1145/3394486.3403126. URL https://doi.org/10.1145/3394486.3403126 .

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

Hanzhang Hu, John Langford, Rich Caruana, Saurajit Mukherjee, Eric Horvitz, and Debadeepta Dey. Efficient forward architecture search. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché Buc, Edward A. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada , pages 10122-10131, 2019. URL http://papers.nips.cc/paper/ 9202-efficient-forward-architecture-search .

Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Simple and efficient architecture search for convolutional neural networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Workshop Track Proceedings . OpenReview.net, 2018. URL https://openreview.net/forum?id=H1hymrkDf .

Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transformation. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence , AAAI'18/IAAI'18/EAAI'18. AAAI Press, 2018. ISBN 978-1-57735-800-8. URL https://dl.acm.org/doi/abs/10.5555/3504035.3504375 .

T. Salome and H. Bersini. An algorithm for self-structuring neural net classifiers. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94) , volume 3, pages 13071312 vol.3, 1994. doi: 10.1109/ICNN.1994.374473. URL https://ieeexplore.ieee.org/ document/374473 .

Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017. URL https://openaccess.thecvf.com/content_cvpr_ 2017/papers/Huang_Densely_Connected_Convolutional_CVPR_2017_paper.pdf .

Antonio García-Díaz and Hugues Bersini. Self-optimisation of dense neural network architectures: An incremental approach. In 2020 International Joint Conference on Neural Networks (IJCNN) , pages 1-8, 2020. doi: 10.1109/IJCNN48605.2020.9207416. URL https://difusion.ulb.ac.be/vufind/Record/ULB-DIPOT:oai:dipot.ulb.ac. be:2013/335725/Details .

Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Efficient multi-objective neural architecture search via lamarckian evolution. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https://openreview.net/forum?id=ByME42AqK7 .

Rami M. Mohammad, Fadi Thabtah, and Lee McCluskey. An improved self-structuring neural network. In Huiping Cao, Jinyan Li, and Ruili Wang, editors, Trends and Applications in Knowledge Discovery and Data Mining , pages 35-47, Cham, 2016. Springer International Publishing. ISBN 978-3-319-42996-0. URL https://link.springer.com/chapter/10. 1007/978-3-319-42996-0_4 .

Timur Ash. Dynamic node creation in backpropagation networks. Connection Science , 1 (4):365-375, 1989. doi: 10.1080/09540098908915647. URL https://doi.org/10.1080/ 09540098908915647 .

Scott Fahlman and Christian Lebiere. The cascade-correlation learning architecture. In D. Touretzky, editor, Advances in Neural Information Processing Systems , volume 2. MorganKaufmann, 1989. URL https://proceedings.neurips.cc/paper_files/paper/1989/ file/69adc1e107f7f7d035d7baf04342e1ca-Paper.pdf .

Tin-Yau Kwok and Dit-Yan Yeung. Constructive algorithms for structure learning in feedforward neural networks for regression problems. IEEE Transactions on Neural Networks , 8(3):630-645, 1997. doi: 10.1109/72.572102. URL https://ieeexplore.ieee.org/document/572102 .

Sam Waugh. Dynamic learning algorithms. 1994. URL https: //citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi= bd69827d03d6f486ca0ef257f760d028ab076b5d .

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

Tianqi Chen, Ian J. Goodfellow, and Jonathon Shlens. Net2Net: Accelerating learning via knowledge transfer. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings , 2016. URL https://arxiv.org/abs/1511.05641v4 .

Tao Wei, Changhu Wang, Yong Rui, and Chang Wen Chen. Network morphism. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning , volume 48 of Proceedings of Machine Learning Research , pages 564-572, New York, New York, USA, 2016. PMLR. URL https://proceedings.mlr.press/v48/ wei16.html .

Mike Wynne-Jones. Node splitting: A constructive algorithm for feed-forward neural networks. In J. Moody, S. Hanson, and R.P. Lippmann, editors, Advances in Neural Information Processing Systems , volume 4. Morgan-Kaufmann, 1991. URL https://proceedings.neurips.cc/ paper_files/paper/1991/file/0fcbc61acd0479dc77e3cccc0f5ffca7-Paper.pdf .

Shizuma Namekawa and Taro Tezuka. Evolutionary neural architecture search by mutual information analysis. In 2021 IEEE Congress on Evolutionary Computation (CEC) , pages 966-972, 2021. doi: 10.1109/CEC45853.2021.9504845. URL https://ieeexplore.ieee. org/document/9504845/ .

Lemeng Wu, Dilin Wang, and Qiang Liu. Splitting steepest descent for growing neural architectures. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/ 2019/file/3a01fc0853ebeba94fde4d1cc6fb842a-Paper.pdf .

Mohamed S Abdelfattah, Abhinav Mehrotra, Łukasz Dudziak, and Nicholas Donald Lane. Zerocost proxies for lightweight NAS. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=0cmMMy8J5q .

Ekaterina Gracheva. Trainless model performance estimation based on random weights initialisations for neural architecture search. Array , 12:100082, 2021. ISSN 2590-0056. doi: https://doi.org/10.1016/j.array.2021.100082. URL https://www.sciencedirect.com/ science/article/pii/S2590005621000308 .

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37 , ICML'15, page 448-456. JMLR.org, 2015. URL https://arxiv.org/abs/1502.03167v3 .

Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: a system for largescale machine learning. In Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation , OSDI'16, page 265-283, USA, 2016. USENIX Association. ISBN 9781931971331. URL https://www.usenix.org/system/files/conference/osdi16/ osdi16-abadi.pdf .

Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Geoffrey Gordon, David Dunson, and Miroslav Dudík, editors, Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , volume 15 of Proceedings of Machine Learning Research , pages 315-323, Fort Lauderdale, FL, USA, 2011. PMLR. URL https://proceedings.mlr.press/v15/glorot11a.html .

Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998. doi: 10.1109/5.726791. URL http://vision.stanford.edu/cs598_spring07/papers/Lecun98.pdf .

Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transferable architectures for scalable image recognition. In Proceedings of

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018. URL https://openaccess.thecvf.com/content_cvpr_2018/html/Zoph_Learning_ Transferable_Architectures_CVPR_2018_paper.html .

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: an imperative style, highperformance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'AlchéBuc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_ files/paper/2019/file/bdbca288fee7f92f2bfa9f7012727740-Paper.pdf .

Jeremy Howard and Sylvain Gugger. Fastai: A layered api for deep learning. Information , 11 (2), 2020. ISSN 2078-2489. doi: 10.3390/info11020108. URL https://www.mdpi.com/ 2078-2489/11/2/108 .

Geoff Pleiss, Danlu Chen, Gao Huang, Tongcheng Li, Laurens van der Maaten, and Kilian Q. Weinberger. Memory-efficient implementation of densenets. 2017. URL https://arxiv. org/abs/1707.06990v1 .

Antonio García-Díaz. DensEMANN: self-constructing DenseNet with TensorFlow. URL https://github.com/AntonioGarciaDiaz/Self-constructing_DenseNet .

Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009. URL https: //www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf .

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms, 2017. URL https://arxiv.org/abs/1708. 07747v2 .

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. 2011. URL http: //ufldl.stanford.edu/housenumbers/nips2011_housenumbers.pdf .

Terrance Devries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout. CoRR , abs/1708.04552, 2017. URL https://arxiv.org/abs/1708. 04552v2 .

Papers with code -CIFAR-10 benchmark (image classification). URL https:// paperswithcode.com/sota/image-classification-on-cifar-10 .

Papers with code - SVHN benchmark (image classification). URL https://paperswithcode. com/sota/image-classification-on-svhn .

Papers with code Fashion-MNIST benchmark (image classification). URL https:// paperswithcode.com/sota/image-classification-on-fashion-mnist .

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, Los Alamitos, CA, USA, 2016. IEEE Computer Society. doi: 10.1109/CVPR.2016.90. URL https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.90 .

Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV , volume 9908 of Lecture Notes in Computer Science , pages 646-661. Springer, 2016. doi: 10.1007/978-3-319-46493-0\_39. URL https: //doi.org/10.1007/978-3-319-46493-0_39 .

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Edwin R. Hancock 555 Richard C. Wilson and William A. P. Smith, editors, Proceedings of the British Machine 556 Vision Conference (BMVC) , pages 87.1-87.12. BMVA Press, 2016. ISBN 1-901725-59-6. doi: 557 10.5244/C.30.87. URL https://dx.doi.org/10.5244/C.30.87 . 558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks, 2015. URL https://arxiv.org/abs/1505.00387v2 .

Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017. URL https://openreview. net/forum?id=r1Ue8Hcxg .

Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu. Practical block-wise neural network architecture generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018. URL https://openaccess.thecvf.com/content_ cvpr_2018/html/Zhong_Practical_Block-Wise_Neural_CVPR_2018_paper.html .

Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V. Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 2902-2911. PMLR, 2017. URL https://proceedings.mlr.press/v70/real17a.html .

Masanori Suganuma, Shinichi Shirakawa, and Tomoharu Nagao. A genetic programming approach to designing convolutional neural network architectures. In Proceedings of the Genetic and Evolutionary Computation Conference , GECCO '17, page 497-504, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450349208. doi: 10.1145/3071178. 3071229. URL https://doi.org/10.1145/3071178.3071229 .

Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image classifier architecture search. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019 , pages 4780-4789. AAAI Press, 2019. ISBN 978-1-57735-809-1. URL https://aaai.org/ojs/index.php/AAAI/ article/view/4405 .

Dongzhan Zhou, Xinchi Zhou, Wenwei Zhang, Chen Change Loy, Shuai Yi, Xuesen Zhang, and Wanli Ouyang. EcoNAS: Finding proxies for economical neural architecture search. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 11393-11401, 2020. doi: 10.1109/CVPR42600.2020.01141. URL https://ieeexplore. ieee.org/document/9157571 .

Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 4095-4104. PMLR, 2018. URL https://proceedings.mlr. press/v80/pham18a.html .

Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https://openreview.net/forum?id= S1eYHoC5FX .

Niv Nayman, Asaf Noy, Tal Ridnik, Itamar Friedman, Rong Jin, and Lihi Zelnik. XNAS: Neural architecture search with expert advice. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. doi: https://proceedings. neurips.cc/paper_files/paper/2019/file/00e26af6ac3b1c1c49d7c3d79c60d000-Paper.pdf.

URL

https://proceedings.neurips.cc/paper_files/paper/2019/file/

.

00e26af6ac3b1c1c49d7c3d79c60d000-Paper.pdf

Peer reviews:
comments: SUMMARY:
In this paper, the authors propose an enhanced version of DensEMANN, which efficiently grows and trains small DenseNet architectures. They employ a macro-algorithm to expand new layers and utilize a micro-algorithm to construct new convolution operations. Through iterative layer growth, this method generates novel architectures within a few GPU hours.

STRENGTHS:
- Detailed experimental settings are provided in this paper.

WEAKNESSES:
- The motivation behind this research requires additional clarification.
- As shown in Table 2, GO methods consumes the leaset GPU days and achieves the best performance. Therefore, it raises the question of why not directly utilize the GO methods.
- The experimental comparison with the original DensEMANN is missing.
- Experiments are only conducted on small datasets. Can DensEMANN be applied on large datasets, for example ImageNet-1k?
- Too many hyper-parameters are introduced in this method, which brings difficulties for applying this method on other tasks.

QUESTIONS:
- By automatically growing small DenseNet architectures, what insights can we obtain in architectural design?

LIMITATIONS:
Please see the Weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-04

comments: SUMMARY:
The authors study an algorithm for neural architecture search (NAS) called DensEMANN, which uses a progressive adaptation of a DenseNet architecture during training to find an efficient neural network for the target task.

STRENGTHS:
The paper is very well written and the authors do a good job of explaining how the DensEMANN algorithm works.

WEAKNESSES:
I’m not sure that the paper currently has enough substance for publication. The authors’ spend the first 5 pages on introduction and description of the previously published DensEMANN algorithm. The changes made to this algorithm are described in section 3.2 in only 20 lines of text and appear to be primarily changes to the various hyperparameters of the existing algorithm.

The primary contribution of the paper is comparison of DensEMANN with other NAS methods on CIFAR-10 in section 4.3. The results are certainly interesting, but I think the authors should focus on quality per unit time rather than quality and parameter counts plotted in Figure 2. Plotting quality against the execution times in table 2 would make it much easier to compare DenseMANN to existing methods in efficiency, which is the primary property of interest, I think.

That being said, I’m not sure empirical comparison of an existing method with state-of-the-art methods is enough novelty to merit publication at NeurIPS. I’d encourage the authors to continue to develop their exploration. For example, clearly establishing a new state-of-the-art in efficiency for NAS. Or, taking the models from DensEMANN and studying their efficiency for deployment.

QUESTIONS:
I do not have specific questions aside from those listed above.

LIMITATIONS:
I did not identify potential negative societal impact of this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents  a new version of DensEMANN, an algorithm for generating small and competitive DenseNet architectures with optimal weight values. The authors aim to approach state-of-the-art performance for well-known benchmarks, or at least the state-of-the-art Pareto front between performance and model size. They achieve this by introducing a new version of the algorithm that uses a combination of layer pruning and weight optimization techniques. The authors evaluate DensEMANN on three popular image classification benchmarks (CIFAR-10, Fashion-MNIST, and SVHN) and show that it outperforms or matches the state-of-the-art methods in terms of accuracy and model size. The contributions of the paper are a new algorithm for generating small and competitive DenseNet architectures, a combination of layer pruning and weight optimization techniques, and state-of-the-art results on popular image classification benchmarks.\n\nSTRENGTHS:\n- The pruning and recovery stages which are the contributions of this paper are very well motivated and clearly explained. The way the pruning and recovery stages are designed is novel and using it in this context is rather unique.\n- The authors have very aptly identified how DensEMANN fits in and very well introduced incremental approaches and NAS architectures.\n\nWEAKNESSES:\n- The difference in this paper with the original DenseEMANN is clearly communicated by the authors however all the the points mentioned except 2 (d):\n\n> The pruning and recovery stages have been heavily modified to avoid long recovery\nstages and their effects. We indeed observed that the kCS of settled filters is not\nconstant but actually decreases very slowly over time. If the recovery stage is too long,\nthis causes a very harsh pruning after which the accuracy cannot be recovered.\n\nare just based on observation or not introduced in the paper or are very straightforward changes, I would suggest to consider only 2 (d) as a contribution of this paper.\n-There are other aspects of this model which are very well framed and novel however it is important to note that these parts of the DensEMANN architecture arer not introduced in this paper but the original DensEMANN paper which reduces the novelty of this work by a huge margin.\n- The paper does not provide a clear explanation of how the parameter limit of 500k was chosen for the experiments. Does going above these number of parameters make DensEMANN very computationally intensive or is unable to grow the network sensibly especially considering that 500k parameters in modern comparison are very few parameters especially for vision tasks?\n\nQUESTIONS:\n- This  did not affect my rating at all, but I do agree with this, I would hope that the authors could add a bit more in these statements to explain the significance of growing-based NAS algorithms:\n\n> The most serious competitor to growing-based NAS algorithms are trainless or zero-cost algorithms\n[30 , 31 , 2 ]. These evaluate candidate NN on basis of their performance with random weights. Such\nmethods can explore large search spaces in a matter of minutes or even seconds [31, 2]. However,\nextra time is still needed for training the final candidate architecture in order to use it.\n\n- The authors evaluate DensEMANN on three popular image classification benchmarks. Can they provide more insights into how the algorithm performs on other datasets (especially more complex larger datasets) or tasks, such as object detection or semantic segmentation (I think DensEMANN should very simply be able to approach this)? The datasets question is also a suggestion to include more experiments on that since the dataset complexity is a very important aspect in judging the efficacy of DensEMANN. However, as of the other tasks question, I was just wondering if the authors had already tried it out and faced any problems?\n- A very common question from this paper is if the authors had tried replicating similar techniques for other larger similar architectures and for more modern architectures. Understanding if DensEMANN is able to generate more complex models is a very big question (and limitation)?\n\nLIMITATIONS:\n- With the current set of evaluation the authors do it is very hard to determine if DensEMANN like techniques can be applied for larger and more modern models", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper proposes a new version of the existing DensEMANN, which grows small DenseNet architectures and trains them on target data. It claims that this version can quickly and efficiently search for small and competitive DenseNet architectures. The proposed approach has been evaluated on a number of benchmarks.

STRENGTHS:
- The idea of automatically generate efficient architectures from a reference makes sense, and can be of great interests in many application scenarios. 
- The proposed approach grows the architecture at both macro and micro levels, which seems to be a valid strategy. 
- The proposed approach has been evaluated on various benchmarks, showing comparable performance with the state of the art.

WEAKNESSES:
- The delta compared to the original algorithm seems to be not very significant. 
- The claim on being able to generate efficient densenet architectures is only supported by the number of parameters. For densenet like models the number of parameters might not be a good indicator for efficiency, due to many skip connections. Thus it seems not a very fair comparison.
- It is not clear how the proposed approach performs on larger datasets such as ImageNet.

QUESTIONS:
- How this approach performs on Imagenet? 
- How the FLOPs/latency of the discovered models comparing with state of the art?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-08

comments: SUMMARY:
This paper proposed a new and improved algorithm to grow small DenseNet architecture from scratch while simultaneously training them from target data.

STRENGTHS:
1. The paper is very clear and readable.
2. The evaluation is comprehensive and detailed, demonstrating the effectiveness.

WEAKNESSES:
1. The novelty is limited. The algorithm is backboned on a well-known algorithm, and the change to it is limited.
2. The scope of this algorithm is limited too.

QUESTIONS:
Please analyze what improvement can be brought by the differences in section 3.2.

LIMITATIONS:
Can this algorithm is adapted to other application fields?
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-25