Title: From One to Zero: Causal Zero-Shot Neural Architecture Search by Intrinsic One-Shot Interventional Information
Venue: NeurIPS 2023
Accepted: no
Keywords: Neural Architecture Search
Abstract: ''Zero-shot'' neural architecture search (ZNAS) is key to achieving real-time neural architecture search. ZNAS comes from ''one-shot'' neural architecture search but searches in a weight-agnostic supernet and consequently largely reduce the search cost. 
  However, the weight parameters are agnostic in the zero-shot NAS and none of the previous methods try to explain it. 
  We question whether there exists a way to unify the one-shot and zero-shot experiences for interpreting the agnostic weight messages. To answer this question, we propose a causal definition for ''zero-shot NAS'' and facilitate it with interventional data from ''one-shot'' knowledge.
  The experiments on the standard NAS-bench-201 and CIFAR-10 benchmarks demonstrate a breakthrough of search cost which requires merely 8 GPU seconds on CIFAR-10 while maintaining competitive precision. 

Abstract

'Zero-shot' neural architecture search (ZNAS) is key to achieving real-time neural architecture search. ZNAS comes from 'one-shot' neural architecture search but searches in a weight-agnostic supernet and consequently largely reduce the search cost. However, the weight parameters are agnostic in the zero-shot NAS and none of the previous methods try to explain it. We question whether there exists a way to unify the one-shot and zero-shot experiences for interpreting the agnostic weight messages. To answer this question, we propose a causal definition for 'zeroshot NAS' and facilitate it with interventional data from 'one-shot' knowledge. The experiments on the standard NAS-bench-201 and CIFAR-10 benchmarks demonstrate a breakthrough of search cost which requires merely 8 GPU seconds on CIFAR-10 while maintaining competitive precision.

1 Introduction

Neural architecture search has been an interesting topic in the AutoML community [27]. Traditional methods search by training the distinct neural architecture iteratively [31] whose training cost is huge. One-shot model cleverly use a supernet to merge all the singular neural architectures into one and consequently, the waste of search time is largely saved [16]. Further, the gradient-based one-shot method [12] is proposed which acquires robust results on NASNet [32]. Though the one-shot model largely reduces the search cost, it still suffers from a weight-sharing problem, and especially, gradient-based approaches cause degenerate architectures [29]. The work [25] gives theoretical proof for this and subtly uses a progressive tuning metric to discretize the one-shot supernet iteratively which gets awesome neural architectures. However, it still gets degenerate architectures with different training settings.

The brilliant work [5] from Google Brain gives a hint for searching neural networks without tuning the parameters. 'To produce architectures that themselves encode solutions, the importance of weights must be minimized'. In this manner, a zero-shot neural architecture search (ZNAS) is born. The work [10] firsts propose the idea of ZNAS to be 'it does not optimize network parameters during search '. From a one-shot perspective, the 'zero-shot' is given credit by 'one-shot' where single neural architectures are supposed to be selected from the weight-agnostic supernet [5]. Considering causal weight messages, the 'zero-shot' select neural architecture with the minimum impact of any weight parameter [5]. Thus a causal definition is supposed to be that the weight messages are multi-environmentally distributed. Compared to one-shot NAS, zero-shot NAS gets imperfect weight messages due to random initialization and searching without training [10, 2].

A training-free approach is first proposed by the work [13]. Different from the previous zero-shot 33 model [10], the work [13] samples well-trained architectures and get validation accuracy to train the 34

statistical proxy before it searches. The work [2] follows the way of the previous work [10] and uses 35 the DARTS search space to conduct zero-shot NAS on CIFAR-10 and ImagNet in a training-free 36 manner. However, the number of samples directly decides the belief of the final precision. The 37 'well-trained' architectures might not be 'perfectly-trained' in different training settings. 38

39

40

41

42

43

44

45

46

47

48

Zero-shot NAS learns the representation of neural architectures to get the best one. Consistently compared to one-shot NAS methods, zeros-shot NAS methods ignore the weight information. By merely measuring the architectural expressivity, they overlooked the impact of weights as a necessary assessment element. From a one-shot NAS perspective, architectural information can be represented by a list of neuron representations [25]. The message of training weights ω supports the neuron's representation [15, 12, 25]. Because the structural dependencies of shared (mutual) messages across neurons are all agnostic [5], in the zero-shot neural architecture search, the neuron's representation is harder to interpret due to the random messages. What is worse, the uniterpretability might result in large bias and variances because the imprecise observational data might be misleading. Finally, it will lead the search to get degenerate architectures through the process of accumulating errors.

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

We first propose to interpret the zero-shot NAS in a causal-representation-learning setting. According to the weight-agnostic setting, we formulate the zero-shot NAS as a novel framework for imperfectinformation NAS. The structural information of zero-shot NAS is interpreted by impact with the latent factors. As a consequence, intrinsic high-level interventional data acquired by one-shot NAS is properly adopted to refine the imperfectness. Moreover, we reformulate the causality by game theory and interpret the imperfect-information NAS as imperfect information game G . Extensive experiments on various benchmark datasets including CIFAR-10, NAS-Bench-201, and ImageNet have shown the super search efficiency ( 10000 × faster than DARTS) of our methods. In this work, our main contributions are as follows:

We propose that the causal zero-shot NAS is to learn the neuron's representation with latent factors in observationally imperfect messages.

We theoretically demonstrate the validation information of either a neuron or a neuron ensemble obeys a Gaussian distribution given a Gaussian input.

The proposed method uses high-level interventional data from one-shot NAS to facilitating zero-shot NAS to solve the imperfectness.

Our method sets the new state-of-the-art in zero-shot NAS of search cost (8 GPU seconds) while maintaining comparable test accuracies.

2 Preliminaries and Related Work

In this section, we talk about the preliminaries and the previous works on one-shot NAS and zero-shot NAS. We talk about the motivation to replace statistical proxy by introducing the basic knowledge on causal interventaional representation learning in causality [20, 1].

2.1 One-shot NAS

One-shot NAS methods [12, 16], that unify all the single-path neural architectures into one supernetwork S (supernet), select the single-path neural architecture as the best one by training the weights ω in a weight-sharing manner and maximizing the validation accuracy ( V ) of architecture A as follows:

The iterative updating of ω and selection of A makes the one-shot NAS a bi-level optimization problem that is NP-hard. Differentiable one-shot model also relies on the observational data from unitedly trained validation accuracies of differentiable subnets [12]. Wang et al. [25] propose a selection-based approach to modify the output of differentiable one-shot NAS [12] to discretize a single-path neural architecture that consists of operations (neurons) with strength. As a consequence, the perturbation-based inductive bias is demonstrated to be helpful to solve the degeneration.

2.2 Statistical proxies in zero-shot NAS

We compare the various training-free and zero-shot NAS methods according to the usage of statistical 82 representation. Some training-free approaches use the statistic of validation accuracy to predict the 83

final architecture. NASWOT [13] samples a number ( N ) of well-trained neural architectures from 84 the NAS-Bench-201 dataset to learn the kernel. However, to get these representations, the training 85 costs tremendously. The zero-shot methods directly use zero-cost statistical proxies to represent the 86 expressivity without weights and validation accuracy. Zen-NAS [10] uses a Gaussian complexity to 87 measure the network expressivity and evolve the architectures to maximize the expressivity. Other 88 training-free approaches such as TE-NAS [2] and NASI [22] imitate the train of NAS by neural 89 tangent kernel (NTK) which largely reduces the waste of train cost. TE-NAS [2] propose to maximize 90 the number of linear region of activation patterns [14]. On the opposite, NASI [22] subtly optimize 91 the trace of NTK by sampling. 92

93

94

95

96

97

98

99

Here raise the question that to what extent the validation accuracy outperforms the statistical proxy. Vice versa, we question if the statistical proxy is in substitute of the validation accuracy. Compared to the proxy-based methods with approximations, the validation-based method is more reproducible. The validation accuracy is an intrinsic robust and upper-bounded proxy to measure the neural architectures. Besides, previous arts of one-shot manner usually use the validation accuracy to be the objective to maximize. Despite these benefits, the zero-shot representation is imperfect due to the weight-agnostic messages.

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

2.3 Causal representation learning

The study [20] demonstrates that causality is a 'subtle concept' which can not be fully described by Boolean or Probabilistic. It is more about reasoning. Reichenbach demonstrates a common cause principle to explain the causality by dependencies among variables [18]. Causal representation learning mainly deals with learning causally for representations. By observational data, we can hardly learn the real circumstances (environments), especially in complex scenes and high-dimensional data scenarios. Causal representation learning seeks to extract high-level information (dependencies) from low-level data. Interventions have taken a prominent role in representation learning literature on causation. The work [1] uses interventional data to facilitate the causal representations to get precise outcomes. Neural architecture search aims at learning the architectural representations automatically. The automatism of the previous arts of neural architecture search might not be causal especially in zero-shot setting.

3 Method

3.1 Imperfect information

Neural architecture search is a task aiming at interpreting the mechanism of architectural knowledge of neural networks given methods of evaluations. Activation patterns, statistical proxies, and naive validation accuracy are adopted to evaluate the score of a neural network. However, we can hardly understand any neural network and even hardly explain the weight distribution of any neural network without assumptions. Observational data are always imperfect due to the infinite environments (search spaces/training schemes/hardware/etc.) of all possible networks with finite observations and limited tools. Architecture information is not stand-alone.

In one-shot NAS, demonstrated in Equation 4, given a neural network, we first train the weights ω and the ω combined with architecture A can give a validation accuracy V . After V is given, we then update the ω to get ¯ ω and a novel architecture A until the validation accuracy V is maximum. In the train, the architecture of a neural network is the key factor that impacts the other two factors ω and validation accuracy V . The search is actually a reverse way of train to the aspect of the intrinsic dependency of accuracy V on the weight ω and architecture A . However, we have overlooked a lot of factors like data distributions, batch sizes, rates of weight decay, and so on and on which we can not optimize as 'one shot'. If the observational data alone can not interpret the phenomenon, it is a must to model the latent factors Z that cause this uninterpretability. Figure 1 illustrates the dependencies of architecture A , validation accuracy V , and weights ω . The dashed line reveals that Z changes the dependencies of selected neurons (or searched architectures) on observational data of ω and V [23], which indeed implies strong causality [20]. In logical condition, the structural relationship between V and ω can be almost broken 1 .

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

Weassume the validation accuracy V of a set of neural architectures {A} obeys a Gaussian distribution.

Due to the random weight information, artificial neural networks (ANN) themselves have architectural information to deliver the neural networks' expressivity with large variances [5]. It is demonstrated that the weight-agnostic neural network still preserves the 92% accuracy-level information for digit classification by the work [5]. However, the weights are agnostic and consequently the validation accuracies are imperfect. We assume the true validation accuracy is the difference of the observational V obser and latent impact of factor Z demonstrated in Equation 3.

3.2 Problem formulation

In Zen-NAS, the adoption of statistical proxy on the feature map is impressive while it is constrained to structural dependencies [10]. We question to what extent, when we search a neural network, the statistical proxies can be replaced with the more robust functions such as validation accuracy causally [20]. In some one-shot [16, 12] and training-free methods [13], the evaluation metrics are usually the validation accuracy of the associated neural architectures.

̸

Inspired by the previous work [25], we evaluate each neuron to select respectively in substitute. Intuitively, we measure the importance of each neuron by a simple validation accuracy of a singular associate neuron while resting other neurons on the same edge. DARTS+PT [25] the perturbationbased approach mutes the irrelevant neurons to conduct an inference while saving the other paralleled edges. For each paralleled edge (layer) E that contains M neurons N s, we mute the other neurons while only saving the i th neuron N ( i ) . The k th paralleled edge E ( k ) i consequently only contains one neuron (operation): E ( k ) i = { 0 × N (1) , 0 × N (2) , . . . , N ( i ) , . . . , 0 × N ( M ) } . When saving the other paralleled edges { E ( j ) } j = k , N ( i ) denotes any single sub-architecture (a neuron) in the supernet S with tuned weights ω S of the supernet. Formally, the one-shot neuron selection for k th paralleled edge is defined as:

where validation accuracy V is measured by an intrinsic inductive bias function F such as a reinforcement learning policy π [31, 32]. V ( N ( i ) ) = V ( { E (1) , E (2) , . . . , E ( k ) i , . . . , E ( N ) } ) in practise.

In zero-shot NAS, the weight information is agnostic, which is impacted by a latent factor Z as shown in Figure 1. [4]. The latent variable obeys a distribution P in dimension Λ :

When we sample larger enough numbers of impacts, the sample of factor Z obeys a Gaussian 162 distribution by the central limit theorem (CLT). The causal zero-shot neural architecture search 163 (Causal-Znas) that searches in imperfect messages is defined as: 164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

̸

for i = 1 , 2 , . . . , M . In this Equation 6, Z means the latent information to impact agnostic-weights (such as a random initialization [5, 10]) and consequently validation accuracies V . Therefore, we get a causal information set of singular neuron representation {V ( N ( i ) ) |Z} for i = 1 , 2 , . . . , M . For each paralleled edge (layer) E that contains M neurons N s: E = { N (1) , N (2) , . . . , N ( M ) } . We calculate the information of singular neuron N i on edge E ( j ) by freezing the other layers (ensembles/edges) { E ( k ) } k = j so that the causal information is only impacted by the current neurons due to the same condition (in the same paralleled edge). Then the causal information set of a paralleled edge E is as:

In a Causal-Znas, a prediction function F is able to measure the selected architectures from the un-trained supernet. To avoid the improper introduction of inductive biases, we use an identity function to measure the importance of neurons.

3.3 Gaussian intervention

Most existing NAS approaches use observational data and make assumptions on the architectural dependencies to achieve provable representation identification. However, in our causal zero-shot neural architecture search, there is a wealth of interventional data available. To perfect the observational validation accuracies V obser in D , we sample V ven from an interventional distribution D ( Z ) to be in substitute for the ones derived by the observation V obser . Formally, we have: V ven ∼ D ( Z ) . Though pure architectural information is imperfectly obseved, we can use an interventional function I ( do intervn [1]) to replenish data from a one-shot perspective:

Ming et al. [10] assume the inputs obey Gaussian distribution and get comparable results with one-shot methods [12, 16]. What we use as the input for each neuron is a Gaussian image which also obeys the assumption of Gaussian inputs of Zen-NAS [10].

Lemma 1. Given a Gaussian input X ∼ N ( µ, σ 2 ) , the output of a neuron N in the first layer is Gaussian.

Proof. Assuming each neuron is a distinct convolution denoted as Conv i for i = 1 , 2 , . . . , M , then the output of this edge is:

where X ∼ N ( µ, σ 2 ) and W ( i ) ∼ N ( µ w , σ 2 w ) for i = 1 , 2 , . . . , M . Given the i.i.d. inputs and weights, the output score (validation accuracy) of the neural network layer is Gaussian since the Convolution of a Gaussian (random variable) is still a Gaussian (random variable). We have Gaussian weights W ( i ) and Conv ( i ) ( X , W ( i ) ) ∼ N ( µ ( i ) , σ 2 ( i ) ) . Then ∑ i Conv ( i ) ( X , W ( i ) ) ∼ N ( ∑ µ ( i ) , ∑ σ 2 ( i ) ) .

Lemma 2. Given a Gaussian input X ∼ N ( µ, σ 2 ) , the output of a neuron N in any layer is Gaussian.

Proof. Apparently, any weighted summation of random variables that obey two distinct Gaussian is still a Gaussian. In neural networks, the layers are stacked. Based on Lemma 1, in the latter layer, the outputs also obey the Gaussian, whose inputs are the former layer's outputs. The convolution (neuron) Conv ′ ( i ) of the next layer with output of latter layer O (in Equation 9) has Conv ′ ( i ) ( O ) ∼ N ( µ ′ ( i ) , σ ′ ( i ) 2 ) .

Corollary 2.1. Given a Gaussian input X ∼ N ( µ, σ 2 ) , the output of any neuron ensemble { N ( i ) } i ∈M is Gaussian.

Formally, we have O ( i ) ∼ N ( i ) ( µ ′ , σ ′ 2 ) . ˜ O = {O (1) , O (2) , . . . , O ( K ) } where ˜ O denotes all the outputs across edges ︷ ︸︸ ︷ E (1) , E (2) , . . . , E ( K ) . Based on Lemma 1 and Lemma 2, we get the Corollary 2.1 to select edges (topology preferences).

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

Proof. By Lemma 1, we have any neuron N ( i ) has a Gaussian output O ( i ) ∼ N ( µ ( i ) , σ 2 ( i ) ) . Any ensemble of neurons has an output ∑ i O ( i ) . Then we have ∑ i O ( i ) ∼ N ( ∑ µ ( i ) , ∑ σ 2 ( i ) ) .

As demonstrated in Equation 8, we propose an intervention function I D to facilitate the imperfect causal representation of the validation information. We propose that the ideal information is distributed in the information set by a probability p . The distribution D is N ( µ, σ ) in the context.

Herein, we question to what extent, the imperfectness can be interventionally refined [1]. We use the parameter p to asymmetrically flipping the random Gaussian I N ( µ,σ 2 ) p [15] to understand the imperfect information in dimension Λ which is mapped to a vanilla Gaussian (in Equation 5). As shown in Figure 2, it compares the information difference between the observational information set and interventional information set impacted by the parameter p . In different environments, the data of interventional data combined with observation obeys a distinct Gaussian, which implies strong coherence and robustness. When p = 1 , the causality is perfectly achieved due to breaking the dependency of validation accuracy V on weights ω ; otherwise, it is imperfect. The mean and variance coefficients of the additional notion of intervention are derived by sampling validation accuracy of one-shot prior. We propose that setting of p is conditional on the fraction of the mean of latent factor to the difference of the mean of observational data and the mean of interventional data.

Proposition 1. When p -→ µ Z µ obser -µ ven , the mean of the intervened data ˜ µ -→ µ true .

As demonstrated in Proposition 1, a sufficient condition of the mean of intervened data is getting closer to the true mean of the validation accuracy is that the p is closer to 1 and interventional data is closer to the true data.

3.4 Causal zero-shot neural architecture search

We formulate the zero-shot NAS into ensemble selection and neuron selection. There are K neuron ensembles ︷ ︸︸ ︷ { N ( i ) } (1) i ∈M , { N ( i ) } (2) i ∈M , . . . , { N ( i ) } ( K ) i ∈M . For each ensemble, there are M neurons (operations). The ensemble selection is the selection of an ensemble { N ( i ) } ( j ) i ∈M of neurons among the K ensembles ( j ∈ K ), while neuron selection follows the same formula and selects a neuron N ( i ) from a neuron ensemble { N ( i ) } ( j ) i ∈M .

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

As is shown in Figure 3, the validation accuracy of both a neuron and a neuron ensemble obey Gaussian distributions respectively. From a macro perspective it is an ensemble selection while from a minor perspective, it is a neuron selection. Thus we talk about both types in the same formula.

As demonstrated in Equation 6, the final outcome neurons are derived by maximizing their validation accuracies according to the latent factor. Given the Gaussian intervention in Equation 8, we further modify the formula of the causal neuron selection by doing intervention (without the additional inductive bias [20]):

, where ˜ V is the validation accuracy with intervention.

The methodology of neuron selection is given in Algorithm 1. The search process of neuron ensemble follows the same formulation as mentioned in this Section. do intervn represents to do intervention. At first, the weight ω of the supernet is randomly initialized [10]. Second, validation scores V on the validation set are prepared for the calculation of the neurons N which adopts probability p to do the intervention. At last, the maximum of values is compared to select the best neuron (operation). In practice, when the probability p is close to 1, the validation accuracy of observation has less need to compute.

Equation 6 reveals a universal formula for causal neural architecture search in the zero-shot settings. The measure function F measures the importance [25] ('responsibility') of a neuron and Shapley value is proposed to be ideal for the selection of a neuron [7] or ensemble [19].

We use the game-theoretic inductive bias to extract the valuable information [20, 7]. G represent the Shapely value [21]. Given Corollary 2.1, we know that any the neuron ensemble obeys a Gaussian distribution. The information set of Shapley value is thus build on top of an ensemble of Gaussian variables. However, we could not guarantee a Gaussian distribution of the Shapley value [24]. As a consequence, we use a Gaussian distribution to do intervention on validation accuracy and then calculate the Shapely value of the intervened validation accuracy. At last, the Shapley value is maximized whose associated neuron is supposed to be more expressive [7].

3.5 Weight-agnostic weights

In the assumptions of various methods, weights are initialized as Gaussian. However, in our framework, we demonstrate that this strong assumption is not a must. Supernet can be initialized in different ways: i) with Gaussian [10], ii) Uniform [5], and iii) Constant number [5].

Corollary 2.2. Given a Gaussian input X ∼ N ( µ, σ 2 ) , if the initial weights are Uniform or Constant number C , the output of any neuron ensemble { N ( i ) } i ∈M is not Gaussian.

Proof. Apparently, the convolution of a Gaussian input with constant or uniform weights obeys a difference of CDF Φ of the Gaussian in the range of constant or uniform.

In the previous work [5], it is proposed that weights are supposed to be initialized by a distribution but not a constant ( C ). To be more precise, we propose that the constant value could not represent the agnostic weights and thus could not reflect the latent information while a uniform distribution can guarantee the randomness. By training on a 'wide range' of uniform weight samples, Gaier et al. propose that 'the best performing values were outside of this training set' [5]. We propose that this phenomenon is essentially resulted from a distribution shift of the Gaussian validation accuracy which causes the change of search procedure. To solve the distribution shift, we could use the difference of CDF of Gaussian ( Φ ) to conduct intervention. Even in a broader view, if the weights distributions are totally unknown, we can use Bayesian method to approximate a distribution D ( Z ) in Equation 8.

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

4 Experiments

We present the results and all experiment details of our method in this section. A robustness analysis is included to examine the stability of our method, which also explains the time efficiency. Results are given on the benchmark datasets, NAS-Bench-201 and CIFAR-10.

4.1 Experimental details

Weuse the search space of DARTS [12] for fair comparisons with the state-of-the-art NAS approaches. During the searching process, we follow adopting the same and hyper-parameters as DARTS [12] to initialize the supernet on the CIFAR-10 and NAS-Bench-201 datasets for a fair comparison with DARTS-variants (one-shot methods). All the training is conducted on a single 2080Ti GPU.

4.2 Results on CIFAR-10

As shown in Table 1, we compare the proposed Causal-Znas and game-version Causal-Znas-G with the state-of-the-art methods. The comparisons are made with respect to the informatics of the model, including test accuracy on the test set (Test Error), the number of parameters (Params), the search costs, and the search strategies. As shown, our results set the new state-of-the-art search speed with a competitive test error rate. Compared to DARTS [12], our method is 10000 × faster with comparable accuracy (2.75% v.s. 2.76%). Compared to DARTS+PT [25], our model is much simpler without introducing the perturbation-based inductive bias [20] and achieves a similar test error rate (2.61% v.s. 2.61%). DrNAS [3] and ISTA-NAS [26] are not only precise (2.54%) but also theoretically sound approaches. ISTA-NAS [26] is extremely fast in one-shot NAS while ours are more competitive (500 × faster) in search efficiency.

We compare our method with other zero-shot NAS approaches in Table 1. It demonstrates that the TE-NAS [2] which is the first algorithm that reaches 4 GPU hours search cost is experimentally awesome. TE-NAS uses the neural tangent kernel to approximate the train so it largely reduces the cost of training the neural networks. Compared to TE-NAS, our proposed approach is 500 × faster and our game-based result (-G) gets a comparable test error rate (2.61% v.s. 2.63 %) with a smaller number of parameters (3.1M v.s. 3.8M). We also surpass the current state-of-the-art zero-shot (training-free) method (NASI) [22] by more than 100 × in search efficiency and get fewer errors in both settings (2.75% v.s.2.79%; 2.89% v.s. 2.90%).

4.3 Results on NAS-Bench-201

NAS-Bench-201 is a pure-architecture-aware dataset where the neural architectures are trained in the same settings, and the info such as performance, parameters, architecture topologies, and operations

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

are available. Compared to NAS-Bench-101 [28], NAS-Bench-201 adopts a different search space and gets results on various datasets such as CIFAR-10, CIFAR-100, and ImageNet16-120.

As shown in Table 2, it compares our proposed method with the state-of-the-art methods on NASBench-201. Compared to NASWOT(N=10) [13], NASWOT(N=100) and NASWOT(N=1000) are much more accurate due to enlarged sample amounts. However, it also cause 10 × and 100 × waste of search costs. NASI [22] also enlarges its search cost to get much more precise results with extension of 90s. Our approach gets the same search cost with NASWOT (3s) while being much more precise on CIFAR-10 (90.03% v.s. 89.14%, 93.49% v.s. 92.44), CIFAR-100 (70.18% v.s. 68.50%, 71.18% v.s. 68.62%) and ImageNet 16-120 (43.83% v.s. 41.09%, 44.43% v.s. 41.31). A 9s extension of search cost ( Ours-G ) by neuron games gets even better results than NASWOT and NASI for their extreme results.

4.4 Results on ImageNet with the DARTS search space

As shown in Table 3, we report the searched results on ImageNet. The validation size of the observation data batch is 1024. On ImageNet, the number of classes is 1000 so a large data batch is necessary. Compared to NASI [22], and TE-NAS [2], our search costs are faster when p = 1 . The larger batches for evaluation enlarge the search cost for observational data resulting in a slightly larger search cost when p = 0 . 5 . Ours(p=1) gets a competitive test error rate (25.0%) in the table and NASI-ADA [22] gets similar result (24.8%) but NASI-ADA has a larger search cost (864s v.s. 8s).

5 Conclusion

In this work, we interpret the zero-shot NAS as a causal representation learning and solve it by interventional data from one-shot NAS. Besides, our work is dedicated to displaying the inheriting relationship among the latent variables. We demonstrate that the neural architectures can be evaluated and selected by a Gaussian distribution given Gaussian inputs. Experiments on benchmark datasets reveal awesome efficiency and competitive accuracy.

330

331

332

References

Kartik Ahuja, Divyat Mahajan, Yixin Wang, and Yoshua Bengio. Interventional causal representation learning. arXiv preprint arXiv:2209.11924 , 2022.

Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in 333 four gpu hours: A theoretically inspired perspective. arXiv preprint arXiv:2102.11535 , 2021. 334

335

336

Xiangning Chen, Ruochen Wang, Minhao Cheng, Xiaocheng Tang, and Cho-Jui Hsieh. Drnas: Dirichlet neural architecture search. arXiv preprint arXiv:2006.10355 , 2020.

337

338

339

340

341

342

Frederick Eberhardt and Richard Scheines. Interventions and causal inference. Philosophy of Science , 74(5):981-995, 2007.

Adam Gaier and David Ha. Weight agnostic neural networks. Advances in neural information processing systems , 32, 2019.

Huang Gao, Liu Zhuang, LVD Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR , volume 1, page 3, 2017.

Amirata Ghorbani and James Y Zou. Neuron shapley: Discovering the responsible neurons. 343 Advances in Neural Information Processing Systems , 33:5922-5932, 2020. 344

345

346

347

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 , pages 770-778, 2016.

348

349

350

351

352

353

354

355

356

357

Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Uncertainty in Artificial Intelligence , pages 367-377. PMLR, 2020.

Ming Lin, Pichao Wang, Zhenhong Sun, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, and Rong Jin. Zen-nas: A zero-shot nas for high-performance image recognition. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 337-346, 2021.

Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceedings of the European conference on computer vision (ECCV) , pages 19-34, 2018.

Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 , 2018.

Joe Mellor, Jack Turner, Amos Storkey, and Elliot J Crowley. Neural architecture search 358 without training. In Proceedings of the International Conference on Machine Learning , pages 359 7588-7598. PMLR, 2021. 360

Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of 361 linear regions of deep neural networks. Advances in neural information processing systems , 27, 362 2014. 363

Yookoon Park, Sangho Lee, Gunhee Kim, and David M. Blei. Unsupervised representation 364 learning via neural activation coding. arXiv preprint arXiv:2112.04014 , 2021. 365

Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture 366 search via parameters sharing. In International Conference on Machine Learning , pages 367 4095-4104. PMLR, 2018. 368

Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image 369 classifier architecture search. In Proceedings of the aaai conference on artificial intelligence , 370 volume 33, pages 4780-4789, 2019. 371

Hans Reichenbach. The Direction of Time . Dover Publications, 1956. 372

Benedek Rozemberczki and Rik Sarkar. The shapley value of classifiers in ensemble games. 373 arXiv preprint arXiv:2101.02153 , 2021. 374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE , 109(5):612-634, 2021.

LS Shapley. Quota solutions op n-person games1. Edited by Emil Artin and Marston Morse , page 343, 1953.

Yao Shu, Shaofeng Cai, Zhongxiang Dai, Beng Chin Ooi, and Bryan Kian Hsiang Low. Nasi: Label-and data-agnostic neural architecture search at initialization. arXiv preprint arXiv:2109.00817 , 2021.

Jin Tian and Judea Pearl. Causal discovery from changes. arXiv preprint arXiv:1301.2312 , 2013.

Isabella Verdinelli and Larry Wasserman. Feature importance: A closer look at shapley values and loco. arXiv preprint arXiv:2303.05981 , 2023.

Ruochen Wang, Minhao Cheng, Xiangning Chen, Xiaocheng Tang, and Cho-Jui Hsieh. Rethinking architecture selection in differ-entiable nas. In International Conference on Learning Representations , 2021.

Yibo Yang, Hongyang Li, Shan You, Fei Wang, Chen Qian, and Zhouchen Lin. Istanas: Efficient and consistent neural architecture search by sparse coding. arXiv preprint arXiv:2010.06176 , 2020.

Quanming Yao, Mengshuo Wang, Hugo Jair Escalante, Isabelle Guyon, Yi-Qi Hu, Yu-Feng Li, Wei-Wei Tu, Qiang Yang, and Yang Yu. Taking human out of learning applications: A survey on automated machine learning. CoRR , abs/1810.13306, 2018.

Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas-bench-101: Towards reproducible neural architecture search. In International Conference on Machine Learning , pages 7105-7114. PMLR, 2019.

Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, and Frank Hutter. Understanding and robustifying differentiable architecture search. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 , 2020.

Hongpeng Zhou, Minghao Yang, Jun Wang, and Wei Pan. Bayesnas: A bayesian approach for neural architecture search. In International Conference on Machine Learning , pages 7603-7613. PMLR, 2019.

Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 , 2016.

Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable 408 architectures for scalable image recognition. In Proceedings of the IEEE conference on computer 409 vision and pattern recognition , pages 8697-8710, 2018. 410

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a 0-shot NAS method. The key point is that there are latent factors that can influence the architecture search procedure, making the validation accuracy of one-shot NAS unreliable. The method adopts Gaussian intervention to the data and evaluates each operation's performance to reduce the bias brought by validation dataset sampling. Some experiment on CIFAR-10, NAS-Bench-201, and ImageNet show that the method can efficiently search architectures.\n\nSTRENGTHS:\n1. The paper is well written and easy to follow.\n2. The paper considers the latent factors that may influence the NAS validation, which has been hardly considered in existing works.\n3. The proposed method uses causal inference techniques to solve the NAS problem.\n\nWEAKNESSES:\n1. Some definitions and connotations are unclear in the paper. See Questions for more details.\n2. The proposed algorithm is too straightforward. It seems directly use intervention technique in NAS procedure, but do not consider any characteristics of NAS problems, which makes the contribution of the method limited.\n3. In the experiment part, the authors claim the efficiency of the method, but lots of 0-shot NAS methods has a very high efficiency. The authors do not compare with those 0-shot NAS methods in this part.\n\nQUESTIONS:\n1. The paper is inconsistent about the connotation of the latent factors. At Line 127, the paper states that many factors like data distributions, batch sizes, rates of weight decay can affect validation. However, according to Section 3.3, the authors seem to consider only the data distribution as the latent factor. What is the exact connotation of the latent factors considered in your paper?\n2. At Line 135, why validation accuracy obeys Gaussian distribution? In my perspective, under some assumptions, the logits can follow a Gaussian distribution. But when it comes to accuracy, it is hard to say it also follows Gaussian Distribution. \n3. How to define \"true validation accuracy\" in Line 140? I tried to understand that the author was trying to say that there is a true distribution of latent factors, and that the results measured in this true distribution are the true accuracy. But how to define the true distribution?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

comments: SUMMARY:
This paper formulates zero-shot NAS as a causal-representation-learning. Further, it uses the high-level interventional data from one-shot NAS to facilitate zero-shot NAS to refine the imperfectness. Extensive experiments achieved comparable performance results on multiple benchmarks.

STRENGTHS:
1) This paper proposes to use the high-level interventional data to facilitate zero-shot NAS to address the imperfect-information issue.

2) This paper provides theoretical support for the proposed approach.

WEAKNESSES:
1) The novelty is incremental compared to baseline Zen-NAS approach, and also directly applies the Shapley value which is also leveraged in Shapley-NAS. 

2) Although the search cost is low, the achieved performance improvement is not significant.

QUESTIONS:
1) Please provide more details about the differences between this work and Zen-NAS、Shapley-NAS, as described in Weakness 1.

2) The experiments should be compared with the baseline methods, including Zen-NAS and Shapley-NAS, as well as with the latest SOTA zero-shot methods such as ZiCo.

3) What about the ranking consistency in different NAS benchmark?

4) Minor Errors

    a) Line 40: "zeros-shot" -> "zero-shot"

    b) Line 313: "92.44" -> "92.44%"; Line 314: "41.31" -> "41.31%"

LIMITATIONS:
The novelty is limited, and the achieved performance improvement is limited.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a causal definition of zero-shot NAS and facilitate this with interventional one-shot knowledge data. The paper theoretically demonstrates the validation information of either a neuron or a neuron 60 ensemble obeys a Gaussian distribution given a Gaussian input. It then uses high level interventional data from one-shot NAS to solve the imperfectness of zero-shot NAS. The zero-cost NAS method is studied on DARTS space and NAS Bench 201 with very low search cost while maintaining comparable test accuracies.\n\nSTRENGTHS:\nThe paper is well-written and novel. Studying NAS from a causality perspective is original and interesting.\n\nWEAKNESSES:\n- The main weakness in my opinion is the weak evaluation of the approach. [NAS-Bench Suite Zero](https://arxiv.org/pdf/2210.03230.pdf) provides easy access and evaluation on about 13 proxies and 28 tasks. Since the performance of a given proxy can vary widely depending upon the search space, task and datasets I am not convinced of the effectiveness of the proxy based on only the results on NATS-Bench and DARTS spaces. Furthermore how would one apply this method to transformer spaces (eg [AutoFormer](https://openaccess.thecvf.com/content/ICCV2021/papers/Chen_AutoFormer_Searching_Transformers_for_Visual_Recognition_ICCV_2021_paper.pdf) , [HAT](https://arxiv.org/abs/2005.14187)) and MobileNets(eg: [OFA](https://arxiv.org/pdf/1908.09791.pdf), which prove queriable validation accuracy based on a surrogate. Can this method be applied in these spaces? Since modern one-shot NAS methods are applied on transformer and mobilenet spaces too, it is important to design a proxy that does indeed generalize well. I encourage the authors to evaluate their method on these search spaces too.\n- The results in table 1 and table 2 show a drop in accuracy ( though the num params is lower and search cost is low). Hence I am not very convinced of the effectiveness of the method in finding effective architectures.\n- Since reproducibility is quite important in NAS, I think it is very important for the authors to release their code (I couldn't find the code attached). \n- Correlation coefficient of the ranking with the true ranking is not studied\n\nQUESTIONS:\nMost questions are covered in the weakness part. \nMy questions are as summarized below:\n1. Could the authors evaluate their method on all the tasks in [NAS-Bench Suite Zero](https://arxiv.org/pdf/2210.03230.pdf) and discuss about the applicability of the method to transformer spaces?\n2. Could the authors report the correlation coefficient of the predicted ranking with the true ranking ?\n3. Would the authors be releasing the code?\n4. Could the authors confirm that the best practices [here](https://www.automl.org/nas_checklist.pdf) are followed?\n\nLIMITATIONS:\nI encourage the authors to discuss the limitations of the proposed method more extensively for eg: assumptions which are architecture type specific? search space generality? Any assumptions which may not hold in practice?", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper proposes a causal zero-shot neural architecture search (NAS).  The NAS problem is decomposed into two components: ensemble selection and neuron selection. By employing the Gaussian intervention to approximate validation accuracy, the authors adapt the perturbation-based approach from DARTS+PT to search for architectures. The performance of their proposed Causal-Znas is evaluated on both NAS-Bench-201 and DARTS search spaces.

STRENGTHS:
- It is reasonable to consider the one-shot interventional information for zero-shot NAS.

WEAKNESSES:
- The overall contribution of this research appears to be limited. The combination of zero-shot NAS with DARTS+PT is not considered novel and does not provide significant insights.
- There are some omissions of recent related works in Section 2.2, such as GradSign[1] and ZiCo[2].
- The experimental comparison provided is insufficient. The results of Zen-NAS are missing in the comparison, and it would be beneficial to include a comparison with GradSign and ZiCo. Additionally, recent works often conduct experiments on other tasks such as NLP and ASR, which could provide further insights.

[1] Zhihao Zhang and Zhihao Jia. Gradsign: Model performance inference with theoretical insights. In ICLR, 2022.

[2] Guihong Li, Yuedong Yang, Kartikeya Bhardwaj, and Radu Marculescu. Zico: Zero-shot nas via inverse coefficient of variation on gradients. In ICLR, 2023.

QUESTIONS:
Please see the Weaknesses.

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-08