SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Is BOHNAS exploring well-performing architectures during search? To investigate this question, we visualized density estimates of the validation error of proposed candidates for all search strategies across our experiments on the hierarchical NAS-Bench-201 search space from Section 6. Figure 11 shows that BOHNAS explored better architecture candidates across all the datasets, i.e., it has smaller median validation errors and the distributions are further shifted towards smaller validation errors than for the other search strategies.
Does incorporating hierarchical information improve performance modeling? To further assess the modeling performance of our surrogate, we compared regression performance of GPs with different kernels, i.e., our hierarchical WL kernel (hWL), WL kernel [48], and NASBOT's kernel [18]. We also tried the GCN encoding [52] but it could not capture the mapping from the complex graph space to performance, resulting in constant performance predictions. Further, note that the adjacency encoding [50] and path encoding [47] cannot be used in our hierarchical search spaces since the former requires a fixed number of nodes and the latter scales exponentially in the number of nodes. We re-used the data from the search runs and ran 20 trials over the seeds  $\{0, \dots, 19\}$ . In every trial, we sampled a training of varying number and test set of 500 architecture and validation
Figure 12: Mean Kendall's  $\tau$  rank correlation with  $\pm 1$  standard error achieved by a GP with various kernels.
Figure 13: Impact of flexible parameterization of convolutional blocks in the hierarchical NAS-Bench-201 search space.
error pairs, respectively. We recorded Kendall's  $\tau$  rank correlation between the predicted and true validation error.
Figure 12 clearly shows that the incorporation of the hierarchical information in the kernel design improves performance modeling. This is especially pronounced for smaller amounts of seen architectures.
What is the impact of flexible parameterization of convolutional blocks? To investigate the impact of the flexible parameterization of the convolutional blocks (i.e., activation functions, normalizations, and type of convolution), we removed the flexible parameterization and allowed only the same primitive computations as in the cell-based NAS-Bench-201 search space, while still searching over the macro architecture. More explicitly, we only allow ReLU non-linearity as the activation function, batch normalization as the normalization, and  $1 \times 1$  or  $3 \times 3$  convolutions. Figure 13 shows that for all datasets except CIFAR-100, flexible parameterization of the convolutional blocks improves performance of the found architectures. Interestingly, we find an architecture on CIFAR-100, which achieves  $26.24\%$  test error with  $1.307\mathrm{MB}$  and  $167.172\mathrm{M}$  number of parameters or FLOPs, respectively. This architecture is superior to the optimal architecture in the cell-based NAS-Bench-201 search space  $(26.49\%)$ . Note that this architecture is also Pareto-optimal for test error vs. number of parameters and test error vs. number of FLOPs.
Do zero-cost proxies also work in vast hierarchical search spaces? To assess zero-cost proxies, we re-used the data from the search runs and recorded Kendall's  $\tau$  rank correlation. Table 7 shows that nwot is surprisingly the best-performing non-baseline zero-cost proxy, even though it is specifically crafted for architectures with ReLU non-linearities. Interestingly, the baseline zero-cost proxies often yield better results than other non-baseline zero-cost proxies. Thus, expressive hierarchical search spaces bring new challenges for the design of zero-cost proxies.
Search space comparisons Supplementary to Figure 3, Figure 14 compares the search space variants hierarchical vs. fixed+shared (cell-based) vs. hierarchical+shared+nonlinear using RS, RE, and NASBOWL. The cell-based search space design shows on par or superior performance on all datasets except for CIFARTile for the three search strategies. This is in stark contrast to the results using BOHNAS (Figure 3). It shows that more expressive search spaces do not necessitate
Table 7: Kendall's  $\tau$  rank correlation of zero-cost-proxies on our hierarchical NAS-Bench-201 space. Note that zen and nwot may not be suitable for our hierarchical search space, as they are specifically designed for architectures with batch normalization and ReLU non-linearity, respectively. We report exemplary search results for NASWOT [65] in Table 6.
Zero-cost proxy	Type	CIFAR-10	CIFAR-100	ImageNet-16-120	CIFARTile	AddNIST
plain [66]	Baseline	-0.01	0.0	-0.08	0.02	-0.04
params [67]	Baseline	0.05	0.24	0.37	0.09	0.14
flops [67]	Baseline	0.16	0.38	0.38	0.36	0.33
l2-norm [66]	Baseline	0.58	0.42	0.29	0.44	0.63
zen-score [101]	Piece. Lin.	0.54	0.5	0.42	0.27	0.45
fisher [102]	Pruning-at-init	-0.24	-0.21	-0.13	-0.12	0.01
grad-norm [66]	Pruning-at-init	-0.17	-0.14	-0.08	-0.01	0.06
grasp [103]	Pruning-at-init	-0.02	0.07	0.05	-0.01	-0.0
snip [104]	Pruning-at-init	-0.1	-0.15	-0.11	0.05	0.13
synflow [105]	Pruning-at-init	0.09	0.35	0.42	-0.21	-0.11
epe-nas [106]	Jacobian	0.11	0.17	0.26	0.15	0.03
jacov [65]	Jacobian	0.3	0.29	0.2	0.21	0.17
nwot [65]	Jacobian	0.36	0.58	0.73	0.46	0.21
Table 8: Vocabulary for the binary and unary operations in the activation function search space. The symbol  $\beta$  indicates a per-channel trainable parameter and  $\sigma(\cdot)$  is the sigmoid function.
add	x1+x2	bgaussian_sq	exp(-β(x1-x2)2)	cubic	x3	cos	cos x	umax	max(x,0)
multi	x1·x2	bgaussian_abs	exp(-β|x1-x2|)	square_root	√x	sinh	sinh x	umin	min(x,0)
sub	x1-x2	wavg	βx1+(1-β)x2	mconst	β	cosh	cosh(x)	sigmoid	σ(x)
div	x1/x2+ε	id	x	aconst	x+β	tanh	tanh(x)	logexp	log(1+exp(x))
bmax	max(x1,x2)	neg	-x	log	log(|x|+ε)	asinh	sinh⁻¹(x)	gaussian	exp(-x²)
bmin	min(x1,x2)	abs	|x|	exp	exp(x)	atanh	tanh⁻¹(x)	erf	erf(x)
bsigmoid	σ(x1)·x2	square	x²	sin	sin(x)	sinc	sinc(x)	const	β
an improvement in performance since a search strategy also needs to be able to such well-performing architectures. It also shows the superiority of BOHNAS over the other NAS approaches (Q1) and provides further evidence that the incorporation of hierarchical information is a key contributor for the search efficiency (Q2). Based on this, we believe that future work using, e.g., graph neural networks as a surrogate, may benefit from also incorporating hierarchical information.
For the hierarchical search space with more non-linear macro topologies (instead of linear macro topologies) and a single, shared cell (hierarchical+shared+non-linear), we find a similar pattern as in Figure 3. We believe that one reason for this is that the architectural regularization (sharing of a single cell) reduces search complexity.
K Searching for activation functions
K.1 Search space
To search for activation functions, we adopted the search space from Ramachandran et al. [64]. More specifically, we define the search space as follows:
$$
\begin{array}{l} \mathrm {L} 2 := \text {B i n T o p o} (\mathrm {L} 1, \mathrm {L} 1, \text {B I N O P}) \mid \text {U n T o p o} (\mathrm {L} 1) \\ \mathrm {L} 1 := \text {B i n T o p o (U N O P , U N O P , B I N O P)} \mid \text {U n T o p o (U N O P)} \\ \begin{array}{l} \text {B I N O P} := \quad \text {a d d} \mid \text {m u l t i} \mid \text {s u b} \mid \text {d i v} \mid \text {b m a x} \mid \text {b m i n} \mid \text {b s i g m o i d} \mid \text {b g a u s s i a n} _ {\text {s q}} \\ \mid \text {b g a u s s i a n} _ {\text {a b s}} \mid \text {w a v g} \end{array} \tag {34} \\ \text {U N O P} := \text {i d} | \text {n e g} | \text {a b s} | \text {s q u a r e} | \text {c u b i c} | \text {s q u e r e _ {r o o t}} | \text {m c o n s t} | \text {a c o n s t} \\ | \log | \exp | \sin | \cos | \sinh | \cosh | \tanh | \operatorname {a s i n h} | \operatorname {a t a n h} | \operatorname {s i n c} \\ | \text {u m a x} | \text {u m i n} | \text {s i g m o d} | \text {l o g e x p} | \text {g a u s s i a n} | \text {e r f} | \text {c o n s t} , \\ \end{array}
$$
where the vocabulary for the binary operations (BINOP) and unary operations (UNOP) is provided in Table 8 and BinTopo and UnTopo define the topological operators of binary or unary operations, respectively. The search space comprises ca.  $10^{8.6}$  potential activation functions. Note that the search space size can be varied by adding more levels or can be made open-ended by using a recursive formulation, i.e.,  $\mathrm{L} := \mathrm{Bin0p}(\mathrm{L},\mathrm{L})\mid \mathrm{Un0p}(\mathrm{L})\mid$  id.
(c) NASBOWL.
Figure 14: Comparison of mean and  $\pm 1$  validation error of hierarchical, hierarchical with shared cell, and cell-based search space designs using Random Search (RS) (a), Regularized Evolution (RE) (b), and NASBOWL (c) for search.
K.2 Training details
Similar to Ramachandran et al. [64], we trained a ResNet-20 [100] on CIFAR-10 [93]. Specifically, we trained the network with SGD with learning rate of 0.1, momentum of 0.9, and weight decay of 0.0001, with a batch size of 128 for 64K steps. The learning rate is divided by 10 after 32K and 48K steps. We used random flip with probability 0.5 followed by a random crop (32x32 with 4 pixel padding) and normalization. We split the training data into 45K training or 5K validation samples, respectively, and report the final test error on the unseen test set (10K samples).
K.3 Supplementary search results
Choice of search strategies We only ran experiments using BOHNAS, RS, RE, and NASBOWL. We did not ran experiments using AREA since its zero-cost proxy specifically uses the binary codes of the ReLU non-linearity. We did not ran NASBOT since it already performed poorly on the hierarchical NAS-Bench-201 search space.
Search costs The search cost was ca. 5 GPU days using eight asynchronous workers, each with an NVIDIA RTX 2080 Ti GPU, for ca. 40 GPU days in total.
Best found activation function Our best found activation function for ResNet-20 is as follows:
$$
\beta (\sigma (- x) \cdot \min  (x, 0)) + (1 - \beta) (\min  (\max  (x, 0), \operatorname {e r f} x)) \quad . \tag {35}
$$
(a) Best found activation function.
(b) Common activation functions.
Figure 15: Visualization of our best found activation function (see Equation 35) for different values of the trainable parameter  $\beta$  (left) and common activation function from the literature (right).
Figure 15 visualizes and compares the found activation function with common activation functions from the literature. The found activation function is a weighted mixture of the sigmoid function and ReLU non-linearity.
L Searching for convolutional cells in the DARTS search space
While our main focus is to go beyond cell-based search spaces in this work, we also demonstrate that our proposed search strategy BOHNAS works in popular cell-based search spaces, e.g., cell-based NAS-Bench-201 [58] in Section 6. In addition, we also evaluated BOHNAS on the popular DARTS search space [6]. The DARTS search space definition is provided in Equation 13 (Appendix F). Following Ru et al. [48], we used the ENAS-style node-attributed DAG representation to apply the WL graph kernel on the normal or reduction cells, respectively.
L.1 Implementation details
In contrast to all other experiments, we set the crossover probability to  $p_{cross} = 0.0$  (and consequently  $p_{mut} = 1.0$ ), as Schneider et al. [107] showed the importance of mutation<sup>3</sup> for the performance of BANANAS [47] on NAS-Bench-301 (surrogate benchmark of the DARTS search space, Zela et al. [108]).
L.2 Training details
Search settings Following Liu et al. [6], we split the CIFAR-10 training dataset into a train and validation set with 25k samples each. We trained a small network with 8 cells for 50 epochs, with a batch size of 64 and set the initial number of channels to 16. We used SGD with learning rate of 0.025 with cosine annealing Loshchilov and Hutter [96], momentum of 0.9, and weight decay of 0.0003. Since the validation performance on CIFAR-10 is very volatile, we averaged the final 5 validation errors, following Ru et al. [48]. In contrast to previous works, we picked the best normal and reduction cells based on the validation performance from the different search runs with random seeds \{666, 777, 888, 999\}, and did not pick the cell-based on its validation performance by training it from scratch, e.g., for a short period of 100 epochs [6].
Retrain settings Following Liu et al. [6], we retrained the a large network with 20 cells for 600 epochs with batch size 96. The initial channels were set to 36. Other hyperparameters are same as used during search. We also applied cutout and auxiliary towers with weight 0.4. Following Chen et al. [45], we set the path dropout probability to 0.3 (instead of 0.2). We used the entire CIFAR-10 training set (50k samples) for training and report the final accuracy on the test set (10k samples). We evaluated the architecture with our discovered normal and reduction cells for the same random seeds used for search. We additionally also retrained the reported architectures of NASBOWL [48],
3In their implementation only the incumbent is mutated, whereas we still ran evolution over multiple steps to remain consistent across our experiments.
(a) Normal cell.
(b) Reduction cell.
Figure 16: Normal and reduction cells discovered by BOHNAS on the DARTS search space on CIFAR-10.
DrNAS [45], and Shapley-NAS [89] to reduce effects of possible confounding factors such as choice of random seeds, software versions, hardware, etc.
To assess the transferability of our found normal and reduction cell on CIFAR-10, we trained the best cell on ImageNet following the mobile setting by Liu et al. [6], where the input image size is  $224 \times 224$  and the number of multiply-add operations is restricted to be less than  $600\mathrm{M}$ . For training, we used the same random seeds used for search on CIFAR-10. Due to the high training cost, we omitted retraining of reported cells of prior works.
L.3 Search results
Search costs We ran search with eight asynchronous workers, each with an NVIDIA RTX 2080 Ti GPU, for exactly 1 GPU day, resulting in a total search cost of 8 GPU days. Note that by using the extensive parallel computing power available in today's computing clusters, the wallclock time for search (1 day) is comparable to that of state-of-the-art gradient-based methods. Retraining on CIFAR-10/ImageNet required ca. 1.5/5 GPU days on a single/eight NVIDIA RTX 2080 Ti GPU(s).
CIFAR-10 Results Figure 16 shows the best found normal and reduction cell. Table 9 shows that BOHNAS can find well-performing normal and reduction cells that are on-par or (mostly) superior to other sample-based search strategies. The discovered normal and reduction cells are also competitive with state-of-the-art gradient-based methods, which are (typically) optimized for this particular search space.
ImageNet results Table 10 shows that BOHNAS achieves competitive (or superior) results with  $24.55\%$  and  $7.4\%$  average top-1 or top-5 test error, respectively. This shows the transferability of the found normal and reduction cells by BOHNAS. The discovered normal and reduction cells are also superior to state-of-the-art gradient-based methods in the transfer setting<sup>4</sup>, which are actually optimized for this particular search space since successive works target the DARTS search space. BOHNAS is only inferior to gradient-based methods that searched for the cells on ImageNet.
M Searching for transformers for generative language modeling and sentiment analysis
To demonstrate that our approach is also applicable to diverse architecture types and tasks, we searched for transformers for language modeling (nanoGPT experiment) as well as sentiment analysis.
4Note that for fair comparison, we compare only reproduced numbers to avoid confounding factors.
Table 9: Comparison with state-of-the-art image classifiers on CIFAR-10 on the DARTS search space.
Search strategy	Type	Avg. val error [%]	Avg. test error [%]	Best test error [%]	Params [M]	Search cost [GPU days]
ASHA [109]	RS with early stopping	-	3.03 ± 0.13	2.85	4.6	9
Random-WS [7]	RS	-	2.85 ± 0.08	2.71	4.3	10
ENAS* [43]	RL	-	-	2.89	4.6	6
AmoebaNet-A [12]	evolution	-	3.34 ± 0.06	-	3.2	3150
AmoebaNet-B [12]	evolution	-	2.55 ± 0.05	-	2.8	3150
GP-NAS [110]	BO	-	-	3.79	3.9	1
BANANAS [47]	BO	-	2.64	2.57	-	12
BONAS-C [52]	BO	-	-	2.46	3.48	7.5
BONAS-D [52]	BO	-	-	2.43	3.3	10
NASBOWL [48]	BO	-	2.61 ± 0.08	2.50	3.7	3
NASBOWL† [48]	BO	11.84 ± 0.15	2.93 ± 0.1	2.83	3.7	N/A
DARTS (v2) [6]	gradient	-	2.76 ± 0.09	-	3.3	4
GDAS [58]	gradient	-	-	2.93	3.4	0.3
DrNAS [45]	gradient	-	2.46 ± 0.03	-	4.1	0.6
DrNAS† [45]	gradient	12.19 ± 0.06	2.65 ± 0.09	2.51	4.1	N/A
Shapley-NAS [89]	gradient	-	2.47 ± 0.04	2.43	3.4/3.6	0.3
Shapley-NAS† [89]	gradient	12.35 ± 0.22	3.09 ± 0.07	2.97	3.6	N/A
BOHNAS	BO	11.27 ± 0.16	2.68 ± 0.12	2.55	3.9	8
* expanded search space from DARTS. † reproduced results using the reported genotypes. - not reported.
M.1 Training details
nanoGPT We followed the training protocol for Shakespearean works [98] of Karpathy [68] to train a character-based language model. Specifically, we trained the networks for 5000 iterations with batch size 64 with 100 warmup iterations and AdamW optimizer [96] with learning rate of 0.001, weight decay of 0.1,  $\beta_{1}$  of 0.9, and  $\beta_{2}$  of 0.95 with decaying learning rate schedule. We equally split the data into training and validation data. The vocabulary is of size of 65. We set the block size to 256 and used a dropout rate of 0.2.
Sentiment analysis For sentiment analysis on IMDb [99], we used a frozen BERT encoder [117] and searched for a classifier head (see Appendix M.2 for details). We trained the classifier head for five epochs with a batch size of 128 with the Adam optimizer with learning rate of 0.001,  $\beta_{1}$  of 0.9, and  $\beta_{2}$  of 0.999. We used a dropout rate of 0.25 and hidden dimensionality of 256.
M.2 Search space definitions
nanoGPT We constructed the search space around the default transformer architecture of Karpathy [68], excluding the embedding layers and the final couple of layers (layer normalization followed by a linear layer) for simplicity. The search space is defined as follows:
$$
\begin{array}{l} S := \text {S e q u e n t i a l 3} (L, L, L) \mid \text {S e q u e n t i a l 2} (L, L) \mid \text {R e s i d u a l 2} (L, L, L) \\ L := \text {S e q u e n t i a l 2} (T, T) \mid \text {S e q u e n t i a l 3} (T, T, T) \mid \text {R e s i d u a l 2} (T, T, T) \\ \mathrm {T} := \text {T r a n s f o r m e r B l o c k} (\text {N H E A D S}, \text {M L P R A T I O}, \text {A C T}) \tag {36} \\ \end{array}
$$
$$
\begin{array}{l} \text {N H E A D S} := 6 \mid 4 \mid 8 \\ \text {M L P R A T I O} := 4 \mid 2 \mid 3 \\ \text {A C T} := \text {g e l u} | \text {r e l u} | \text {s i l u}. \\ \end{array}
$$
Additionally, we searched for the embedding dimensionality  $\{96, 192, 384\}$ . However, since the embedding dimensionality acts globally on the architecture, we treated it as a categorical hyperparameter. To this end, we combined our hierarchical graph kernel (hWL) with a Hamming kernel in our search strategy, BOHNAS. This shows that both our search space design framework as well as search strategy, BOHNAS, can also be used for joint NAS and hyperparameter optimization.
Table 10: Comparison with state-of-the-art image classifiers on ImageNet in the mobile setting on the DARTS search space. We reproduced results for DrNAS [45] and ShapleyNAS [89] using the reported genotype on one seed (777).
Search strategy	Type	Top-1 test error [%]	Top-5 test error [%]	Params [M]	Search cost [GPU days]
Inception-v1 [111]	manual	30.1	10.1	6.6	N/A
MobileNet [112]	manual	29.4	10.5	4.2	N/A
ShuffleNet 2× (v1) [113]	manual	26.4	10.2	~ 5	N/A
ShuffleNet 2× (v1) [114]	manual	25.1	-	~ 5	N/A
NASNet-A [23]	RL	26.0	8.4	5.3	2000
AmoebaNet-C [12]	evolution	24.3	7.6	6.4	3150
PNAS [115]	BO	25.8	8.1	5.1	255
BONAS-C [52]	BO	24.6	7.5	5.1	7.5
BONAS-D [52]	BO	25.4	8.0	4.8	10
MnasNet-92 [55]	RL	25.2	8.0	4.4	-
Random-WS† [7]	RS	25.3 ± 0.25	92.2 ± 0.15	5.6 ± 0.1	-
DARTS (v2) [6]	gradient	26.7	8.7	4.7	1
GDAS [58]	gradient	26.0	8.5	5.3	0.3
PC-DARTS [116]	gradient	25.1	7.8	3.6	0.1
P-DARTS† [116]	gradient	24.6	7.5	4.9	N/A
P-DARTS [116]	gradient	24.4	7.4	4.9	0.3
DrNAS*† [45]	gradient	24.3	7.4	5.7	N/A
DrNAS* [45]	gradient	23.7	7.1	5.7	4.6
Shapley-NAS† [89]	gradient	24.9	7.6	5.1	N/A
Shapley-NAS [89]	gradient	24.3	7.6	5.1	0.3
Shapley-NAS*† [89]	gradient	25.0	7.6	5.4	N/A
Shapley-NAS* [89]	gradient	23.9	-	5.4	4.2
BOHNAS	BO	24.55 ± 0.03	7.44 ± 0.04	5.4	8
BOHNAS (best)	BO	24.48	7.3	5.4	8
* search run on ImageNet, otherwise on CIFAR-10. † reproduced results using the reported genotypes. - not reported.
Figure 17: Best found transformer architecture (excluding embedding layers and final few layers) with an embedding dimensionality of 192 for the nanoGPT experiment.
Sentiment analysis We searched for a classifier head on top of the BERT encoder [117] followed by a linear projection. Specifically, we define the search space as follows:
$$
\text {C E L L} := \text {C e l l} (\text {O P S}, \text {O P S}, \text {O P S}, \text {O P S}, \text {O P S}, \text {O P S}) \tag {37}
$$
$$
O P S := i d \mid z e r o \mid t r a n s f o r m e r \mid g r u \mid m l p ,
$$
where the topological operator (CELL) corresponds to a densely-connected four node DAG (similar to the NAS-Bench-201 cell-based search space [58]), and transformer, gru, and mlp are a transformer block, GRU unit, or two-layer MLP with GeLU non-linearity, respectively. We feed the output of the searched classifier cell into a GRU unit followed by dropout (with a rate of 0.25) and final linear layer.
M.3 Search results
Search cost We searched with eight asynchronous workers, each with an NVIDIA RTX 2080 Ti GPU, for exactly 1 GPU day, resulting in a total search cost of 8 GPU days.
Found transformer and classifier head We depict the found transformer for the nanoGPT experiment in Figure 17. Figure 18 depicts the found classifier head for sentiment analysis.
Figure 18: Found classifier head for the sentiment analysis experiment.
N Best practices checklist for NAS research
NAS research has faced challenges with reproducibility and fair empirical comparisons for a long time [109, 118]. In order to promote fair and reproducible NAS research, Lindauer and Hutter [13] created a best practices checklist for NAS research. Below, we address all points on the checklist.
1. Best Practices for Releasing Code
(a) Did you release code for the training pipeline used to evaluate the final architectures?, Did you release code for the search space?, Did you release code for your NAS method? [Yes] All code can be found at https://github.com/automl/hierarchical_nas_construction.
(b) Did you release the hyperparameters used for the final evaluation pipeline, as well as random seeds? [Yes] We provide experimental details (including hyperparameters as well as random seeds) in Section 6, Appendices J.1, K.2, L.2 and M.1.
(c) Did you release hyperparameters for your NAS method, as well as random seeds? [Yes] We discuss implementation details (including hyperparameters as well as random seeds) of our NAS method in Section 6, Appendices I, L.1 and M.2.
2. Best practices for comparing NAS methods