SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 2 Kriging Believer algorithm to select one batch of points.
Input: Observation data  $\mathcal{D}_t$  , batch size  $b$    
Output: The batch points  $\mathcal{B}_{t + 1} = \{\mathbf{x}_{t + 1}^{(1)},\dots ,\mathbf{x}_{t + 1}^{(b)}\}$ $\tilde{\mathcal{D}}_t = \mathcal{D}_t\cup \tilde{\mathcal{D}}_p$    
for  $j = 1,\ldots ,b$  do Select the next  $\mathbf{x}_{t + 1}^{(j)}$  by maximizing acquisition function  $\alpha (\mathbf{x}|\tilde{\mathcal{D}}_t)$  Compute the predictive posterior mean  $\mu (\mathbf{x}_{t + 1}^{(j)}|\tilde{\mathcal{D}}_t)$ $\tilde{\mathcal{D}}_t\gets \tilde{\mathcal{D}}_t\cup (\mathbf{x}_{t + 1},\mu (\mathbf{x}_{t + 1}^{(j)}|\tilde{\mathcal{D}}_t))$    
end for
where  $\mu (\mathbf{x}|\mathcal{D}_t)$  and  $\sigma^2 (\mathbf{x}|\mathcal{D}_t)$  are the mean and variance of the predictive posterior distribution at a candidate point  $\mathbf{x}$ , and  $\phi (\cdot)$  and  $\Phi (\cdot)$  denote the PDF and CDF of the standard normal distribution, respectively.
To make use of ample distributed computing resource, we adopted Kriging Believer [63] which uses the predictive posterior of the surrogate model to assign hallucinated function values  $\{\tilde{f}_p\}_{p\in \{1,\dots ,P\}}$  to the  $P$  candidate points with pending evaluations  $\{\tilde{\mathbf{x}}_p\}_{p\in \{1,\dots ,P\}}$  and perform next BO recommendation in the batch by pseudo-augmenting the observation data with  $\tilde{\mathcal{D}}_p = \{(\tilde{\mathbf{x}}_p,\tilde{f}_p)\}_{p\in \{1,\dots ,P\}}$  namely  $\tilde{\mathcal{D}}_t = \mathcal{D}_t\cup \tilde{\mathcal{D}}_p$ . The algorithm of Kriging Believer at one BO iteration to select a batch of recommended candidate points is summarized in Algorithm 2.
H.2 Hierarchical Weisfeiler-Lehman kernel
Inspired by Ru et al. [48], we adopted the Weisfeiler-Lehman (WL) graph kernel [62] in the GP surrogate model to handle the graph nature of neural architectures. The basic idea of the WL kernel is to first compare node labels, and then iteratively aggregate labels of neighboring nodes, compress them into a new label and compare them. Algorithm 3 summarizes the WL kernel procedure.
Ru et al. [48] identified three reasons for using the WL kernel: (1) it is able to compare labeled and directed graphs of different sizes, (2) it is expressive, and (3) it is relatively efficient and scalable. Hierarchical search spaces can express a diverse spectrum of neural architectures with very heterogeneous topological structure. Therefore, reason (1) is a very important property of the WL kernel to account for the diversity of neural architectures and makes other kernels (e.g., adjacency [50] or path encoding [47]) not applicable. Moreover, if we allow for many hierarchical levels, we can construct very large neural architectures. Therefore, reasons (2) and (3) are essential for accurate and fast modeling.
However, modeling solely based on the (standard) WL graph kernel neglects the useful hierarchical information from our construction process. Moreover, the large size of neural architectures makes it still challenging to capture the more global topological patterns. We therefore propose to use hierarchical information through a hierarchy of WL graph kernels that take into account the different granularities of the architectures and combine them in a weighted sum. To obtain the different granularities, we use the fold operators  $F_{l}$  that removes parts beyond the  $l$ -th hierarchical level. Thereby, we obtain the folds
$F_{3}(\omega) = \omega = \text{Sequential}(\text{Residual}(\text{conv}, \text{id}, \text{conv}), \text{Residual}(\text{conv}, \text{id}, \text{conv}), \text{linear})$ ,  $F_{2}(\omega) = \text{Sequential}(\text{Residual}, \text{Residual}, \text{linear})$ ,  $F_{1}(\omega) = \text{Sequential}$ ,
Algorithm 3 Weisfeiler-Lehman subtree kernel computation [62].
Input: Graphs  $G_{1}, G_{2}$ , maximum iterations  $H$   
Output: Kernel function value between the graphs  
Initialize the feature vectors  $\phi(G_{1}) = \phi_{0}(G_{1}), \phi(G_{2}) = \phi_{0}(G_{2})$  with the respective counts of original node labels (i.e., the  $h = 0$  WL features)  
for  $h = 1, \ldots, H$  do  
    Assign a multiset  $M_{h}(v) = \{l_{h-1}(u) | u \in \mathcal{N}(v)\}$  to each node  $v \in G$ , where  $l_{h-1}$  is the node label function of the  $h-1$ -th WL iteration and  $\mathcal{N}$  is the node neighbor function  
    Sort elements in multiset  $M_{h}(v)$  and concatenate them to string  $s_{h}(v)$   
    Compress each string  $s_{h}(v)$  using the hash function  $f$  s.t.  $f(s_{h}(v)) = f(s_{h}(w)) \iff s_{h}(v) = s_{h}(u)$   
    Add  $l_{h-1}$  as prefix for  $s_{h}(v)$   
    Concatenate the WL features  $\phi_{h}(G_{1}), \phi_{h}(G_{2})$  with the counts of the new labels:  $\phi(G_{1}) = [\phi(G_{1}), \phi_{h}(G_{1})], \phi(G_{2}) = [\phi(G_{2}), \phi_{h}(G_{2})]$   
    Set  $l_{h}(v) := f(s_{h}(v)) \forall v \in G$   
end for  
Compute inner product  $k = \langle \phi_{h}(G_{1}), \phi_{h}(G_{2}) \rangle$  between WL features  $\phi_{h}(G_{1}), \phi_{h}(G_{2})$  in RKHS  $\mathcal{H}$
Figure 8: Labeled computational graphs  $\Phi(F_2)$  and  $\Phi(F_3)$  of the folds  $F_2$  and  $F_3$ , respectively.
for the string  $\omega$  from Equation 1. Figure 8 visualizes the labeled computational graphs  $\Phi(F_2)$  and  $\Phi(F_3)$  of the folds  $F_2$  or  $F_3$ , respectively. Note that we ignore the first fold since it does not represent a labeled DAG. Also, note that we additionally add the previous nonterminals to provide additional information about the construction history. These graphs can be fed into (standard) WL graph kernels. Therefore, we can construct a hierarchy of WL graph kernels  $k_{WL}$  as follows:
$$
k _ {h W L} \left(\omega_ {i}, \omega_ {j}\right) = \sum_ {l = 2} ^ {L} \lambda_ {l} \cdot k _ {W L} \left(\Phi \left(F _ {l} \left(\omega_ {i}\right), \Phi \left(F _ {l} \left(\omega_ {j}\right)\right)\right) \quad , \right. \tag {22}
$$
where  $\omega_{i}$  and  $\omega_{j}$  are two strings. Note that  $\lambda_{l}$  governs the importance of the learned graph information across the hierarchical levels and can be optimized through the marginal likelihood. Note that while the hierarchical kernel structure seems superfluous for this example, for search spaces with many hierarchical levels it greatly improves the ability of the kernel to capture the more topological patterns (Section 6).
H.3 Examples for the evolutionary operations
For the evolutionary operations, we adopted ideas from grammar-based genetic programming [60, 61]. In the following, we will show how these evolutionary operations manipulate the function composition of an architecture, e.g.,
Sequential(Residual(conv, id, conv), Residual(conv, id, conv), linear) (23)
from the search space
S ::= Sequential(S, S, S) | Residual(S, S, S) | conv | id | linear. (24)
Figure 1 shows how we can derive the function composition in Equation 23 from the search space in Equation 24. For mutation operations, we first randomly pick a substring (i.e., part of the composition), e.g., Residual(conv, id, conv). Then, we randomly sample a new function composition with the
same nonterminal symbol S as start symbol, e.g., Sequential(conv, id, linear), and replace the previous substring, yielding
$$
\text {S e q u e n t i a l} (\text {S e q u e n t i a l} (\text {c o n v}, \text {i d}, \text {l i n e a r}), \text {R e s i d u a l} (\text {c o n v}, \text {i d}, \text {c o n v}), \text {l i n e a r}) \tag {25}
$$
For (self-)crossover, we swap the two substrings, e.g., Residual(conv, id, conv) and Residual(conv, id, conv) with the same nonterminal S as start symbol, yielding
$$
\text {S e q u e n t i a l} (\text {R e s i d u a l} (\text {c o n v}, \text {i d}, \text {c o n v}), \text {R e s i d u a l} (\text {c o n v}, \text {i d}, \text {c o n v}), \text {l i n e a r}) \quad . \tag {26}
$$
Note that unlike the commonly used crossover operation, which uses two parents, self-crossover has only one parent. Future works could also consider a self-copy operation that copies a part of the function composition to another part of the (total) function composition of an architecture to explicitly regularizing diversity.
I Implementation details of the search strategies and dataset details
BOHNAS, NASBOWL, and NASBOT We used the same hyperparameters for the three BO strategies. We ran the BO approaches asynchronously in parallel throughout our experiments with a batch size of  $B = 1$ , i.e., at each BO iteration a single architecture is proposed for evaluation. We used 10 or 50 random samples to initialize the BO strategies for the architecture (hierarchical NAS-Bench-201, DARTS, transformer) or activation function search spaces, respectively. For the acquisition function optimization, we used a pool size of  $P = 200$ , where the initial population consisted of the current ten best-performing architectures and the remainder were randomly sampled architectures to encourage exploration in the huge search spaces. During evolution, the mutation probability was set to  $p_{mut} = 0.5$  and crossover probability was set to  $p_{cross} = 0.5$ . From the crossovers, half of them were self-crossovers of one parent and the other half were common crossovers between two parents. The tournament selection probability was set to  $p_{tour} = 0.2$ . We evolved the population at least for ten iterations and a maximum of 50 iterations using a early stopping criterion based on the fitness value improvements over the last five iterations.
Regularized Evolution (RE) and AREA RE [9, 12] and AREA [65] iteratively mutate the best architectures out of a sample of the population. We reduced the population size from 50 to 30 to account for fewer evaluations in the architecture search experiments, and used a sample size of 10. For AREA, we randomly sampled 60 candidates and selected the initial population (30) based on its zero-cost proxy score (nwot). We also ran RE and AREA asynchronously for better comparability.
J Searching the hierarchical NAS-Bench-201 search space
In this section, we provide training and dataset details (Appendix J.1), provide search space definitions of variants of the hierarchical NAS-Bench-201 search space (Appendix J.2) and provide supplementary results to Section 6 and conduct extensive analyses (Appendix J.3).
J.1 Training details
Training protocol We evaluated all search strategies on CIFAR-10/100 [93], ImageNet-16-120 [94], CIFARTile, and AddNIST [95]. Note that CIFARTile and AddNIST are novel datasets and therefore have not yet been optimized by the research community. We provide further dataset details below. For training of architectures on CIFAR-10/100 and ImageNet-16-120, we followed the training protocol of Dong and Yang [58]. We trained architectures with SGD with learning rate of 0.1, Nesterov momentum of 0.9, weight decay of 0.0005 with cosine annealing [96], and batch size of 256 for 200 epochs. The initial channels were set to 16. For both CIFAR-10 and CIFAR-100, we used random flip with probability 0.5 followed by a random crop (32x32 with 4 pixel padding) and normalization. For ImageNet-16-120, we used a 16x16 random crop with 2 pixel padding instead.
For training of architectures on AddNIST and CIFARTile, we followed the training protocol from the CVPR-NAS 2021 competition [95]: We trained architectures with SGD with learning rate of 0.01, momentum of 0.9, and weight decay of 0.0003 with cosine annealing, and batch size of 64 for 64 epochs. We set the initial channels to 16 and did not apply any further data augmentation.
Table 5: Licenses for the datasets we used in our experiments.
Dataset	License	URL
CIFAR-10 [93]	MIT	https://www.cs.toronto.edu/~kriz/cifar.html
CIFAR-100 [93]	MIT	https://www.cs.toronto.edu/~kriz/cifar.html
ImageNet-16-120 [94]	MIT	https://patrykchrabaszcz.github.io/Imagenet32/
CIFARTile [95]	GNU	https://github.com/RobGeada/cvpr-nas-datasets
AddNIST [95]	GNU	https://github.com/RobGeada/cvpr-nas-datasets
ImageNet [97]	Custom	https://www(image-net.org/index.php
Tiny Shakespeare [98]	MIT	https://github.com/karpathy/char-rnn
IMDb [99]	Custom	https://ai.stanford.edu/~amaas/data/sentiment/
Dataset details In Table 5, we provide the licenses for the datasets used in our experiments. For training of architectures on CIFAR-10, CIFAR-100 [93], and ImageNet-16-120 [94], we followed the dataset splits and training protocol of NAS-Bench-201 [58]. For CIFAR-10, we split the original training set into a new training set with  $25\mathrm{k}$  images and validation set with  $25\mathrm{k}$  images for search following [58]. The test set remained unchanged. For evaluation, we trained architectures on both the training and validation set. For CIFAR-100, the training set remained unchanged, but the test set was partitioned in a validation set and new test set each with  $5\mathrm{K}$  images. For ImageNet-16-120, all splits remained unchanged. For AddNIST and CIFARTile, we used the training, validation, and test splits as defined in the CVPR-NAS 2021 competition [95].
J.2 Derivatives of the hierarchical NAS-Bench-201 search space
We can derive several derivatives of the hierarchical NAS-Bench-201 search space (Section 4). This derivative let us study the impact of different parts of the architecture (e.g., branching of the macro architecture or repetition of the same architectural blocks) in Section 6. Below, we provide their formal definitions.
Derivative: fixed+shared (cell-based)
This derivative uses a fixed macro topology with a single, shared cell. Formally, we first define the fixed macro search space:
$$
D 2:: := \text {S e q u e n t i a l} 3 (D 1, D 1, D 0)
$$
$$
D 1 := \text {S e q u e n t i a l} 3 (C, C, D)
$$
$$
D 0 := \text {S e q u e n t i a l} 3 (C, C, S H A R E D) \tag {27}
$$
$$
D := \text {S e q u e n t i a l 2} (\text {S H A R E D}, \text {d o w n})
$$
$$
C := \text {S e q u e n t i a l 2} (\text {S H A R E D}, \text {S H A R E D})
$$
where the intermediate variable SHARED shares the cell across the fixed macro architecture and maps to a single instance of the cell-level space:
$$
\mathrm {C L} := \operatorname {C e l l} (\mathrm {O P}, \mathrm {O P}, \mathrm {O P}, \mathrm {O P}, \mathrm {O P}, \mathrm {O P})
$$
$$
O P \quad : := \quad z e r o \mid i d \mid C O N V B L O C K \mid a v g _ {-} p o o l
$$
$$
\begin{array}{r l} \text {C O N V B L O C K} & := \text {S e q u e n t i a l 3 (A C T , C O N V , N O R M)} \\ \text {A C T} & := \text {r e l a r} \end{array} \tag {28}
$$
$$
\text {O N V} := \operatorname {c o n v 1 x 1} | \operatorname {c o n v 3 x 3}
$$
$$
\begin{array}{l} \text {N O R M} := \text {b a t c h} \end{array} .
$$
Note that the cell-level space is equivalent to the original NAS-Bench-201 search space [58], and since the macro level search space is fixed to the same macro architecture used in the original NAS-Bench-201 search space, this derivative is consequently equivalent to the original NAS-Bench-201 search space.
Derivative: hierarchical+shared
For this derivative, we searched over the macro search space but limited the cell-level part to a single instance (a shared cell). Formally, we define macro-level of this derivative as follows:
D2 ::= Sequential13(D1, D1, D0) | Sequential13(D0, D1, D1) | Sequential14(D1, D1, D0, D0)  
D1 ::= Sequential13(C, C, D) | Sequential14(C, C, C, D) | Residual13(C, C, C, D)  
D0 ::= Sequential13(C, C, SHARED) | Sequential14(C, C, C, SHARED) | Residual13(C, C, SHARED, SHARED)  
D ::= Sequential12(SHARED, down) | Sequential13(SHARED, SHARED, down) | Residual12(SHARED, down, down)  
C ::= Sequential12(SHARED, SHARED) | Sequential13(SHARED, SHARED, SHARED) | Residual12(SHARED, SHARED, SHARED)
where the intermediate variable SHARED shares the cell across the macro architecture and maps to a single instance of the cell-level space:
$$
\begin{array}{l} \text {C O N V B L O C K} := \text {S e q u e n t i a l 3 (A C T , C O N V , N O R M)} \\ \text {A C T} = \left\{ \begin{array}{l} \text {a l p h a} _ {1}, \text {a l p h a} _ {2}, \dots , \text {a l p h a} _ {n} \end{array} \right. \end{array} \tag {30}
$$
$$
\text {A C T} := \text {r e l u} | \text {h a r d s w i s h} | \text {m i s h}
$$
$$
\text {C O N V} := \operatorname {c o n v 1 x 1} | \operatorname {c o n v 3 x 3} | \operatorname {d c o n v 3 x 3}
$$
$$
\text {N O R M} \quad := \quad \text {b a t c h} \mid \text {i n s t a n c e} \mid \text {l a y e r}
$$
Derivative: hierarchical+non-linear
For this derivative, we replaced some of the linear macro topologies (Sequential) with non-linear macro topologies (Diamond); see Figure 5 for the topological operator Diamond. This results in architectures with more branching at the macro-level. Formally, we define this derivative as follows:
D2 := Sequential3(D1, D1, D0) | Sequential3(D0, D1, D1) | Sequential4(D1, D1, D0, D0)  
D1 := Sequential3(C, C, D) | Residual3(C, C, D, D) | Diamond3(C, C, C, C, D, D)  
D0 := Sequential3(C, C, CL) | Residual3(C, C, CL, CL) | Diamond3(C, C, C, C, CL, CL)  
D := Sequential2(CL, down) | Residual2(CL, down, down) | Diamond2(CL, CL, down, down)  
C := Sequential2(CL, CL) | Residual2(CL, CL, CL) | Diamond2(CL, CL, CL, CL)  
CL := Cell(OP, OP, OP, OP, OP, OP)  
OP := zero | id | CONVBLOCK | avg_pool  
CONVBLOCK := Sequential3(ACT, CONV, NORM)  
ACT := relu | hardswish | mish  
CONV := conv1x1 | conv3x3 | dconv3x3  
NORM := batch | instance | layer. (31)
Derivative: hierarchical+shared+non-linear
For this derivative, we replaced some of the linear macro topologies (Sequential) with non-linear macro topologies (Diamond). This results in architectures with more branching at the macro-level. Different to the derivative hierarchical+non-linear we used a single, shared cell. Formally, we
define this derivative as follows:
$$
\mathrm {D} 2 := \text {S e q u e n t i a l 3} (\mathrm {D} 1, \mathrm {D} 1, \mathrm {D} 0) \mid \text {S e q u e n t i a l 3} (\mathrm {D} 0, \mathrm {D} 1, \mathrm {D} 1) \mid \text {S e q u e n t i a l 4} (\mathrm {D} 1, \mathrm {D} 1, \mathrm {D} 0, \mathrm {D} 0)
$$
$$
D 1 := \text {S e q u e n t i a l} 3 (C, C, D) \mid \text {R e s i d u a l} 3 (C, C, D, D) \mid \text {D i a m o n d} 3 (C, C, C, C, D, D)
$$
$$
\begin{array}{l} \text {D 0 : = S e q u e n t i a l 3 (C , C , S H A R E D)} \mid \text {R e s i d u a l 3 (C , C , S H A R E D , S H A R E D)} \\ \mid \text {D i a m o n d 3 (C , C , C , C , S H A R E D , S H A R E D)} \end{array}
$$
$$
\begin{array}{l} \text {D} := \text {S e q u e n t i a l 2 (S H A R E D , d o w n)} | \text {R e s i d u a l 2 (S H A R E D , d o w n , d o w n)} \\ \mid \text {D i a m o n d 2 (S H A R E D , S H A R E D , d o w n , d o w n)} \end{array}
$$
$$
\begin{array}{l} \text {C} := \text {S e q u e n t i a l 2 (S H A R E D , S H A R E D)} \mid \text {R e s i d u a l 2 (S H A R E D , S H A R E D , S H A R E D)} \\ \mid \text {D i a m o n d 2 (S H A R E D , S H A R E D , S H A R E D , S H A R E D)} \quad , \end{array}
$$
$$
\mathrm {C L} := \mathrm {C e l l} (\mathrm {O P}, \mathrm {O P}, \mathrm {O P}, \mathrm {O P}, \mathrm {O P}, \mathrm {O P})
$$
$$
O P := \text {z e r o} \mid i d \mid C O N V B L O C K \mid a v g _ {-} p o o l
$$
$$
\begin{array}{l} \text {C O N V B L O C K} := \text {S e q u e n t i a l 3 (A C T , C O N V , N O R M)} \\ \text {A C T} = \left\{ \begin{array}{l} \text {b o l d s y m b o l {i n c}} \end{array} \right. \end{array} \tag {33}
$$
$$
\text {C O N V} := \text {c o n v 1 x 1} | \text {c o n v 3 x 3} | \text {d c o n v 3 x 3}
$$
J.3 Supplementary results and analyses
Search costs Search time varied across datasets from ca. 0.5 days (CIFAR-10) to ca. 1.8 days (ImageNet-16-120) using eight asynchronous workers, each with an NVIDIA RTX 2080 Ti GPU, for ca. 4 to ca. 14.4 GPU days in total.
Test errors We report the test errors of our best found architectures in Table 6. We observe that our search strategy BOHNAS finds the strongest performing architectures across all dataset on the hierarchical as well as cell-based search space. Note that we found, e.g., on ImageNet-16-120, an architecture which is on par with the optimal architecture of the cell-based NAS-Bench-201 search space (52.78% vs. 52.69%). When using BOHNAS in the hierarchical search space with more non-linear macro topologies and a single, shared cell (hierarchical+shared+non-linear) we can even better architectures, which improved the test error by 0.51%, 1.1%, and 4.99% on CIFAR-10, CIFAR-100, or ImageNet-16-120, respectively.
Figure 9 shows the test error vs. the number of parameters or FLOPs. Our best found architectures on hierarchical NAS-Bench-201 (hierarchical) fall well within the parameter and FLOPs ranges of the cell-based NAS-Bench-201 search space across all datasets, except for the parameters on CIFAR-10. Note that our best found architecture on ImageNet-16-120 on hierarchical NAS-Bench-201 is Pareto-optimal for test error vs. number of parameters and test error vs. number of FLOPs.
Best architectures Below we report the best found architecture per dataset on the hierarchical NAS-Bench-201 search space (Section 4) for each dataset. Figure 10 visualizes the novel and diverse design of the architectures (including stem and classifier head).
CIFAR-10 (mean test error  $5.65\%$ , #params 2.204 MB, FLOPs 127.673 M):
Sequential4(Residual3(Residual2(Cell(id, zero, Sequential1(Sequential3(hardswish, conv1x1, layer)), Sequential1(Sequential3(hardswish, conv3x3, layer)), zero, Sequential1(Sequential3(mish, conv3x3, instance))), Cell(Sequential1(Sequential3(relu, dconv3x3, layer)), id, avg_pool, Sequential1(Sequential3(relu, dconv3x3, layer)), id, zero), Cell(zero, Sequential1(Sequential3(relu, conv1x1, layer)), id, Sequential1(Sequential3(hardswish, conv1x1, instance)), Sequential1(Sequential3(hardswish, conv3x3, layer)), Sequential1(Sequential3(hardswish, dconv3x3, layer))), Residual2(Cell(id, zero, Sequential1(Sequential3(relu, conv1x1, layer)), Sequential1(Sequential3(mish, conv1x1, layer)), Sequential1(Sequential3(hardswish, conv3x3, layer)), zero), Cell(id, zero, id, Sequential1(Sequential3(relu, conv3x3, batch)), id, id), Cell(Sequential1(Sequential3(hardswish, conv3x3, layer)), Sequential1(Sequential3(relu, conv1x1, layer)), Sequential1(Sequential3(hardswish, conv1x1, layer)),
Table 6: Test errors (and ±1 standard error) of popular baseline architectures (e.g., ResNet [100] and EfficientNet [1] variants), and our best found architectures on the cell-based and hierarchical NAS-Bench-201 search space. Note that we picked the ResNet and EfficientNet variant based on the test error, consequently giving an overestimate of their test performance. We also compared to gradient-based approaches which cannot be applied to hierarchical search spaces without any novel adoption.
reported numbers.
* optimal numbers as reported in Dong and Yang [58].
best) test error (and  $\pm 1$  standard error) across three seeds {777, 888, 999} of the best architecture of the three search runs with lowest validation error.
test errors on the hierarchical search space variant with more non-linear macro topologies and a single, shared cell (hierarchical+shared+non-linear, refer to Appendix J.2 for its definition).