SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Grammar-guided acquisition function optimization Due to the discrete nature of the function compositions, we adopt ideas from grammar-based genetic programming [60, 61] for acquisition function optimization. For mutation, we randomly replace a substring (i.e., part of the function composition) with a new, randomly generated string with the same nonterminal as start symbol. For
Table 1: Test errors [%] of the best found architectures of BOHNAS (ours) compared with the respective best methods from the literature using the same training protocol. Table 6 (Appendix J.3), Table 2, and Tables 9 and 10 (Appendix L.3) provide the full tables for the NAS-Bench-201, activation function search, or DARTS training protocols, respectively.
Dataset Training prot.	C10 NB201†	C100 NB201†	IM16-120 NB201†	CTile NB201†	AddNIST NB201†	C10 Act. func.	C10 DARTS	IM DARTS (transfer)
Previous best BOHNAS	5.63	26.51	53.15	35.75	7.4	8.32	2.65*	24.63*
5.02	25.41	48.16	30.33	4.57	8.31	2.68	24.48
Difference	+0.51	+1.1	+4.99	+5.42	+2.83	+0.01	-0.03	+0.15
${}^{ \dagger  }$  includes hierarchical search space variants. * reproduced results using the reported genotype.
Figure 2: Mean  $\pm$  standard error on the hierarchical NAS-Bench-201 search space. Appendix J.3 provides further results and extensive analyses.
crossover, we randomly swap two substrings produced by the same nonterminal as start symbol. We consider two crossover operators: a novel self-crossover operation swaps two substrings of the same string, and the common crossover operation swaps substrings of two different strings. Importantly, all evolutionary operations by design only result in valid function compositions (architectures) of the generated language. We provide visual examples for the evolutionary operations in Appendix H.
In our experiments, we used expected improvement as acquisition function and the Kriging Believer [63] to make use of parallel compute resources to reduce wallclock time. The Kriging Believer hallucinates function evaluations of pending evaluations at each iteration to avoid redundant evaluations.
In the section, we show the versatility of our search space design framework, show how we can study the impact of architectural design choices on performance, and show the search efficiency of our search strategy BOHNAS by answering the following research questions:
RQ1 How does our search strategy BOHNAS compare to other search strategies?
RQ2 How important is the incorporation of hierarchical information in the kernel design?
RQ3 How well do zero-cost proxies perform in large hierarchical search spaces?
RQ4 Can we find well-performing transformer architectures for, e.g., language modeling?
RQ5 Can we discover novel architectural patterns (e.g., activation functions) from scratch?
RQ6 Can we find better-performing architectures in huge hierarchical search spaces with a limited number of evaluations, despite search being more complex than, e.g., in cell-based spaces?
RQ7 How does the popular uniformity architecture design principle, i.e., repetition of similar architectural patterns, affect performance?
RQ8 Can we improve performance further by allowing for more non-linear macro architectures while still employing the uniformity architecture design principle?
To answer these questions, we conducted extensive experiments on the hierarchical NAS-Bench-201 (Section 4), activation function [64], DARTS [6], and newly designed transformer-based search spaces (Appendix M.2). The activation function search shows the versatility of our search space design framework, where we search not for architectures, but for the composition of simple mathematical
Figure 3: Mean  $\pm 1$  standard error for several architecture search space designs (see Section 4 and Appendix J.2 for their CFGs) using BOHNAS for search.
functions. The search on the DARTS search space shows that BOHNAS is also backwards-compatible with cell-based spaces and achieves on-par or even superior performance to state-of-the-art gradient-based methods; even though these methods are over-optimized for the DARTS search space due to successive works targeting the same space. Search on the transformer space further shows that our search space design framework is also capable to cover the ubiquitous transformer architecture. To promote reproducibility, we discuss adherence to the NAS research checklist in Appendix N. We provide supplementary results (and ablations) in Appendices J.3, K.3, L.3 and M.3.
6.1 Evaluation details
We search for a total of 100 or 1000 evaluations with a random initial design of 10 or 50 architectures on three seeds \{777, 888, 999\} or one seed \{777\} on the hierarchical NAS-Bench-201 or activation function search space, respectively. We followed the training protocols and experimental setups of Dong and Yang [58] or Ramachandran et al. [64]. For search on the DARTS space, we search for one day with a random initial design of 10 on four seeds \{666, 777, 888, 999\} and followed the training protocol and experimental setup of Liu et al. [6], Chen et al. [45]. For searches on transformer-based spaces, we ran experiments each for one day with a random initial design of 10 on one seed \{777\}. In each evaluation, we fully trained the architectures or activation functions (using ResNet-20) and recorded the final validation error. We provide full training details and the experimental setups for each space in Appendices J.1, K.2, L.2 and M.1. We picked the best architecture, activation function, DARTS cells, or transformer architectures based on the final validation error (for NAS-Bench-201, activation function, and transformer-based search experiments) or on the average of the five last validation errors (for DARTS). All search experiments used 8 asynchronous workers, each with a single NVIDIA RTX 2080 Ti GPU.
We chose the search strategies Random Search (RS), Regularized Evolution (RE) [12], AREA [65], NASBOT [18], and NASBOWL [48] as baselines. Note that we could not use gradient-based approaches for our experiments on the hierarchical NAS-Bench-201 search space since they do not scale to large hierarchical search spaces without any novel adoption (see Appendix G for a discussion). Also note that we could not apply AREA to the activation function search since it uses binary activation codes of ReLU as zero-cost proxy. Appendix I provides the implementation details of the search strategies.
In the following we answer all of the questions RQ1-RQ8. Figure 2 shows that BOHNAS finds superior architectures on the hierarchical NAS-Bench-201 search space compared to common baselines (answering RQ1); including NASBOWL, which does not use hierarchical information in its kernel design (partly answering RQ2), and zero-cost-based search strategy AREA (partly answering RQ3). We further investigated the hierarchical kernel design in Figure 12 in Appendix J.3 and found that it substantially improves regression performance of the surrogate model, especially on smaller amounts of training data (further answering RQ2). We also investigated other zero-cost proxies but they were mostly inferior to the simple baselines 12-norm [66] or flops [67] (Table 7 in Appendix J.3, further answering RQ3). We also found that BOHNAS is backwards compatible with cell-based search spaces. Table 1 shows that we found DARTS cells that are on-par on CIFAR-10 and (slightly) superior on ImageNet (with search on CIFAR-10) to state-of-the-art gradient-based methods; even though those are over-optimized due to successive works targeting this particular space.
Our searches on the transformer-based search spaces show that we can indeed find well-performing transformer architectures for, e.g., language modeling or sentiment analysis (RQ4). Specifically, we found a transformer for generative language modeling that achieved a best validation loss of 1.4386 with only  $3.33\mathrm{M}$  parameters. For comparison, Karpathy [68] reported a best validation loss of 1.4697 with a transformer with  $10.65\mathrm{M}$  parameters. Note that as we also searched for the embedding dimensionality, that acts globally on the architecture, we combined our hierarchical graph kernel (hWL) with a Hamming kernel. This demonstrates that our search space design framework and search strategy, BOHNAS, are applicable not only to NAS but also to joint NAS and Hyperparameter
Table 2: Results of the activation function search on CIFAR-10 with ResNet-20.
Search strategy	Test error [%]
ReLU	8.93
Swish	8.61
RS	8.91
RE	8.47
NASBOWL	8.32
BOHNAS	8.31
Optimization (HPO). The found classifier head for sentiment analysis achieved a test accuracy of  $92.26\%$ . We depict the found transformer and classifier head in Appendix M.3.
The experiments on activation function search (see Table 2) show that our search space design framework as well as BOHNAS can be effectively used to search for architectural patterns from even more primitive, mathematical operations (answering RQ1 & RQ5). Unsurprisingly, in this case, NASBOWL is on par with BOHNAS, since the computational graphs of the activation functions are small. This result motivates further steps in searching in expressive search spaces from even more atomic primitive computations.
To study the impact of the search space design and, thus, the architectural design choices on performance (RQ6-RQ8), we used various derivatives of our proposed hierarchical NAS-Bench-201 search space (see Section 4 and Appendix J.2 for the formal definitions of these search spaces). Figure 3 shows that we can indeed find better-performing architectures in hierarchical search spaces compared to simpler cell-based spaces, even though search is more complex due to the substantially larger search space (answering RQ6). Further, we find that the popular architectural uniformity design principle of reusing the same shared building block across the architecture (search spaces with keyword shared in Figure 3) improves performance (answering RQ7). This aligns well with the research in (manual and automated) architecture engineering over the last decade. However, we also found that adding more nonlinear macro topologies (search spaces with keyword non-linear in Figure 3), thereby allowing for more branching at the macro-level of architectures, surprisingly further improves performance (answering RQ8). This is in contrast to the common linear macro architectural design without higher degree of branching at the macro-level; except for few notable exceptions [69, 70].
While most previous works in NAS focused on cell-based search spaces, Table 1 reveals that our found architectures are superior in 6/8 cases (others are on par) to the architectures found by previous works. On the NAS-Bench-201 training protocol, our best found architectures on CIFAR-10, CIFAR-100, and ImageNet-16-120 (depicted in Figure 4) reduce the test error by  $0.51\%$ ,  $1.1\%$ , and  $4.99\%$  to the best reported numbers from the literature, respectively. Notably, this is even better than the optimal cells in the cell-based NAS-Bench-201 search space (further answering RQ6). This clearly emphasizes the potential of NAS going beyond cell-based search spaces (RQ6) and prompts us to rethink macro architectural design choices (RQ7, RQ8).
Figure 4: Best architecture found by BOHNAS for ImageNet-16-120 in the hierarchical NAS-Bench-201 search space. Best viewed with zoom. Figure 10 in Appendix J.3 provides the abbreviations of the operations and the architectures for the other datasets.
Our versatile search space design framework based on CFGs (Section 3) unifies all search spaces we are aware of from literature in a single framework; see Appendix F for several exemplar search spaces, and allows search over all (abstraction) levels of neural architectures. However, we cannot construct any architecture search space since we are limited to context-free languages (although our enhancements in Section 3.3 overcome some limitations), e.g., architecture search spaces of the type  $\{a^{n}b^{n}c^{n}|n\in \mathbb{N}_{>0}\}$  cannot be generated by CFGs (this can be proven using Ogden's lemma [71]).
While more expressive search spaces facilitate the search over a wider spectrum of architectures, there is an inherent trade-off between the expressiveness and the search complexity. The mere existence of potentially better-performing architectures does not imply that we can actually find them with a limited search budget (Appendix J.3); although our experiments suggest that this may be more feasible than previously expected (Section 6). In addition, these potentially better-performing architectures may not work well with current training protocols and hyperparameters due to interaction effects between them and over-optimization for specific types of architectures. A joint optimization of neural architectures, training protocols, and hyperparameters could overcome this limitation, but further fuels the trade-off between expressiveness and search complexity.
Finally, our search strategy, BOHNAS, is sample-based and therefore computationally intensive (search costs for up to 40 GPU days for our longest search runs). While it would benefit from weight sharing, current weight sharing approaches are not directly applicable due to the exponential increase of architectures and the consequent memory requirements. We discuss this issue further in Appendix G.
We introduced a unifying search space design framework for hierarchical search spaces based on CFGs that allows us to search over all (abstraction) levels of neural architectures, foster regularity, and incorporate user-defined constraints. To efficiently search over the resulting huge search spaces, we proposed BOHNAS, an efficient BO strategy with a kernel leveraging the available hierarchical information. Our experiments show the versatility of our search space design framework and how it can be used to study the performance impact of architectural design choices beyond the micro-level. We also show that BOHNAS can be superior to existing NAS approaches. Our empirical findings motivate further steps into investigating the impact of other architectural design choices on performance as well as search on search spaces with even more atomic primitive computations. Next steps could include the improvement of search efficiency by means of multi-fidelity optimization or meta-learning, or simultaneously search for architectures and the search spaces themselves.
This research was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under grant number 417962828, and the Bundesministerium für Umwelt, Naturschutz, nukleare Sicherheit und Verbraucherschutz (BMUV, German Federal Ministry for the Environment, Nature Conservation, Nuclear Safety and Consumer Protection) based on a resolution of the German Bundestag (67KI2029A). Robert Bosch GmbH is acknowledged for financial support. We gratefully acknowledge support by the European Research Council (ERC) Consolidator Grant "Deep Learning 2.0" (grant no. 101045765). Funded by the European Union. This research was partially supported by TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the ERC. Neither the European Union nor the ERC can be held responsible for them.
Funded by the European Union
A Vocabulary for primitive computations and topological operators
Table 3 and Figure 5 describe the primitive computations and topological operators used throughout our experiments, respectively. We defined the order of arguments of the topological operator as follows: we order arguments first by their sink nodes and then by the source node (following Dong and Yang [58]). Note that by adding more primitive computations and/or topological operators we could construct even more expressive search spaces.
Table 3: Primitive computations. "Name" corresponds to the string function symbols in our CFGs and "Function" is the associated implementation of the primitive computation in pseudocode. The subscripts  $g, k, s$  ,and  $p$  are abbreviations for groups,kernel size,strides,and padding,respectively. During assembly of neural architectures  $\Phi \left( \omega \right)$  ,we replace string function symbols with the associated primitive computation.
Name	Function
avg_pool	AvgPoolk=3,s=1,p=1(x)
batch	BN(x)
conv1x1	Convk=1,s=1,p=0(x)
conv3x3	Convk=3,s=1,p=1(x)
dconv3x3	Convg=C,k=3,s=1,p=1(x)
down	conv3x3s=1(conv3x3s=2(x))+Convk=1,s=1(AvgPoolk=2,s=2(x))
hardswish	Hardswish(x)
id	Identity(x)
instance	IN(x)
layer	LN(x)
mish	Mish(x)
relu	ReLU(x)
zero	Zeros(x)
(a) Sequential2(a,b).
(b) Sequential3(a,b,c).
(c) Sequential4(a, b, c, d).
(d) Residual2(a, b, c).
(e) Residual3(a, b, c, d).
(f) Cell(a,b,c,d,e,f).
(g) Diamond2(a, b, c, d).
(h) Diamond3(a, b, c, d, e, f).
Figure 5: Topological operators. Each subfigure makes the connection between the topogolocial operator and associated computational graph explicit, i.e., the arguments of the topological operators (a, b, ...) are mapped to the respective edges of the corresponding computational graph.
B Extended related work
Complementary to Section 2, we provide a comparison of our proposed search space design to others from the literature in Table 4. Our search space design framework allows us to search over all aspects of the architecture and incorporate user-defined constraints as well as foster regularity.
Table 4: Comparison of search space designs.
Design approach	Example	Search over/from	Constraints	Regularity
Micro structure	Macro structure	Primitives
Global	Kandasamy et al. [18]	✓	✓*	-	-	-
Hyperparameter-based	Tan and Le [1]	✓	✓†	-	-	-
Chain-structured	Roberts et al. [19]	✓	-	✓	-	-
Cell-based	Zoph et al. [23]	✓	-	-	-	✓°
n-level hierarchical assembly	Liu et al. [2]	✓	✓†	-	-	✓°
Graph generator hierarchy	Ru et al. [8]	✓	✓*	-	-	-
CoDeepNEAT	Miikkulainen et al. [11]	✓	✓*	-	-	✓
Formal systems	Negrinho et al. [42]	✓	✓†	-	✓	-
Enhanced CFGs	Ours	✓	✓	✓	✓	✓
$^{\dagger}$ search only linear or confined macro topologies; *requires post-hoc validation scheme; °do not share architectural patterns hierarchically
While our work focuses exclusively on NAS, we will discuss below how it relates to the areas of optimizer search (as well as automated machine learning from scratch), and neural-symbolic programming.
Optimizer search is a closely related field to NAS, where we automatically search for an optimizer (i.e., an update function for the weights) instead of an architecture. Initial works used learnable parametric or non-parametric optimizers. While the former approaches [72-75] have poor scalability and generality, the latter works overcome those limitations. More specifically, Bello et al. [76] searched for an instantiation of hand-crafted patterns via reinforcement learning, while Wang et al. [77] proposed a tree-structured search space<sup>1</sup> and searched for optimizers via a modified Monte Carlo sampling approach. AutoML-Zero [78] took an even more general approach by searching over entire machine learning algorithms, including optimizers, from a generic search space built from basic mathematical operations with an evolutionary algorithm. Chen et al. [79] used RE to search for optimizers from a generic search space (inspired by AutoML-Zero) for the training of vision transformers [80].
Complementary to the above, there is also recent interest in automatically synthesizing programs from domain-specific languages. Gaunt et al. [81] proposed a hand-crafted program template and simultaneously optimized the parameters of the differentiable program with gradient descent. Valkov et al. [82] proposed a type-directed (top-down) enumeration and evolution approaches over differentiable functional programs. Shah et al. [83] hierarchically assembled differentiable programs and used neural networks for the approximation of missing expression in partial programs. Cui and Zhu [84] treated CFGs stochastically with trainable production rule sampling weights, which were optimized with a gradient descent [6]. However, naively (directly) applying gradient-based approaches (without any novel adaption) does not scale to hierarchical search spaces due to the exponential explosion of supernet weights, but still renders an interesting direction for future work.
Compared to these lines of work, we enhance CFGs to handle changes in spatial resolution, explicitly promote regularity in the search space design, and (compared to most of them) incorporate constraints. Note that the latter two could also be applied in those domains. We also proposed a BO search strategy to search efficiently with a tailored kernel design to handle the hierarchical nature of the architectural search space.
C Details on extended Backus-Naur form
The (extended) Backus-Naur form [57] is a meta-language to describe the syntax of CFGs. We use meta-rules of the form S ::=  $\alpha$  where  $S \in N$  is a nonterminal and  $\alpha \in (N \cup \Sigma)^*$  is a string of nonterminals and/or terminals. We denote nonterminals in UPPER CASE, terminals corresponding to topological operators in Initial upper case/teletype, and terminals corresponding to primitive computations in lower case/teletype, e.g., S ::= Residual(S, id, S). To compactly express production rules with the same left-hand side nonterminal, we use the
<sup>1</sup>Note that the tree-structured search space can equivalently be described with a CFG (with a constraint on the number of maximum depth of the syntax trees).
(a) Computational graph representation of current derivation in Equation 8.
(b) Computational graph representation of current derivation in Equation 9.
(c) Computational graph representation of current derivation in Equation 10.
Figure 6: Comprehensive example for the construction of the macro topology in the hierarchical NAS-Bench-201 search space. We visualize the computational graphs of the derivation steps from Equations 8 to 10. Note that CLs (representing spatial resolution preserving cells) do not change the spatial resolution and the primitive computation down downsamples the features by a factor of two. Our proposed mechanism to overload the nonterminals (Section 3.3) effectively distributes the downsampling operations across the macro topology so that all incoming features at each node will have the same spatial resolution.
vertical bar | to indicate a choice of production rules with the same left-hand side, e.g., S ::= Sequential(S, S, S) | Residual(S, id, S) | conv.
D Comprehensive example for the construction of the macro topology
Consider the following derivation of the macro topology of an architecture in the hierarchical NAS-Bench-201 (see Section 4 for the search space definition):
$$
\begin{array}{l} \mathrm {D} 2 \rightarrow \mathrm {S} 3 (\mathrm {D} 1, \mathrm {D} 1, \mathrm {D} 0) (8) \\ \rightarrow \mathrm {S} 3 (\mathrm {R} 3 (\mathrm {C}, \mathrm {C}, \mathrm {D}, \mathrm {D}), \mathrm {R} 3 (\mathrm {C}, \mathrm {C}, \mathrm {D}, \mathrm {D}), \mathrm {S} 3 (\mathrm {C}, \mathrm {C}, \mathrm {C L})) (9) \\ \rightarrow \mathrm {S} 3 (\mathrm {R} 3 (\mathrm {S} 2 (\mathrm {C L}, \mathrm {C L}), \mathrm {R} 2 (\mathrm {C L}, \mathrm {C L}, \mathrm {C L}), \mathrm {S} 2 (\mathrm {C L}, \text {d o w n}), \mathrm {R} 2 (\mathrm {C L}, \text {d o w n}, \text {d o w n})),\\\mathrm {R} 3 (\mathrm {R} 2 (\mathrm {C L}, \mathrm {C L}, \mathrm {C L}), \mathrm {S} 2 (\mathrm {C L}, \mathrm {C L}), \mathrm {R} 2 (\mathrm {C L}, \text {d o w n}, \text {d o w n}), \mathrm {R} 2 (\mathrm {C L}, \text {d o w n}, \text {d o w n})) (10) \\ \mathbb {S} 3 (\mathbb {S} 2 (\mathrm {C L}, \mathrm {C L}), \mathrm {R} 2 (\mathrm {C L}, \mathrm {C L}, \mathrm {C L}), \mathrm {C L}) \quad , \\ \end{array}
$$
where we abbreviate Sequential and Residual with S or R for sake of brevity. Figure 6 visualizes the corresponding computational graphs at each step and shows how our proposed mechanism of overloading nonterminals (Section 3.3) effectively distributes downsampling operations (down) across the macro topology of an architecture so that incoming features at each node will have the same spatial resolution.
E Comprehensive example for the implementation of constraints