SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Simon Schrodi¹ Danny Stoll¹ Binxin Ru²  
Rhea Sanjay Sukthanker¹ Thomas Brox¹ Frank Hutter¹
$^{1}$ University of Freiburg  $^{2}$ University of Oxford  $\{\text{schrodi,stolld,sukthank,brox,fh}\} @$ cs.uni-freiburg.de robin@robots.ox.ac.uk
The discovery of neural architectures from simple building blocks is a longstanding goal of Neural Architecture Search (NAS). Hierarchical search spaces are a promising step towards this goal but lack a unifying search space design framework and typically only search over some limited aspect of architectures. In this work, we introduce a unifying search space design framework based on context-free grammars that can naturally and compactly generate expressive hierarchical search spaces that are 100s of orders of magnitude larger than common spaces from the literature. By enhancing and using their properties, we effectively enable search over the complete architecture and can foster regularity. Further, we propose an efficient hierarchical kernel design for a Bayesian Optimization search strategy to efficiently search over such huge spaces. We demonstrate the versatility of our search space design framework and show that our search strategy can be superior to existing NAS approaches. Code is available at https://github.com/automl/hierarchical_nas_construction.
Neural Architecture Search (NAS) aims to automatically discover neural architectures with state-of-the-art performance. While numerous NAS papers have already demonstrated finding state-of-the-art architectures (prominently, e.g., Tan and Le [1], Liu et al. [2]), relatively little attention has been paid to understanding the impact of architectural design decisions on performance, such as the repetition of the same building blocks. Moreover, despite the fact that NAS is a heavily-researched field with over 1000 papers in the last two years [3, 4], NAS has primarily been applied to over-engineered, restrictive search spaces (e.g., cell-based ones) that did not give rise to truly novel architectural patterns. In fact, Yang et al. [5] showed that in the prominent DARTS search space [6] the manually-defined macro architecture is more important than the searched cells, while Xie et al. [7] and Ru et al. [8] achieved competitive performance with randomly wired neural architectures that do not adhere to common search space limitations.
Hierarchical search spaces are a promising step towards overcoming these limitations, while keeping the search space of architectures more controllable compared to global, unrestricted search spaces. However, previous works limited themselves to search only for a hierarchical cell [9], linear macro topologies [1, 2, 10], or required post-hoc checking and adjustment of architectures [8, 11]. This limited their applicability in understanding the impact of architectural design choices on performance as well as search over all (abstraction) levels of neural architectures.
In this work, we propose a functional view of neural architectures and a unifying search space design framework for the efficient construction of (hierarchical) search spaces based on Context-Free Grammars (CFGs). We compose architectures from simple multivariate functions in a hierarchical manner
Figure 1: Derivation of the function composition of the neural architecture from Equation 1 (left). Note that the derivations correspond to edge replacements [14-16] in the computational graph representation (right). The intermediate derivations provide various granularities of a neural architecture. Appendix A provides the vocabulary of primitive computations and topological operators.
using the recursive nature of CFGs and consequently obtain a function composition representing the architecture. Further, we enhance CFGs with mechanisms to efficiently define search spaces over the complete architecture with non-linear macro topology (e.g., see Figure 4 or Figure 10 in Appendix J for examples), foster regularity by exploiting that context-free languages are closed under substitution, and demonstrate how to integrate user-defined search space constraints.
However, since the number of architectures scales exponentially in the number of hierarchical levels – leading to search spaces 100s of orders of magnitude larger than commonly used ones in NAS – for many prior approaches search becomes either infeasible (e.g., DARTS [6]) or challenging (e.g., regularized evolution [12]). As a remedy, we propose Bayesian Optimization for Hierarchical Neural Architecture Search (BOHNAS), which constructs a hierarchical kernel upon various granularities of the architectures, and show its efficiency through extensive experimental evaluation.
Our contributions We summarize our key contributions below:
1. We propose a unifying search space design framework for (hierarchical) NAS based on CFGs that enables us to search across the complete architecture, i.e., from micro to macro, foster regularity, i.e., repetition of architectural patterns, and incorporate user-defined constraints (Section 3). We demonstrate its versatility in Sections 4 and 6.
2. We propose a hierarchical extension of NAS-Bench-201 (as well as derivatives) to allow us to study search over various aspects of the architecture, e.g., the macro architecture (Section 4).
3. We propose BOHNAS to efficiently search over large hierarchical search spaces (Section 5).
4. We thoroughly show how our search space design framework can be used to study the impact of architectural design principles on performance across granularities.
5. We show the superiority of BOHNAS over common baselines on 6/8 (others on par) datasets above state-of-the-art methods, using the same training protocol, including, e.g., a  $4.99\%$  improvement on ImageNet-16-120 using the NAS-Bench-201 training protocol. Further, we show that we can effectively search on different types of search spaces (convolutional networks, transformers, or both) (Section 6).
6. We adhere to the NAS best practice checklist [13] and provide code at https://github.com/automl/hierarchical_nas_construction to foster reproducible NAS research (Appendix N).
We discuss related works in Neural Architecture Search (NAS) below and discuss works beyond NAS in Appendix B. Table 4 in Appendix B summarizes the differences between our proposed search space design based on CFGs and previous works.
Most previous works focused on global [17, 18], hyperparameter-based [1], chain-structured [19-22], or cell-based [23] search space designs. Hierarchical search spaces subsume the aforementioned spaces while being more expressive and effective in reducing search complexity. Prior works considered  $n$ -level hierarchical assembly [2, 9, 24], parameterization of a hierarchy of random
graph generators [8], or evolution of topologies and repetitive blocks [11]. In contrast to these prior works, we search over all (abstraction) levels of neural architectures and do not require any post-generation testing and/or adaptation of the architecture [8, 11]. Further, we can incorporate user-defined constraints and foster regularity in the search space design.
Other works used formal "systems": string rewriting systems [25, 26], cellular (or tree-structured) encoding schemes [27-30], hyperedge replacement graph grammars [31, 32], attribute grammars [33], CFGs [10, 34-40], And-Or-grammars [41], or a search space design language [42]. Different to these prior works, we search over the complete architecture with non-linear macro topologies, can incorporate user-defined constraints, and explicitly foster regularity.
For search, previous works, e.g., used reinforcement learning [17, 43], evolution [44], gradient descent [6, 45, 46], or Bayesian Optimization (BO) [18, 47, 48]. To enable the effective use of BO on graph-like inputs for NAS, previous works have proposed to use a GP with specialized kernels [18, 48, 49], encoding schemes [47, 50], or graph neural networks as surrogate model [51-53]. In contrast to these previous works, we explicitly leverage the hierarchical nature of architectures in the performance modeling of the surrogate model.
3 Unifying search space design framework
To efficiently construct expressive (hierarchical) NAS spaces, we review two common neural architecture representations and describe how they are connected (Section 3.1). In Sections 3.2 and 3.3 we propose to use CFGs and enhance them (or exploit their properties) to efficiently construct expressive (hierarchical) search spaces.
3.1 Neural architecture representations
Representation as computational graphs A neural architecture can be represented as an edge-attributed (or equivalently node-attributed) directed acyclic (computational) graph  $G(V,E,o_e,m_v)$ , where  $o_e$  is a primitive computation (e.g., convolution) or a computational graph itself (e.g., residual block) applied at edge  $e \in E$ , and  $m_v$  is a merging operation for incident edges at node  $v \in V$ . Below is an example of an architecture with two residual blocks followed by a linear layer:
where the edge attributes conv, id, and linear correspond to convolutional blocks, skip connections, or linear layer, respectively. This architecture can be decomposed into sub-components/graphs across multiple hierarchical levels (see Figure 1 on the previous page): the operations convolutional blocks and skip connection construct the residual blocks, which construct with the linear layer the entire architecture.
Representation as function compositions The example above can also naturally be represented as a composition of multivariate functions:
Sequential(Residual(conv, id, conv), Residual(conv, id, conv), linear) (1)
where the functions conv, id, linear, as well as Sequential and Residual are defined with an arity of zero or three, respectively. We refer to functions with arity of zero as primitive computations and functions with non-zero arity as topological operators that define the (functional/graph) structure. Note the close resemblance to the way architectures are typically implemented in code. To construct the associated computational graph representation (or executable neural architecture program), we define a bijection  $\Phi$  between function symbols and the corresponding computational graph (or function implemented in code, e.g., in Python), using a predefined vocabulary. Appendix D shows exemplary how a functional representation is mapped to its computational graph representation.
Finally, we introduce intermediate variables  $x_{i}$  that can share architectural patterns across an architecture, e.g., residual blocks  $x_{res} = \text{Residual(conv, id, conv)}$ . Thus, we can rewrite the architecture from Equation 1 as follows:
$$
\operatorname {S e q u e n t i a l} \left(x _ {r e s}, x _ {r e s}, \text {l i n e a r}\right). \tag {2}
$$
3.2 Construction based on context-free grammars
To construct neural architectures as compositions of (multivariate) functions, we propose to use Context-Free Grammars (CFGs) [54] since they naturally and in a formally grounded way generate (expressive) languages that are (hierarchically) composed from an alphabet, i.e., the set of (multivariate) functions. CFGs also provide a simple and formally grounded mechanism to evolve architectures that ensure that evolved architectures stay within the defined search space (see Section 5 for details). With our enhancements of CFGs (Section 3.3), we provide a unifying search space design framework that is able to represent all search spaces we are aware of from the literature. Appendix F provides examples for NAS-Bench-101 [50], DARTS [6], Auto-DeepLab [2], hierarchical cell space [9], Mobile-net space [55], and hierarchical random graph generator space [8].
Formally, a CFG  $\mathcal{G} = \langle N, \Sigma, P, S \rangle$  consists of a finite set of nonterminals  $N$  and terminals  $\Sigma$  (i.e., the alphabet of functions) with  $N \cap \Sigma = \emptyset$ , a finite set of production rules  $P = \{A \to \beta | A \in N, \beta \in (N \cup \Sigma)^*\}$ , where the asterisk denotes the Kleene star [56], and a start symbol  $S \in N$ . To generate a string (i.e., the function composition), starting from the start symbol  $S$ , we recursively replace nonterminals with the right-hand side of a production rule, until the resulting string does not contain any nonterminals. For example, consider the following CFG in extended Backus-Naur form [57] (refer to Appendix C for background on the extended Backus-Naur form):
$$
S := \text {S e q u e n t i a l} (S, S, S) \mid \text {R e s i d u a l} (S, S, S) \mid \text {c o n v} \mid \text {i d} \mid \text {l i n e a r}. \tag {3}
$$
Figure 1 shows how we can derive the function composition of the neural architecture from Equation 1 from this CFG and makes the connection to its computational graph representation explicit. The set of all (potentially infinite) function compositions generated by a CFG  $\mathcal{G}$  is the language  $L(\mathcal{G})$ , which naturally forms our search space. Thus, the NAS problem can be formulated as follows:
$$
\arg \min  \mathcal {L} (\Phi (\omega), \mathcal {D}) \quad , \tag {4}
$$
$$
\omega \in L (\mathcal {G})
$$
where  $\mathcal{L}$  is an error measure that we seek to minimize for some data  $\mathcal{D}$ , e.g., final validation error of a fixed training protocol.
3.3 Enhancements
Below, we enhance CFGs or utilize their properties to efficiently model changes in the spatial resolution, foster regularity, and incorporate constraints.
Flexible spatial resolution flow Neural architectures commonly build a hierarchy of features that are gradually downsampled, e.g., by pooling operations. However, many NAS works do not search over the macro topology of architectures (e.g., Zoph et al. [23]), only consider linear macro topologies (e.g., Liu et al. [2]), or require post-generation testing for resolution mismatches with an adjustment scheme (e.g., Ru et al. [8], Miikkulainen et al. [11]).
To overcome these limitations, we propose a simple mechanism to search over the macro topology with flexible spatial resolution flow by overloading nonterminals: We assign to each nonterminal the number of downsampling operations required in its subsequent derivations. This effectively distributes the downsampling operations recursively across the architecture.
For example, the nonterminals D1, D2 of the production rule  $\mathrm{D}2 \rightarrow \mathrm{Residual}(\mathrm{D}1, \mathrm{D}2, \mathrm{D}1)$  indicate that 1 or 2 downsampling operations must be applied in their subsequent derivations, respectively. Thus, the input features of the residual topological operator Residual will be downsampled twice in both of its paths and, consequently, the merging paths will have the same spatial resolution. Appendix D provides an example that also makes the connection to the computational graph explicit.
Fostering regularity through substitution To foster regularity, i.e., reuse of architectural patterns, we implement intermediate variables (Section 3.1) by exploiting the property that context-free languages are closed under substitution. More specifically, we can substitute an intermediate variable  $x_{i} \in \Sigma$  with a string  $\omega$  of another language  $L^{\prime}$ , e.g., constructing cell topologies. By substituting the same intermediate variable multiple times, we reuse the same architectural pattern  $(\omega)$  and, thereby, effectively foster regularity. Note that the language  $L^{\prime}$  may in turn have its own intermediate variables that map to languages constructing other architectural patterns, e.g., activation functions.
For example, consider the languages  $L(\mathcal{G}_m)$  and  $L(\mathcal{G}_c)$  constructing the macro or cell topology of a neural architecture, respectively. Further, we add a single intermediate variable  $x_1$  to the terminals  $\Sigma_{\mathcal{G}_m}$  that map to the string  $\omega_1 \in L(\mathcal{G}_c)$ , e.g., the searchable cell. Thus, after substituting all  $x_1$  with  $\omega_1$ , we effectively share the same cell topology across the macro topology of the architecture.
Constraints When designing a search space, we often want to adhere to constraints. For example, we may only want to have two incident edges per node - as in the DARTS search space [6] - or ensure that for every neural architecture the input is associated with its output. Note that such constraints implicate context-sensitivity but CFGs by design are context-free. Thus, to still allow for constraints, we extend the sampling and evolution procedures of CFGs by using a (one-step) lookahead to ensure that the next step(s) in sampling procedure (or evolution) does not violate the constraint. We provide more details and a comprehensive example in Appendix E.
4 Example: Hierarchical NAS-Bench-201 and its derivatives
In this section, we propose a hierarchical extension to NAS-Bench-201 [58]: hierarchical NAS-Bench-201 that subsumes (cell-based) NAS-Bench-201 [58] and additionally includes a search over the macro topology as well as the parameterization of the convolutional blocks, i.e., type of convolution, activation, and normalization. Below, is the definition of our proposed hierarchical NAS-Bench-201:
D2 := Sequential3(D1, D1, D0) | Sequential3(D0, D1, D1) | Sequential4(D1, D1, D0, D0)  
D1 := Sequential3(C, C, D) | Sequential4(C, C, C, D) | Residual3(C, C, D, D)  
D0 := Sequential3(C, C, CL) | Sequential4(C, C, C, CL) | Residual3(C, C, CL, CL)  
D := Sequential2(CL, down) | Sequential3(CL, CL, down) | Residual2(CL, down, down)  
C := Sequential2(CL, CL) | Sequential3(CL, CL, CL) | Residual2(CL, CL, CL)  
CL := Cell(OP, OP, OP, OP, OP, OP)  
OP := zero | id | CONVBLOCK | avg_pool  
CONVBLOCK := Sequential3(ACT, CONV, NORM)  
ACT := relu | hardswish | mish  
CONV := conv1x1 | conv3x3 | dconv3x3  
NORM := batch | instance | layer
The blue productions of the nonterminals  $\{\mathrm{D}2,\mathrm{D}1,\mathrm{D}0,\mathrm{D},\mathrm{C}\}$  construct the (non-linear) macro topology with flexible spatial resolution flow, possibly containing multiple branches. The red and yellow productions of the nonterminals  $\{\mathrm{CL},\mathrm{OP}\}$  construct the NAS-Bench-201 cell and  $\{\mathrm{CONVBLOCK},\mathrm{ACT},\mathrm{CONV},\mathrm{NORM}\}$  parameterize the convolutional block. Note that the red productions correspond to the original NAS-Bench-201 cell-based (sub)space [58]. Appendix A provides the vocabulary of topological operators and primitive computations and Appendix D provides a comprehensive example on the construction of the macro topology.
We omit the stem (i.e.,  $3 \times 3$  convolution followed by batch normalization) and classifier head (i.e., batch normalization followed by ReLU, global average pooling, and linear layer) for simplicity. We used element-wise summation as merge operation. For the number of channels, we adopted the common design to double the number of channels whenever we halve the spatial resolution. Alternatively, we could handle a varying number of channels by using, e.g., depthwise concatenation as merge operation; thereby also subsuming NATS-Bench [59]. Finally, we added a constraint to ensure that the input is associated with the output since zero could disassociate the input from the output.
Search space capacity The search space consists of ca.  $10^{446}$  architectures (Appendix G describes how to compute the search space size), which is hundreds of orders of magnitude larger than other popular (finite) search spaces from the literature, e.g., the NAS-Bench-201 or DARTS search spaces only entail ca.  $1.5 \cdot 10^{4}$  and  $10^{18}$  architectures, respectively.
Derivatives We can derive several variants of our hierarchical NAS-Bench-201 search space (hierarchical). This allows us to investigate search space as well as architectural design principles in Section 6. We briefly sketch them below and refer for their formal definitions to Appendix J.2:
- fixed+shared (cell-based): Fixed macro topology (only leftmost blue productions) with the single, shared NB-201 cell (red productions); equivalent to NAS-Bench-201 [58].
- hierarchical+shared: Hierarchical macro search (blue productions) with a single, shared cell (red & yellow productions).
- hierarchical+non-linear: Hierarchical macro search with more non-linear macro topologies (i.e., some Sequential are replaced by Diamond topological operators), allowing for more branching at the macro-level of architectures.
- hierarchical+shared+non-linear: Hierarchical macro search with more non-linear macro topologies (more branching at the macro-level) with a single, shared cell.
5 Bayesian Optimization for hierarchical NAS
Expressive (hierarchical) search spaces present challenges for NAS search strategies due to their huge size. In particular, gradient-based methods (without any yet unknown novel adoption) do not scale to expressive hierarchical search spaces since the supernet would yield an exponential number of weights (Appendix G provides an extensive discussion). Reinforcement learning approaches would also necessitate a different controller network. Further, we found that evolutionary and zero-cost methods did not perform particularly well on these search spaces (Section 6). Most Bayesian Optimization (BO) methods also did not work well [18, 52] or were not applicable (adjacency [50] or path encoding [47]), except for NASBOWL [48] with its Weisfeiler-Lehman (WL) graph kernel design in the surrogate model.
Thus, we propose the novel BO strategy Bayesian Optimization for Hierarchical Neural Architecture Search (BOHNAS), which (i) uses a novel hierarchical kernel that constructs a kernel upon different granularities of the architectures to improve performance modeling, and (ii) adopts ideas from grammar-guided genetic programming [60, 61] for acquisition function optimization of the discrete space of architectures. Below, we describe these components and provide more details in Appendix H.
Hierarchical Weisfeiler-Lehman kernel (hWL) We adopt the WL graph kernel design [48, 62] for performance modeling. However, modeling solely based on the final computational graph of the architecture, similar to Ru et al. [48], ignores the useful hierarchical information inherent in our construction (Section 3). Moreover, the large size of the architectures also makes it difficult to use a single WL kernel to capture the more global topological patterns.
As a remedy, we propose the hierarchical WL kernel (hWL) that hierarchically constructs a kernel upon various granularities of the architectures. It efficiently captures the information in all hierarchical levels, which substantially improves search and surrogate regression performance (Section 6). To compute the kernel, we introduce fold operators  $F_{l}$  that remove all substrings (i.e., inner functions) beyond the  $l$ -th hierarchical level, yielding partial function compositions (i.e., granularities of the architecture). E.g., the folds  $F_{1}, F_{2}$  and  $F_{3}$  for the function composition  $\omega$  from Equation 1 are:
$$
F _ {3} (\omega) = \text {S e q u e n t i a l (R e s i d u a l (c o n v , i d , c o n v) , R e s i d u a l (c o n v , i d , c o n v) , l i n e a r)}
$$
$$
F _ {2} (\omega) = \text {S e q u e n t i a l} (\text {R e s i d u a l}, \text {R e s i d u a l}, \text {l i n e a r}) \quad , \tag {6}
$$
$$
F _ {1} (\omega) = \text {S e q u e n t i a l}.
$$
Note that  $F_{3}(\omega) = \omega$  and observe the similarity to the derivations in Figure 1. We define hWL for two function compositions  $\omega_{i}, \omega_{j}$  constructed over  $L$  hierarchical levels, as follows:
$$
k _ {h W L} \left(\omega_ {i}, \omega_ {j}\right) = \sum_ {l = 2} ^ {L} \lambda_ {l} \cdot k _ {W L} \left(\Phi \left(F _ {l} \left(\omega_ {i}\right)\right), \Phi \left(F _ {l} \left(\omega_ {j}\right)\right)\right), \tag {7}
$$
where  $\Phi$  bijectively maps the function compositions to their computational graphs. The weights  $\lambda_{l}$  govern the importance of the learned graph information at different hierarchical levels  $l$  (granularities of the architecture) and can be optimized (along with other GP hyperparameters) by maximizing the marginal likelihood. We omit the fold  $F_{1}(\omega)$  since it does not contain any edge features. Appendix H.2 provides more details on our proposed hWL.