SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the design of a search space, one often wants to adhere to one or more constraints. For example, one wants to have exactly two incident edges per node [6] or wants to ensure that every architecture in the search space is valid, i.e., there needs to exist at least one path from the input to the output. CFGs (by design) cannot handle such constraints since it requires context-sensitivity. Thus, we extend CFGs with a  $n$ -step lookahead to guarantee compliance with the constraints. The  $n$ -step lookahead considers all  $n$ -step scenarios (e.g., all possible  $n$ -step derivations) and filters out all scenarios that would result in a violation with at least one constraint. In practice, we can leverage the recursive
nature of CFGs and find that one-step lookaheads are sufficient for the constraints considered in our work. In the following, we consider how the lookahead effectively ensures compliance for an exemplary constraint.
We consider the constraint "the input is associated with the output". To guarantee this constraint, it is sufficient to ensure that for each topological operator during sampling (or evolution) there is at least one path from the input to the output (due to the recursive nature of CFGs). To this end, we test whether the removal of an edge would result in a disassociation of the input from the output of the computational graph of the current topological operator (no path from input to output) with an one-step lookahead. More specifically, consider the following topological operator Ce11(a, b, c, d,e, f):
and the primitive computations zero and non-zero, where the former disassociates two nodes, i.e., removes their edge. The topological operator  $\mathsf{Cell}$  has the following four paths from the input (1) to output (4): 1-2-3-4, 1-2-4, 1-3-4, and 1-4. Note that to not violate the constraint at least one of these paths needs to exist after sampling (or evolution). Below, we describe a potential sampling of primitive computations for the topological operator:
1. Substitute a with non-zero: valid since non-zero does not disassociate the input from the output.
2. Substitute b with zero: valid since the paths 1-2-3-4, 1-2-4, and 1-4 are still possible. Remove 1-3-4 from the paths.
3. Substitute c with zero: valid since the paths 1-2-4 and 1-4 are still possible. Remove 1-2-3-4 from the paths.
4. Substitute  $d$  with zero: valid since the path 1-2-4 is still possible. Remove 1-4 from the paths.
5. Substitute e with
- zero: invalid since it would remove the last remaining possible path 1-2-4.
- non-zero: valid since non-zero does not disassociate the input from the output.
6. Substitute  $f$  with zero: valid since the path 1-2-4 is still possible. Note that we also could substitute  $f$  with non-zero but this operation would be pruned away since the node 3 is not connected to the input.
Consequently, the sampling generates the following computational graph that adheres to the constraint:
Note that this procedure only samples architectures which comply with the constraint. Further, we note that all architectures will be valid, i.e., there are no spatial resolution or channel mismatches due to our enhancements of CFGs (Section 3.3). We can similarly implement this procedure for evolution as well as for other constraints.
F Common search spaces from the literature
In Section 4, we demonstrated how to construct the hierarchical NAS-Bench-201 search space, that subsumes the cell-based NAS-Bench-201 search space [58], with our search space design framework. Below, we first provide a best practice guide to design search spaces and then show how to also define the following popular search spaces from the literature: NAS-Bench-101 search space [50], DARTS
Figure 7: Visualization of the Darts topological operator.
search space [6], Auto-DeepLab search space [2], hierarchical cell search space [9], Mobile-net search space [55], and hierarchical random graph generator search space [8]. For more details on the search spaces, please refer to the respective works.
Best practice to construct search space with context-free grammars
We first note that the design largely depends on the target task. That is, an image-based task may require a different design than a text-based task. In general, when looking for high-performing architectures for a given task, we recommend reviewing the relevant literature and consulting domain experts if one is not particularly familiar with the application domain. In particular, we recommend distilling commonalities and differences among architectures that form the basis of the design for the CFG. Starting from this basis, we recommend expanding the search space by adding more and more production rules to the CFG to allow for the exploration of interesting novel architectures. This procedure ensures search can efficiently find well-performing architectures by exploiting prior knowledge in the search space design. We leave the integration of prior knowledge into BOHNAS for future work. If computational resources are virtually unlimited, one could also search for for architectures from scratch, similar to the AutoML-Zero [78].
NAS-Bench-101 search space
The NAS-Bench-101 search space [50] consists of a fixed macro architecture and a searchable cell, i.e., directed acyclic graph with maximally 7 nodes and 9 edges. We adopted the API of NAS-Bench-101 in our search space design and, thus, we can define the NAS-Bench-101 search space as follows
$$
\mathrm {N B 1 0 1} := \operatorname {M o d e l S p e c} (\operatorname {M A T R I X}, \operatorname {O P S})
$$
$$
\begin{array}{l} \text {M A T R I X} := \text {M a t r i x} (\underbrace {\text {B I N A R Y} , \dots , \text {B I N A R Y}} _ {2 1 x}) \\ \text {B I N A R Y} := 0 \mid 1 \end{array} \tag {11}
$$
$$
\text {B I N A R Y} \quad : := 0 \mid 1 \tag {11}
$$
$$
O P S := \operatorname {L i s t} (\mathrm {O P}, \mathrm {O P}, \mathrm {O P}, \mathrm {O P}, \mathrm {O P})
$$
$$
O P \quad := \operatorname {c o n v} _ {1} x 1 \mid \operatorname {c o n v} _ {3} x 3 \mid \max  _ {\text {p o o l}}
$$
and add a constraint to ensure that  $|\mathrm{BINARY} \to 1| \leq 9$ .
DARTS search space
The DARTS search space [6] consists of a fixed macro architecture and a cell, i.e., a seven node directed acyclic graph (Darts; see Figure 7 for the topological operator). We omit the fixed macro architecture from our search space design for simplicity. Each cell receives the feature maps from the two preceding cells as input and outputs a single feature map. Each intermediate node (i.e., Node3, Node4, Node5, and Node6) is computed based on two of its predecessors. Thus, we can define the
DARTS search space as follows:
DARTS  $\coloneqq$  DartsNODE3, NODE4, NODE5, NODE6)   
NODE3  $\coloneqq$  Node3(OP, OP)   
NODE4  $\coloneqq$  Node4(OP, OP, OP)   
NODE5  $\coloneqq$  Node5(OP, OP, OP, OP) (12)   
NODE6  $\coloneqq$  Node6(OP, OP, OP, OP, OP)   
OP  $\coloneqq$  sep_conv_3x3 | sep_conv_5x5 | dil_conv_3x3 | dil_conv_5x5 | max_pool | avg_pool | id | zero ,
where the topological operator Node3 receives two inputs, applies the operations separately on them, and sums them up. Similarly, Node4, Node5, and Node6 apply their operations separately to the given inputs and sum them up. The topological operator Darts feeds the corresponding feature maps into each of those topological operators and finally concatenates all intermediate feature maps. Since in the final discrete architecture each intermediate node only receives two non-zero inputs, we need to add a constraint to ensure that each intermediate node only has two incident edges. More specifically, we enforce that for each intermediate node only two OP primitive computations are non-zero and the others are zero. For example, one of the primitive computations of the topological operator Node4 is enforced to be zero.
To avoid adding an explicit constraint, we can utilize that the topological operators can be an arbitrary (Python) function (with an associated computational graph which is only required for search strategies using the graph information in their performance modeling). Thus, we can equivalently express the DARTS search space as follows:
DARTS  $\coloneqq$  GENOTYPE(OP, IN1, OP, IN1, OP, IN2, OP, IN2, OP, IN3, OP, IN3, OP, IN4, OP, IN4)  
OP  $\coloneqq$  sep_conv_3x3 | sep_conv_5x5 | dil_conv_3x3 | dil_conv_5x5 | max_pool | avg_pool | id  
IN1  $\coloneqq$  0 | 1  
IN2  $\coloneqq$  0 | 1 | 2  
IN3  $\coloneqq$  0 | 1 | 2 | 3  
IN4  $\coloneqq$  0 | 1 | 2 | 3 | 4 , (13)
where GENOTYPE resembles the Python object (namedtuple), which can be plugged (almost) directly into standard DARTS implementations.
Auto-DeepLab search space
Auto-DeepLab [2] combines a cell-level with a network-level search space to search for segmentation networks, where the cell is shared across the searched macro architecture, i.e., a twelve step (linear) path across different spatial resolutive levels. The cell-level design is similar to the one of Liu et al. [6] and, thus, we can use a similar search space design as in Equation 12. For the network-level, we introduce a constraint that ensures that the path is of length twelve, i.e., we ensure exactly twelve derivations in our CFG at the network-level. Further, we overload the nonterminals so that they correspond to the respective spatial resolutional level, e.g., D4 indicates that the original input is downsampled by a factor of four (see Section 3.3 for details). For the sake of simplicity, we omit the first two layers and atrous spatial pyramid poolings as they are fixed, and hence define the network-level search space as follows:
$$
\begin{array}{l} \text {D 4} := \text {S a m e (C E L L , D 4)} \mid \text {D o w n (C E L L , D 8)} \\ \text {D 8} := \text {U p (C E L L , D 4)} \mid \text {S a m e (C E L L , D 8)} \mid \text {D o w n (C E L L , D 1 6)} \\ \text {D 1 0} := \text {s i g m a (V T O S , D 6)} \mid \text {s i g m a (V T O S , D 1 0)} \mid \text {s i g m a (V T O S , D 2 0)} \end{array} \tag {14}
$$
$$
D 1 6 := \operatorname {U p} (\text {C E L L}, \mathrm {D} 8) \mid \text {S a m e} (\text {C E L L}, \mathrm {D} 1 6) \mid \text {D o w n} (\text {C E L L}, \mathrm {D} 3 2)
$$
$$
D 3 2 := \operatorname {U p} (\text {C E L L}, \text {D 1 6}) \mid \text {S a m e} (\text {C E L L}, \text {D 3 2}),
$$
where the topological operators Up, Same, and Down upsample/halve, do not change/do not change, or downsample/double the spatial resolution/channels, respectively. The intermediate variable CELL maps to the shared cell.
Hierarchical cell search space
The hierarchical cell search space [9] consists of a fixed (linear) macro architecture and a hierarchically assembled cell with three levels which is shared across the macro architecture. Thus, we can omit
the fixed macro architecture from our search space design for simplicity. The first, second, and third hierarchical levels correspond to the primitive computations (i.e., id, max_pool, avg_pool, sep_conv, depth_conv, conv, zero), six densely connected four node directed acyclic graphs (DAG4), and a densely connected five node directed acyclic graph (DAG5), respectively. The zero operation could lead to directed acyclic graphs which have fewer nodes. Therefore, we introduce a constraint enforcing that there are always four (level 2) or five (level 3) nodes for every directed acyclic graph. More specifically, we only need to ensure each node has at least on incident and outgoing edge. Alternatively, we could write out all all variants explicitly. Further, since a densely connected five node directed acyclic graph graph can have (at most) ten edges, we need to introduce the intermediate variables (i.e., M1, ..., M6) to enforce that only six (possibly) different four node directed acyclic graphs are used. Consequently, we define a CFG for the third hierarchical level
$$
\text {L E V E L 3} := \text {D A G 5} (\underbrace {\text {L E V E L 2} , \dots , \text {L E V E L 2}} _ {\times 1 0}) \tag {15}
$$
$$
\text {L E V E L 2} \quad : := \mathrm {M} 1 \mid \mathrm {M} 2 \mid \mathrm {M} 3 \mid \mathrm {M} 4 \mid \mathrm {M} 5 \mid \mathrm {M} 6 \mid \text {z e r o},
$$
where we substitute the intermediate variables M1, ..., M6 with the six lower-level motifs constructed by the first and second hierarchical level
$$
\text {L E V E L 2} := \text {D A G 4} (\underbrace {\text {L E V E L 1} , \dots , \text {L E V E L 1}} _ {\times 6}) \tag {16}
$$
$$
\text {L E V E L 1} := \text {i d} | \text {m a x - p o o l} | \text {a v g - p o o l} | \text {s e p - c o n v} | \text {d e p t h - c o n v} | \text {c o n v} | \text {z e r o}.
$$
Mobile-net search space
Factorized hierarchical search spaces, e.g., the Mobile-net search space [55], factorize a (fixed) macro architecture – often based on an already well-performing reference architecture – into separate blocks (e.g., cells). For the sake of simplicity, we assume a three sequential blocks (Block) reference architecture (Sequential). In each of those blocks, we search for the convolution operations (CONV), kernel sizes (KSIZE), squeeze-and-excitation ratio (SERATIO) [85], skip connections (SKIP), number of output channels (FSIZE), and number of layers per block (#LAYERS), where the latter two are discretized using the reference architecture. Consequently, we can express this search space as follows:
MACRO := Sequential(CONVBLOCK, CONVBLOCK, CONVBLOCK)  
CONVBLOCK := Block(CONV, KSIZE, SERATIO, SKIP, FSIZE, #LAYERS)  
CONV := conv | dconv | mbconv  
KSIZE := 3 | 5  
SERATIO := 0 | 0.25  
SKIP := pooling | id_residual | no Skip  
FSIZE := 0.75 | 1.0 | 1.25  
#LAYERS := -1 | 0 | 1,
where conv, donv and mbconv correspond to convolution, depthwise convolution, and mobile inverted bottleneck convolution [86], respectively.
Hierarchical random graph generator search space
The hierarchical random graph generator search space [8] consists of three hierarchical levels of random graph generators (i.e., Watts-Strogatz [87] and Erdős-Rényi [88]). We denote with Watts-Strogatz_i the random graph generated by the Watts-Strogatz model with i nodes. Thus,
we can represent the search space as follows:
$$
\begin{array}{l} \text {T O P} := \text {W a t t s - S t r o g a t z _ {3} (K , P t) (M I D , M I D , M I D)} \mid \dots \\ | \text {W a t t s - S t r o g a t z} _ {1 0} (\mathrm {K}, \mathrm {P t}) (\underbrace {\mathrm {M I D} , \dots , \mathrm {M I D}} _ {\times 1 0}) \\ \end{array}
$$
$$
\begin{array}{l} \text {M I D} := \operatorname {E r d o s - R e n y i \_ 1 (P m) (B O T)} \mid \dots \\ \mid \text {E r d o s - R e n y i} _ {- 1 0} (\mathrm {P m}) (\underbrace {\mathrm {B O T} , \dots , \mathrm {B O T}} _ {\times 1 0}) \tag {18} \\ \end{array}
$$
$$
\begin{array}{l} B O T::= W a t t s - S t r o g a t z _ {3} (K, P b) \left(N O D E, N O D E, N O D E\right) | \dots \\ | \text {W a t t s - S t r o g a t z} _ {- 1 0} (\mathrm {K}, \mathrm {P b}) (\underbrace {\text {N O D E} \dots , \text {N O D E}} _ {\times 1 0}) \\ K := 2 \mid 3 \mid 4 \mid 5, \\ \end{array}
$$
where each terminal  $\mathrm{Pt}$ ,  $\mathrm{Pm}$ , and  $\mathrm{Pb}$  map to a continuous number in  $[0.1, 0.9]^2$  and the placeholder variable NODE maps to a primitive computation, e.g., separable convolution. Note that we omit other hyperparameters, such as stage ratio, channel ratio etc., for simplicity.
G Computation of search space and supernet size
In this section, we show how to efficiently compute the size of search spaces and the size of a (hypothetical) supernet.
Search space size There are two cases to consider: (i) a CFG contains cycles (i.e., part of the derivation can be repeated infinitely many times), yielding an open-ended, infinite search space; and (ii) a CFG contains no cycles, yielding in a finite search space whose size we can compute.
Consider a production  $\mathrm{A} \rightarrow \mathrm{Residual}(\mathrm{B}, \mathrm{B}, \mathrm{B})$  where Residual is a terminal, and A and B are nonterminals with  $\mathrm{B} \rightarrow \mathrm{conv} \mid \mathrm{id}$ . Consequently, there are  $2^{3} = 8$  possible instances of the residual block. If we add another production rule for the nonterminal A, e.g.,  $\mathrm{A} \rightarrow \mathrm{Sequential}(\mathrm{B}, \mathrm{B}, \mathrm{B})$ , we would have  $2^{3} + 2^{3} = 16$  possible instances. Further, adding a production rule  $\mathrm{C} \rightarrow \mathrm{Sequential}(\mathrm{A}, \mathrm{A}, \mathrm{A})$  would yield a search space size of  $(2^{3} + 2^{3})^{3} = 4096$ .
More generally, we introduce the function  $P_{\mathrm{A}}$  that returns the set of productions for nonterminal  $\mathrm{A} \in N$ , and the function  $\mu : P \to N$  that returns all the nonterminals for a production  $p \in P$ . We can then recursively, starting from the start symbol  $\mathrm{A} \in N$ , compute the size of the search space as follows:
$$
f (\mathrm {A}) := \sum_ {p \in P _ {\mathrm {A}}} \left\{ \begin{array}{l l} 1 & , \mu (p) = \emptyset , \\ \prod_ {\mathrm {A} ^ {\prime} \in \mu (p)} f \left(\mathrm {A} ^ {\prime}\right) & , \text {o t h e r w i s e} \end{array} . \right. \tag {19}
$$
When a CFG contains some constraint, we ensure to only account for valid architectures (i.e., compliant with the constraints) by ignoring productions which would lead to invalid architectures.
Supernet size Stochastic supernets are a key component of recently popular fast search strategies, e.g., gradient-based methods. A supernet is composed of all possible architectures of a search space. We outline how one could naively adopt these gradient-based methods to hierarchical search spaces and discuss why these supernets (without any novel adaption) do not scale to large hierarchical search spaces.
DARTS [6] and its variants (e.g., Chen et al. [45], Dong and Yang [58], Xiao et al. [89]) relax the categorical (discrete) choice of primitive computations  $\mathcal{O}$ , e.g.,  $\bar{o}^{(i,j)}(x) = \sum_{o\in \mathcal{O}}m(x)o(x)$ , where  $m(\cdot)$  is a mixing function, e.g., DARTS uses the softmax over all possible primitive computations  $m(x) = \frac{\exp(\alpha_o^{(i,j)})}{\sum_{o'\in\mathcal{O}}\exp(\alpha_{o'}^{(i,j)})}$ , and  $o(\cdot)$  is a primitive computation, e.g., convolution.
2While CFGs do not support continuous production choices, we extend the notion of substitution by substituting a string representation of a Python (float) variable for the intermediate variables Pt,  $\mathrm{Pm}$ , and Pb.
For hierarchical search spaces, we could extend DARTS by also relaxing the categorical choices of the topological operators  $t_l \in \mathcal{T}_l$  (e.g., Liu et al. [2]), i.e.,  $\overline{t}_l^{(i,j)}(x) = \sum_{t_l \in \mathcal{T}_l} m_l(x) t_l(x)$  with, e.g.,  $m_l(x) = \frac{\exp(\alpha_{t_l}^{(i,j)})}{\sum_{t_l' \in \mathcal{T}_l} \exp(\alpha_{t_l'}^{(i,j)})}$ , where  $l$  indicates the hierarchical level,  $i, j$  denote the edge  $e = (i, j)$  of the topological operator  $t_l$ , and  $\mathcal{T}_1 = \mathcal{O}$ . Note that this corresponds to a stochastic treatment of CFGs with learnable sampling probabilities  $\alpha_{t_l}^{(i,j)}$  for their productions. Further, note that we (implicitly) assume independence of the current derivation step from the previous ones, even though it actually is conditionally dependent on the previous derivation steps.
To search with gradient descent, we need to create a supernet that contains all (exponentially many) discrete architectures. To this end, we define densely connected  $n$ -node directed acyclic graphs  $G_{l}$  that entails all possible topological operators at each hierarchical level  $l \in [1,\dots,L]$ , in which each topological operator  $t_{l} \in \mathcal{T}_{l}$  can be identified by the indication vector  $z_{t_l} \in \{0,1\}^{n(n - 1) / 2}$ , where  $n(n - 1) / 2 = |E_{G_l}|$  is the number of edges. Consequently, we can recursively compute the number of possible architectures represented by the supernet as follows
$$
g (l) := \left\{ \begin{array}{l l} C _ {G _ {1}} & , l = 1, \\ \prod_ {e \in \{1, \dots , | E _ {G _ {l}} | \}} g (l - 1) & , o t h e r w i s e \end{array} , \right. \tag {20}
$$
where  $G_{1}$  corresponds directed acyclic graph for the cell-level search space of size  $C_{G_1}$ , e.g.,  $10^{9}$  for the DARTS cell. We can similarly compute the number of searchable cells as follows
$$
g ^ {\prime} (l) := \left\{ \begin{array}{l l} \left| E _ {G _ {2}} \right| & , l = 2, \\ \prod_ {e \in \{1, \dots , \left| E _ {G _ {l}} \right| \}} g ^ {\prime} (l - 1) & , \text {o t h e r w i s e} \end{array} . \right. \tag {21}
$$
Consequently, the supernet is of size of  $g'(L) \cdot S_{G_1}$ , where  $S_{G_1}$  is the size of a searchable cell. It is apparent that the supernet size scales linearly in the number of cells  $g'(L)$ , which, however, scales exponentially in the number of hierarchical levels. This makes the application of methods using supernets, e.g., gradient-based ones, (generally) challenging in hierarchical search spaces with many hierarchical levels.
H More details on the proposed search strategy
In this section, we provide more details and comprehensive examples for our search strategy Bayesian Optimization for Hierarchical Neural Architecture Search (BOHNAS) presented in Section 5.
H.1 Bayesian Optimization
Bayesian Optimization (BO) is a powerful family of search techniques for finding the global optimum of a black-box objective problem. It is particularly suited when the objective is expensive to evaluate and, thus, sample efficiency is highly important [90].
To minimize a black-box objective problem with BO, we first need to build a probabilistic surrogate to model the objective based on the observed data so far. Based on the surrogate model, we design an acquisition function to evaluate the utility of potential candidate points by trading off exploitation (where the posterior mean of the surrogate model is low) and exploration (where the posterior variance of the surrogate model is high). The next candidate points to evaluate is then selected by maximizing the acquisition function [91]. The general procedure of BO is summarized in Algorithm 1.
We used the widely used acquisition function, expected improvement (EI) [92], in our BO strategy. EI evaluates the expected amount of improvement of a candidate point  $\mathbf{x}$  over the minimal value  $f^{\prime}$  observed so far. Specifically, denote the improvement function as  $I(\mathbf{x}) = \max (0,f' - f(\mathbf{x}))$ , the EI acquisition function has the form
$$
\begin{array}{l} \alpha_ {E I} (\mathbf {x} | \mathcal {D} _ {t}) = \mathbb {E} [ I (\mathbf {x}) | \mathcal {D} _ {t} ] = \int_ {- \infty} ^ {f ^ {\prime}} (f ^ {\prime} - f) \mathcal {N} \left(f; \mu (\mathbf {x} | \mathcal {D} _ {t}), \sigma^ {2} (\mathbf {x} | \mathcal {D} _ {t})\right) d f \\ = (f ^ {\prime} - f) \Phi \left(f ^ {\prime}; \mu (\mathbf {x} | \mathcal {D} _ {t}), \sigma^ {2} (\mathbf {x} | \mathcal {D} _ {t})\right) + \sigma^ {2} (\mathbf {x} | \mathcal {D} _ {t}) \phi (f ^ {\prime}; \mu (\mathbf {x} | \mathcal {D} _ {t}), \sigma^ {2} (\mathbf {x} | \mathcal {D} _ {t})) , \\ \end{array}
$$
Algorithm 1 Bayesian Optimization algorithm [90].
Input: Initial observed data  $\mathcal{D}_t$  , a black-box objective function  $f$  total number of BO iterations  $T$    
Output: The best recommendation about the global optimizer  $\mathbf{x}^*$    
for  $t = 1,\dots ,T$  do Select the next  $\mathbf{x}_{t + 1}$  by maximizing acquisition function  $\alpha (\mathbf{x}|\mathcal{D}_t)$  Evaluate the objective function at  $f_{t + 1} = f(\mathbf{x}_{t + 1})$ $\mathcal{D}_{t + 1}\gets \mathcal{D}_t\cup (\mathbf{x}_{t + 1},f_{t + 1})$  Update the surrogate model with  $\mathcal{D}_{t + 1}$    
end for