Title: einspace: Searching for Neural Architectures from Fundamental Operations
Venue: NeurIPS 2024
Accepted: yes
Keywords: neural architecture search

nas

deep learning architectures

context-free grammars

cfg

pcfg

neural networks

search space
Abstract: Neural architecture search (NAS) finds high performing networks for a given task. Yet the results of NAS are fairly prosaic; they did not e.g. create a shift from convolutional structures to transformers. This is not least because the search spaces in NAS often aren’t diverse enough to include such transformations *a priori*. Instead, for NAS to provide greater potential for fundamental design shifts, we need a novel expressive search space design which is built from more fundamental operations. To this end, we introduce `einspace`, a search space based on a parameterised probabilistic context-free grammar. Our space is versatile, supporting architectures of various sizes and complexities, while also containing diverse network operations which allow it to model convolutions, attention components and more. It contains many existing competitive architectures, and provides flexibility for discovering new ones. Using this search space, we perform experiments to find novel architectures as well as improvements on existing ones on the diverse Unseen NAS datasets. We show that competitive architectures can be obtained by searching from scratch, and we consistently find large improvements when initialising the search with strong baselines. We believe that this work is an important advancement towards a transformative NAS paradigm where search space expressivity and strategic search initialisation play key roles.

einspace : Searching for Neural Architectures from Fundamental Operations

Abstract

Neural architecture search (NAS) finds high performing networks for a given task. Yet the results of NAS are fairly prosaic; they did not e.g. create a shift from convolutional structures to transformers. This is not least because the search spaces in NAS often aren't diverse enough to include such transformations apriori . Instead, for NAS to provide greater potential for fundamental design shifts, we need a novel expressive search space design which is built from more fundamental operations. To this end, we introduce einspace , a search space based on a parameterised probabilistic contextfree grammar. Our space is versatile, supporting architectures of various sizes and complexities, while also containing diverse network operations which allow it to model convolutions, attention components and more. It contains many existing competitive architectures, and provides flexibility for discovering new ones. Using this search space, we perform experiments to find novel architectures as well as improvements on existing ones on the diverse Unseen NAS datasets. We show that competitive architectures can be obtained by searching from scratch, and we consistently find large improvements when initialising the search with strong baselines. We believe that this work is an important advancement towards a transformative NAS paradigm where search space expressivity and strategic search initialisation play key roles.

1 Introduction

Thegoalofneuralarchitecture search (NAS) [14, 42] is to automatically find a network architecture for a given task, removing the need for expensive human expertise. NAS uses (i) a defined search space of all possible architectures that can be chosen, and (ii) a search algorithm e.g. [68, 58, 40] to navigate through the space, selecting the most suitable architecture with respect to search objectives. Despite significant research investment in NAS, with over 1000 papers released since 2020 [59], manually designed architectures still dominate the landscape. If someone looks through recent deep learning papers, they will most likely come across a (manually designed) transformer [56], or perhaps a (manually designed) MLP-Mixer [53], or even a (manually designed) ResNet [19]. Why isn't NAS being used instead?

ResNet18

ViT

MLP-Mixer

Derivation tree

Derivation tree

Derivation tree

Part of the problem is that most NAS search spaces are not expressive enough, relying heavily on high-level operations and rigid structures. For example in the DARTS [30] search space, each architecture consists of repeating cells; each cell is a directed acyclic graph where nodes are hidden states, and edges are operations drawn from a fixed set of mostly convolutions. This encodes a very specific priorarchitectures contain convolutions with multi-scale, filter-like structures [50]-making it impossible to discover anything beyond ConvNet characteristics. Indeed, random search [63, 27] is often a strong baseline in NAS; networks sampled from unexpressive spaces behave very similarly [57] which makes it hard to justify an (often expensive) search.

One solution is to take the ex nihilo approach to search space construction. In AutoML-Zero [41] the authors create a very expressive search space that composes basic mathematical operations without any additional priors. However, searching through this space is far too expensive for mainstream use, requiring several thousand CPUs across several days to (re)discover simple operations like linear layers and ReLUs. Recent interest in hierarchical search spaces [47] has enabled the study of search across differing architectural granularities which naturally allows for greater flexibility. However, attempts so far have been limited to single architecture families like ConvNets [47] or transformers [67]. The hybrid search spaces that do exist have limited options both on the operation-level and macro structure [26, 61].

For NAS to be widely used we need the best of both worlds: a search space that is both highly expressive, and in which we can straightforwardly use existing tried-and-tested architectures as powerful priors for search. To this end, we propose einspace : a neural architecture search space based on a parameterised probabilistic context-free grammar (CFG). It is highly expressive , able to represent diverse network widths and depths as well as macro and micro structures. With its expressivity, the space contains disparate state-of-the-art architectures such as ResNets [19], transformers [56, 13] and the MLP-Mixer [53], as shown in Figure 1. Other notable architectures contained in einspace are DenseNet [23], WideResNet (WRN) [64], ResMLP [54] and the Vision Permutator [21].

Werealise our proposed search space through the creation of function-mapping groups that define a broadclass of fundamental network operations and further describe how such elements can be composed into full architectures under the natural recursive capabilities of our CFG. To guarantee the validity of all architectures generated within the expressive space, we first extend our base CFG with parameters that ensure diverse components can be combined into complex structures. Next, we balance the contention between search space flexibility and search space complexity by introducing mild constraints on our search space via branching and symmetry-based priors. Finally, we integrate probabilities into our production rules to further control the complexity of architectures sampled from our space.

To demonstrate the effectiveness of einspace , we perform experiments on the Unseen NAS datasets [16]-eight diverse classification tasks including vision, language, audio, and chess problems-using simple random and evolutionary search strategies. We find that in such an expressive search space, the choice of search strategy is important and random search underperforms. When using the powerful priors of human-designed architectures to initialise the search, we consistently find

both large performance gains and significant architectural changes. Code to reproduce our experiments is available at https://github.com/linusericsson/einspace .

Using only simple search strategies, we can still identify competitive architectures, indicating that further refining these strategies in einspace could lead to significant advancements. We hope that this novel perspective on search spaces-focusing on expressiveness and incorporating the priors of existing state-of-the-art architectures-has the potential to drive NAS research towards a new paradigm.

2 Background

Neural architecture search

The search space used in NAS has a significant impact on results [63, 65]. This has facilitated the need to investigate search space design alongside the actual search algorithms [37]. Early macro design spaces [24, 68] made use of naive building blocks while accounting for skip connections and branching layers. Further design strategies have looked at chain-structured [3, 4, 6, 43], cell-based [69, 66, 30, 15] and hierarchical approaches. Hierarchical search spaces have been shown to be expressive and effective in reducing search complexity and methods include factorised approaches [52], n -level hierarchical assembly [28, 29, 48], parameterisation of hierarchical random graph generators [45] and topological evolutionary strategies [35]. Additional work on search spaces have proposed new candidate operations and module designs such as hand-crafted multi-branch cells [51], tree-structures [3], shuffle operations [32], dynamic modules [22], activation functions [38] and evolutionary operators [10]. In AutoML-Zero [41], the authors try to remove human bias from search space construction by defining a space of basic mathematical operations as building blocks.

The pioneering work of [47] constructs search spaces using CFGs. We take this direction further and construct einspace as a probabilistic CFG allowing for unbounded derivations, balanced by careful tuning of the branching rate. We aim to strike a balance between the level of complexity in the search space and incorporating components from diverse state-of-the-art architectures. Crucially, our space enables flexibility in both macro structure and at the individual operation level. While previous search spaces can be instantiated for specific architecture classes[30, 12], our single space incorporates multiple classes in one, ConvNets, transformers and MLP-only architectures. Such hybrid spaces have been explored before [26], but they have been limited in their flexibility, offering only direct choices between convolution and attention operations and disallowing the construction of novel components.

Prominent search strategies employed for NAS include Bayesian optimisation [34, 58], reinforcement learning [66, 68, 69] and genetic algorithms [5, 39, 40]. A popular thread of work, towards improving computational efficiency via amortising training cost, involves the sharing of weights between different architectures via a supernet [1, 3, 9, 18, 30, 31]. Efficiency has been further improved by sampling only a subset of supernet channels [60], thus reducing both space exploration redundancies and memory consumption. Alternative routes to mitigating space requirements have considered both architecture and operation-choice pruning [7, 15]. We however highlight that random search often proves to be a very strong baseline [27, 63]; a consequence of searching within narrow spaces. This is commonly the case for highly engineered search spaces that contain a high fraction of strong architectures [59]. Contrasting this, in our einspace we observe that random search across many tasks performs poorly, underpinning the value of a good search strategy for large, diverse search spaces [2, 41].

Context-free grammars

Acontext-free grammar (CFG; [20]) is a tuple ( N, Σ ,R,S ) , where N is a finite set of non-terminal symbols, Σ is a finite set of terminal symbols, R is the set of production rules -where each rule maps a non-terminal A ∈ N to a string of non-terminal or terminals A → ( N ∪ Σ) + -and S is the starting symbol . ACFGdescribes a context-free language, containing all the strings that the CFG can generate. Byrecursively selecting a production, starting with the rules containing the starting symbol, we can generate strings within the grammar. CFGs can be parameterised : each non-terminal, in each rule in R , is annotated with parameters p 1 ,...,p n that influence the production. These parameters can condition production, based on an external state or contextual information, thus extending the power of the grammar.

A probabilistic context-free grammar (PCFG) associates each production rule with a probability [33]. These define the likelihood of selecting a particular rule given a parent non-terminal. The assigned probabilities allow for stochastic string generation.

3 einspace : A Search Space of Fundamental Operations

Our neural architecture search space, einspace 2 , is introduced here. Based on a parameterised PCFG, it provides an expressive space containing many state-of-the-art neural architectures. We first describe the groups of operations we include in the space, then how macro structures are represented. We then present the CFG that defines the search space and its parameterised and probabilistic extensions.

As a running example we will be constructing a simple convolutional block with a skip connection within einspace , explaining at each stage how it relates to the architecture. The block will consist of a convolution, a normalisation and an activation, wrapped inside a skip connection.

3.1 Fundamental Operations

Eachfundamental operation within einspace takes as input a tensor, either passed as input to the whole network or an intermediate tensor from a previous operation, and operates on it further. An operation can be thought of as a layer in a processing pipeline that defines the overall network. The fundamental operations can be separated into four distinct groups of functions that define their role in a network architecture. The terms one-to-one , one-to-many and many-to-one below refer to the number of input andoutput tensors of the functions within that group. For full details of the operations, see Appendix A.2

Branching . One-to-many functions that direct the flow of information through the network by cloning or splitting tensors. Examples include the branching within self-attention modules into queries, keys and values. In our visualisations, these are coloured yellow .

Aggregation . Many-to-one functions that merge the information from multiple tensors into one. Examples include matrix multiplication, summation and concatenation. In our visualisations, these are coloured purple .

Routing . One-to-one functions that change the shape or the order of the content in a tensor without altering its information. Examples include axis permutations as well as the im2col and col2im operations. In our visualisations, these are coloured green .

Computation . One-to-one functions that alter the information of the tensor, either by parameterised operations, normalisation or non-linearities. Examples include linear layers, batch norm and activations like ReLU and softmax. In visualisations, these are coloured blue . [ S [ M [ C norm]][ M [ C relu]]]

In our example, the skip connection will be handled by a combination of branching and aggregation functions, the convolution is decomposed into the routing functions im2col and col2im , with a linear layer from the computation group between them. The normalisation and activation come from the computation group. In the next subsection, we discuss the larger structures of the architecture.

3.2 Macro Structure

The groups of functions above describe the fundamental operations that make up an architecture. We now describe how these functions are composed in different ways to form larger components.

A module is defined as a composition of functions from above that takes one input tensor and produces one output tensor, with potential branching inside. A module may contain multiple computation and routing operations, but each branching must be paired with a subsequent aggregation operation. Thus, the whole network can be seen as a module that takes a single tensor as input and outputs a single prediction. A network module may itself contain multiple modules, directly pertaining to the hierarchical phrase nature of CFG structures. Wedivide modules into four types, visualised in Figure 2.

norm

relu

linear

identity

clone

add

im2col

linear

col2im

softmax

[

S

[

B

clone][

M1

[

C

linear]][

M2

[

C

identity]][

A

add]]

[

S

[

P

im2col][

M

[

C

linear]][

R

col2im]]

[

S

[

M

[

C

softmax]]]

clone clone

identity

Sequential module . Apair of modules and/or functions that are applied to the input tensor sequentially. Using our grammar, defined in Section3.3, this can be produced using the rule ( M → MM ), or equivalently from the starting symbol S . This also applies to the rules below.

Branching module . Abranching function first splits the input into multiple branches. Each branch is processed by some inner set of modules and/or functions. The outputs of all branches are subsequently merged in an aggregation function. In the grammar below this can be produced by the rule ( M → B M A ).

Routing module . A routing function is applied, followed by a module and/or function. A final routing function then processes the output tensor. In the grammar below this is produced by the rule ( M → P M R ). For more details on the role of the routing module, see Appendix A.3.

Computation module . This module only contains a single function, selected from the one-to-one computation functions described above. While this module is trivial, we will see later how its inclusion is helpful when designing our CFG and its probabilistic extension. In the grammar below this is produced by the rule ( M → C ).

To construct our example, we will be using all four modules. The branching module combines the clone and add functions from before to create a 2-branch structure. One branch is a simple skip connection by using the identity function inside a computation module. The other branch is the more complex sequence. The convolutional layer is created by combining im2col , linear and col2im in a routing module. The norm and activation are each wrapped in a computation module and these are all composed in sequential modules. Figure 2 shows similar module instantiations in action.

3.3 Search Space as a Context-Free Grammar

The following CFG defines our einspace , where uppercase symbols represent non-terminals and lowercase represent terminals. The colours refer to the function groups.

Our networks are all constructed according to the high-level blueprint: backbone → head where head is a predefined module that takes an output feature from the backbone and processes it into a prediction (see Appendix B for more details). The backbone is thus the section of the network that is generated by the above CFG. When searching for architectures we search across different backbones.

Completing our running example, we present the full derivation of the architecture in the CFG in Figure 3.

S

B

clone

M

1

M

3

P

im2col

M

5

C

R

col2im

M

4

M

6

C

norm

M

7

C

M

2

C

identity

A

add

linear relu

3.4 Prior Choices

When designing a search space, we must balance the need for flexibility-which allows more valid architectures to be included-and constraints - which reduce the size of the search space. We can view constraints as imposing priors on which architectures we believe are worth including. As discussed, many previous frameworks are too restrictive; therefore, we aim to impose minimal priors, listed below.

Convolutional prior . Wedesign our routing module to enable convolutions to be easily constructed, while also allowing components like patch embeddings and transpose operations. We thus enforce that a routing function is followed by another routing function later in the module. Moreover, im2col only appears in the production rule of the first routing function ( P ) and col2im in the last ( R ). As shown in Figure 2, to construct a convolution, we start from the rule ( M → P M R ) and derive the following ( P → im2col ), ( M → C → linear ) and ( R → col2im ).

Branching prior . Wealso impose a prior on the types of branching that can occur in a network. The branching functions clone and group can each have a branching factor of 2, 4 or 8. For a factor of 2, we allow each inner function to be unique, processing the two branches in potentially different ways. For branching factors of 4 or 8, the inner function M is repeated as is, processing all branches identically (though all inner functions are always initialised with separate parameters). Symbolically, given a branching factor of 2 we have ( BM 1 M 2 A ) but with a branching factor of 4 we have ( BM 1 M 1 M 1 M 1 A ). Examples of components instantiated by a branching factor of 2 include skip connections, and for 4, or 8, multi-head attention.

3.5 Feature Mode

Different neural architectures operate on different feature shapes. ConvNets maintain 3D features throughout most of the network while transformers have 2D features. To enable such different types of computations in the same network, we introduce the concept of a mode 3 that affects the shape of our features and which operations are available at that point in the network. Before and after each module, we fix the feature tensor to be of one of two specific shapes, depending on which mode we are in.

Im mode . Maintains a 3D tensor of shape (C, H, W) , where C is the number of channels, H is the height and W is the width. Most convolutional architectures operate in this mode.

Col mode . Maintains a 2D tensor of shape (S, D) , where S is the sequence length and D is the token dimensionality. This is the mode in which most transformer architectures operate.

The mode is changed by the routing functions im2col and col2im . Most image datasets will provide inputs in the Im mode, while most tasks that use a language modality will provide it in Col mode.

Our example architecture maintains the Im mode at almost all stages, apart from inside the routing modules where the im2col function briefly puts us in the Col mode before col2im brings us back.

3.6 Parameterising the Grammar

Due to the relatively weak priors we impose on the search space, sampling a new architecture naively will often lead to invalid networks. For example, the shape of the output tensor of one operation may not match the expected input shape of the next. Alternatively, the branching factor of a branching function may not match the branching factor of its corresponding aggregation function.

Wetherefore extend the grammar with parameters. Each rule r now has an associated set of parameters ( s,m,b ) that defines in which situations this rule can occur. When we sample an architecture from the grammar, we start by assigning parameter values based on the expected input to the architecture. For example, they might be the input tensor shape, feature mode and branching factor:

Given this, we can continuously infer the current parameters during each stage of sampling by knowing how each operation changes them. When we expand a production rule, we must choose a rule which has matching parameters. If at some point, the sampling algorithm has no available valid options, it

will backtrack and change the latest decision until a full valid architecture is found. Hence, we ensure that we can sample architectures without risk of obtaining invalid ones.

As an example of this, the CFG rule for P was previously

Enhanced with parameters, this now becomes two rules

This signifies that an im2col operation is not available in the Col mode. Similarly, the available aggregation options depend on the branching factor of the current branching module

3.7 Balancing Architecture Complexity

When sampling an architecture, we construct a decision tree where non-leaf nodes represent decision points and leaf nodes represent architecture operations. In each iteration, we either select a non-terminal module to expand the architecture and continue sampling, or choose a terminal function to conclude the search at that depth. Continuously selecting modules results in a deeper, more complex network, whereas selecting functions leads to a shallower, simpler network. We can balance this complexity by assigning probabilities to our production rules, thereby making a PCFG. Recall our CFG rule

If we choose one of the first three options we are delving deeper in the search tree since there is yet another M to be expanded, but if we choose ( M → C ), the computation-module , then we will reach a terminal function. Thus, to balance the depth of our traversal and therefore expected architecture complexities, we can set probabilities for each of these rules:

The value of p ( M → C | M ) is especially important as it can be interpreted as the probability that we will stop extending the search tree at the current location.

We could set these probabilities to match what we wish the expected depth of architectures to be (for empirical results on the architecture complexity, see Table 10 in the Appendix). However, we can actually ensure that the CFG avoids generating infinitely long architecture strings by setting the probabilities such that the branching rate of the CFG is less than one [8]. For details of how, see Appendix A.4. So, as shown in Figure 4, we set the computation module probability to p ( M → C | M ) = 0 . 32 and the probabilities of the other modules to 1 -0 . 32 3 . For simplicity, all other rule probabilities are uniform.

For a thorough example of how sampling is performed in einspace , please see Appendix A.1.

4 Experiments

4.1 Experimental Details

Datasets

As our search space strives for expressivity and diverse architectures, we adopt a diverse benchmark suite from the recent paper on Unseen NAS [16], containing datasets at different difficulties across

super-critical

sub-critical

0.0

0.2

0.4

0.6

0.8

1.0

p

(

M

C

M

)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Branching rate

Branching rate

p

(

M

C

M

) = 0.31

vision, language, audio and further modalities. We run individual searches on these datasets, that are each split into train, validation and test sets. See Appendix B.2 for the detailed dataset descriptions.

While Unseen NAS forms the basis of this section, we run additional experiments on the diverse NASBench360 benchmark [55] in Appendix C.1, where we beat competing NAS methods on CIFAR100, FSD50K and Darcy Flow, and to the best of our knowledge set a new state-of-the-art on NinaPro.

Search strategy

Weexplore three search strategies within einspace : random sampling, random search, and regularised evolution (RE). Random sampling estimates the average expected test performance from K random architecture samples. Random search samples K architectures and selects the best performer on a validation set. In regularised evolution , we start by constructing an initial population of 100 individuals, which are either randomly sampled from the search space or seeded with existing architectures. For ( K -100) iterations, the algorithm then randomly samples 10 individuals and selects the one with the highest fitness as the parent. This parent is mutated to produce a new child. This child is evaluated and enters the population while the oldest individual in the population is removed, following a first-in-first-out queue structure. An architecture is mutated in three straightforward steps:

Sample a Node : Uniformly at random sample a node in the architecture derivation tree.

Resample the Subtree : Replace the subtree rooted at the sampled node by regenerating it based on the grammar rules. This step allows the exploration of new configurations, potentially altering a whole subtree if a non-leaf node is chosen.

Validate Architecture : Check if the new architecture can produce a valid output in the forward pass, given an input of the expected shape, and that it fits within resource constraints, e.g. GPU memory. If it does, accept it; otherwise, discard and retry the mutation.

Note that these are very simple search strategies, and that there is huge potential to design more intelligent approaches, e.g. including crossover operations in the evolutionary search, using hierarchical Bayesian optimisation [47] or directly learning the probabilities of the CFG [11]. In this work, we focus on the properties of our search space and investigate whether simple search strategies are able to find good architectures, and leave investigations on more complex search strategies for future work.

Baselines

Wecompare these search strategies to PC-DARTS [60], DrNAS [7] and Bonsai-Net [15] with results transcribed from [16]. We also compare to the performance of a trained ResNet18 (RN18). More details on the baselines, training recipes and network instantiations can be found in Appendix B.1

4.2 RandomSampling and Search

In previous NAS search spaces e.g. [30, 62, 12], complex search methods often perform very similarly to random search [27, 63]. Indeed, we can see this in Table 1 comparing the PC-DARTS strategy to DARTS random search.

However for einspace , this is not the case for most datasets. Random sampling improves on pure random guessing (not shown), but is far from the baseline performance of a ResNet18. The random search baseline is also far behind, but intriguingly outperforms baseline NAS approaches on Chesseract.

4.3 Evolutionary Search from Scratch

We now turn to a more sophisticated search strategy. We perform regularised evolution in einspace for 1000 itera- tions across all datasets, initialising the population with 100 random samples. In Table 1 the results are shown in the column named RE(Scratch). The performance of this strategy is significantly higher than random search on several datasets, indicating that the search strategy is more important in an expressive search space like einspace compared to DARTS. Compared to the top performing NAS methods, however, it is significantly behind on some datasets.

4.4 Evolutionary Search from Existing SOTA Architectures

To fully utilise the powerful priors of existing human-designed structures, we now invoke search where the initial population of our evolutionary search is seeded with a collection of existing state-of-the-art architectures.

Wefirst seed the entire population with the ResNet18 architecture. The search applies mutations to these networks for 500 iterations. In Table 1, these results can be found in the RE(RN18) column.

To further highlight the expressivity of einspace , we perform experiments searching from an initial population seeded with a mix of ResNet18, WRN164, ViT and MLP-Mixer architectures. To our knowledge, no other NAS space is able to represent such a diverse set of architectures in a single space. These results are shown in the RE(Mix) column.

Overall, we find that on every single task , we can find an improved version of the initial architecture using RE(RN18) and on all but one using RE(Mix). Moreover, in some cases we can beat the existing state-of-the-art, especially on tasks further from the traditional computer vision setting. In particular,

where previous NAS methods fail-i.e. the Language dataset-the architecture in Figure 6 has a direct improvement over the ResNet18 by 5.76%. See also the architecture in Figure 5 and the collection in Figure 8 in the Appendix for the breadth of structures that are found in einspace .

Wefurther compare einspace to the previous, CFG-based, hNASBench201 from Schrodi et al. [46]. This allows for an initial study on the effects of our search space design choices and, in particular, the increased expressiveness compared to hNASBench201. These results show how einspace compares favourably to a different search space under the same evolutionary search. Overall, we highlight that our search results on einspace are competitive, even with far weaker search space priors.

One dataset where our searches struggle is CIFARTile, a more complex version of the CIFAR10 dataset. While large improvements are made to the baseline networks, they still lag behind other NAS methods. This shows how the strong and restricted focus on ConvNets within the DARTS search space is highly successful for traditional computer vision style tasks that have been common in the literature.

5 Limitations

Our search space, designed for diversity, is extremely large and uniquely unbounded in terms of depth and width. This complexity makes formulating one-shot methods like ENAS [36] or DARTS [30] challenging. Instead, developing an algorithm to learn the probabilities of the PCFG may be more feasible. This approach, however, must address the grammar's context-free nature where sampling probabilities do not consider network depth, feature shape, or previous decisions, although this could be mitigated by using the parameters outlined in Section 3.6. Due to the relatively slow nature of our evolutionary search strategy (see Table 9 in Appendix C.5), we believe that finding more efficient search strategies for expressive spaces like ours is an important and exciting direction for future work.

Another issue is ambiguity arising from the possibility of multiple derivation trees for a single architecture, primarily due to the multiple ways of stacking sequential modules. Moreover, we have found that through sampling and mutation, some architectures' components reduce to the identity operation, from e.g. stacked identity and permute operations within sequential and routing modules. Finding ways to represent the equivalence classes of derivation trees can thus be powerful for reducing effective search space size.

Finally, we designed einspace to be diverse, but some key structures cannot be represented in its current form. There are no options for recurrent computation, as found in RNNs and the new wave of state-space models like Mamba [17]. We believe this can be integrated via the inclusion of a recurrent module that repeats the computation of the components within it-however we leave more detailed exploration of this direction to future work. We also chose to keep the options for activations and normalisation layers as small as possible since in practice the benefit from changing these is minor.

6 Conclusion

Wehave introduced einspace : an expressive NAS search space based on a parameterised PCFG. We show that our work enables the construction of a comprehensive and diverse range of existing state-ofthe-art architectures and can further facilitate discovery of novel architectures directly from fundamental operations. With only simple search strategies, we report competitive resulting architectures across a diverse set of tasks, highlighting the potential value of defining highly expressive search spaces. We further demonstrate the utility of initialising search with existing architectures as priors. We believe that future work on developing intelligent search strategies within einspace can lead to exciting advancements in neural architectures.

References

Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc V Le. Understanding and simplifying one-shot architecture search. In International Conference on Machine Learning , 2018.

Gabriel Bender, Hanxiao Liu, Bo Chen, Grace Chu, Shuyang Cheng, Pieter-Jan Kindermans, and Quoc V Le. Can weight sharing outperform random architecture search? An investigation with tunas. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition , 2020.

Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. In International Conference on Learning Representations , 2019.

Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. In International Conference on Learning Representations , 2020.

Angelica Chen, David Dohan, and David So. Evoprompting: Language models for code-level neural architecture search. Advances in Neural Information Processing Systems , 2023.

Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021.

Xiangning Chen, Ruochen Wang, Minhao Cheng, Xiaocheng Tang, and Cho-Jui Hsieh. DrNAS: Dirichlet neural architecture search. In International Conference on Learning Representations , 2020.

Zhiyi Chi. Statistical properties of probabilistic context-free grammars. Computational Linguistics , 25(1):131-160, 1999.

Xiangxiang Chu, Bo Zhang, and Ruijun Xu. FairNAS: Rethinking evaluation fairness of weight sharing neural architecture search. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021.

Yuanzheng Ci, Chen Lin, Ming Sun, Boyu Chen, Hongwen Zhang, and Wanli Ouyang. Evolving search space for neural architecture search. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021.

Shay Cohen. Latent-Variable PCFGs: Background and applications. In Proceedings of the 15th Meeting on the Mathematics of Language , 2017.

Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. In ICLR , 2020.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16 × 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021.

Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research , 20(55):1-21, 2019.

Rob Geada, Dennis Prangle, and A Stephen McGough. Bonsai-net: One-shot neural architecture search via differentiable pruners. arXiv preprint arXiv:2006.09264 , 2020.

Rob Geada, David Towers, Matthew Forshaw, Amir Atapour-Abarghouei, and A Stephen McGough. Insights from the use of previously unseen neural architecture search datasets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024.

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 , 2023.

Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In Proceedings of the European Conference on Computer Vision , 2020.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2016.

John E Hopcroft, Rajeev Motwani, and Jeffrey D Ullman. Introduction to automata theory, languages, and computation. ACMSigAct News , 32(1):60-65, 2001.

Qibin Hou, Zihang Jiang, Li Yuan, Ming-Ming Cheng, Shuicheng Yan, and Jiashi Feng. Vision permutator: A permutable MLP-like architecture for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(1):1328-1334, 2023.

Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2018.

Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017.

Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P Xing. Neural architecture search with bayesian optimisation and optimal transport. Advances in Neural Information Processing Systems , 2018.

Jacques Labelle and Yeong-Nan Yeh. Generalized dyck paths. Discrete Mathematics , 82(1): 1-6, 1990. ISSN 0012-365X. doi: https://doi.org/10.1016/0012-365X(90)90039-K. URL https://www.sciencedirect.com/science/article/pii/0012365X9090039K .

Changlin Li, Tao Tang, Guangrun Wang, Jiefeng Peng, Bing Wang, Xiaodan Liang, and Xiaojun Chang. BossNAS: Exploring hybrid CNN-transformers with block-wisely self-supervised neural architecture search. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021.

Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Conference on Uncertainty in Artificial Intelligence , 2019.

Chenxi Liu, Liang-Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L Yuille, and Li Fei-Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019.

Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. Hierarchical representations for efficient architecture search. In International Conference on Learning Representations , 2018.

Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations , 2019.

Zhichao Lu, Ian Whalen, Vishnu Boddeti, Yashesh Dhebar, Kalyanmoy Deb, Erik Goodman, and Wolfgang Banzhaf. NSGA-Net: neural architecture search using multi-objective genetic algorithm. In Proceedings of the Genetic and Evolutionary Computation Conference , 2019.

Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European Conference on Computer Vision , 2018.

Christopher ManningandHinrichSchutze. Foundationsofstatistical natural language processing . MITpress, 1999.

Hector Mendoza, Aaron Klein, Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. Towards automatically-tuned neural networks. In Proceedings of the Workshop on Automatic Machine Learning . PMLR, 2016.

Risto Miikkulainen, Jason Liang, Elliot Meyerson, Aditya Rawal, Dan Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, et al. Evolving deep neural networks. In Artificial Intelligence in the Age of Neural Networks and Brain Computing , pages 269-287. Elsevier, 2024.

Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International Conference on Machine Learning , 2018.

Ilija Radosavovic, Justin Johnson, Saining Xie, Wan-Yen Lo, and Piotr Dollár. On network design spaces for visual recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2019.

Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941 , 2017.

Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In International Conference on Machine Learning , 2017.

Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence , 2019.

Esteban Real, Chen Liang, David So, and Quoc V Le. AutoML-zero: Evolving machine learning algorithms from scratch. In International Conference on Machine Learning , 2020.

Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. A comprehensive survey of neural architecture search: Challenges and solutions. ACM Computing Surveys (CSUR) , 54(4):1-34, 2021.

Nicholas Roberts, Mikhail Khodak, Tri Dao, Liam Li, Christopher Ré, and Ameet Talwalkar. Rethinking neural operations for diverse tasks. Advances in Neural Information Processing Systems , 2021.

Alex Rogozhnikov. Einops: Clear and reliable tensor manipulations with einstein-like notation. In International Conference on Learning Representations , 2022.

RobinRu,PedroEsperanca, andFabioMariaCarlucci. Neuralarchitecture generator optimization. Advances in Neural Information Processing Systems , 2020.

Simon Schrodi, Danny Stoll, Binxin Ru, Rhea Sukthanker, Thomas Brox, and Frank Hutter. Construction of hierarchical neural architecture search spaces based on context-free grammars. In Advances in neural information processing systems , 2023.

Simon Schrodi, Danny Stoll, Binxin Ru, Rhea Sukthanker, Thomas Brox, and Frank Hutter. Construction of hierarchical neural architecture search spaces based on context-free grammars. Advances in Neural Information Processing Systems , 2024.

David So, Wojciech Ma´ nke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V Le. Primer: Searching for efficient transformers for language modeling. In Advances in Neural Information Processing Systems , 2021.

Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for Simplicity: The All Convolutional Net. In ICLR Workshops , 2015.

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2015.

Mingxing Tan and Quoc V Le. Mixconv: Mixed depthwise convolutional kernels. In Proceedings of the British Machine Vision Conference , 2019.

Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. MnasNet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019.

Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. MLP-Mixer: An all-MLP Architecture for Vision. In Advances in Neural Information Processing Systems , 2021.

Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou. ResMLP: Feedforward Networks for Image Classification With Data-Efficient Training. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(4):5314-5321, 2023.

RenboTu,NicholasRoberts, Mikhail Khodak, Junhong Shen, Frederic Sala, and Ameet Talwalkar. NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Tasks. In Neural Information Processing Systems Datasets and Benchmarks Track , 2022.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , 2017.

Xingchen Wan, Binxin Ru, Pedro M Esperança, and Zhenguo Li. On redundancy and diversity in cell-based neural architecture search. In International Conference on Learning Representations , 2022.

Colin White, Willie Neiswanger, and Yash Savani. BANANAS: Bayesian optimization with neural architectures for neural architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence , 2021.

Colin White, Mahmoud Safari, Rhea Sukthanker, Binxin Ru, Thomas Elsken, Arber Zela, Debadeepta Dey, and Frank Hutter. Neural Architecture Search: Insights from 1000 papers. arXiv preprint arXiv:2301.08727 , 2023.

Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. Pc-darts: Partial channel connections for memory-efficient architecture search. In International Conference on Learning Representations , 2020.

Shangshang Yang, Xiaoshan Yu, Ye Tian, Xueming Yan, Haiping Ma, and Xingyi Zhang. Evolutionary neural architecture search for transformer in knowledge tracing. Advances in Neural Information Processing Systems , 2023.

Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas-bench-101: Towards reproducible neural architecture search. In International Conference on Machine Learning , 2019.

Kaicheng Yu, Christian Sciuto, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluating the search phase of neural architecture search. In International Conference on Learning Representations , 2020.

Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks. In Proceedings of the British Machine Vision Conference , 2016.

Xinbang Zhang, Zehao Huang, Naiyan Wang, Shiming Xiang, and Chunhong Pan. You only search once: Single shot neural architecture search via direct sparse optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43(9):2891-2904, 2020.

Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu. Practical block-wise neural network architecture generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2018.

Yuan Zhou, Haiyang Wang, Shuwei Huo, and Boyu Wang. Hierarchical full-attention neural architecture search based on search space compression. Knowledge-Based Systems , 269:110507, 2023.

Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations , 2017.

Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2018.

A Search Space Details

A.1 Sampling

In this section, we explain the process of sampling an architecture from our parameterised PCFG through an example. We specifically focus on how the running example from the main paper, a simple convolutional block with a skip connection, could be generated.

We begin the process with the starting symbol S , which could produce several forms based on the production rules, including ( M M ), ( B M A ), or ( P M R ). Since our block includes a skip connection, the macro structure of our architecture is best represented by a branching module ( B M A ).

Within this module, we expand the string from left to right, thereby starting with ( B ). The specific branching operation that fits our goal is ( B → clone ) as we wish to later combine a transformed version of the tensor with itself. Since we have two branches, they are produced separately (see branching prior) and our module becomes ( B M 1 M 2 A ).

For the first branch, ( M 1 ), we need a set of components that constitute a convolution followed by batch normalisation and an activation. Since this involves several composed operations we first expand into a sequential module ( M 1 → M 3 M 4 ). The first of these operations represents the convolution. Within einspace , a convolution, conv ( x ) , is decomposed into the three operations, col2im ( linear ( im2col ( x ))) . Thus, in our grammar we unfold it as a routing module, ( M 3 → P M 5 R ) which further produces ( P → im2col ), ( M 5 → C → linear ) and ( R → col2im ). The normalisation and activation are then generated under ( M 4 ), defined as another sequential module ( M 4 → M 6 M 7 ) with ( M 6 → C → norm ) and ( M 7 → C → relu ). The second branch, M 2 , acts as a skip connection and is thus derived as ( M 2 → C → identity ).

To finalise the architecture, the aggregation symbol ( A ) merges the tensors back into one unit. To complete the residual connection, we use the rule ( A → add ).

The full derivation tree in this example is shown in Figure 3 of the main paper. This general sampling process allows the creation of complex neural network architectures from a structured and interpretable set of rules.

A.2 Fundamental Operations

Our grammar in the main paper is somewhat simplified. There are some fundamental operations that have hyperparameters that allow multiple versions to be chosen. They are detailed here.

Branching functions . For the production rule ( B → clone | group ), the full set of options is:

where b refers to the branching factor and dim is the dimension we group over.

Aggregation functions . Similarly, for the production rule ( A → matmul | add | concat ), the full set of options is:

where scaled=True makes the matmul operation equivalent to the scaled dot product used in many transformer architectures, dim is the dimension we concatenate over and b is the branching factor.

Routing functions . The im2col and col2im functions are implemented to offer the standard functionality that enables convolutional operations, including variables that set the kernel sizes, stride, dilation and padding. Below are the full set of options for im2col . The col2im only takes the predicted output

shape as a parameter so we can include only a single version of this operation.

where k is the kernel size, s is the stride and p the padding.

For the permute operation, there are six versions of the order parameter o . There is one for the Col mode and five for the Im mode:

For completeness, the identity operation can also be included, making two versions in the Col mode (with identity = permute(o=(1,2)) ) and six in the Im mode (with identity = permute(o=(1,2,3)) ).

Computation functions . For linear layers, we vary the output dimension d across powers of two:

The norm operation takes on the batch-norm functionality in the Im mode and layer-norm in Col mode. The softmax is just a softmax operation applied to the final dimension, the relu activation is implemented as the single option leaky-relu and pos-enc is a learnable positional encoding.

A.3 Patch Embeddings and Convolutions

In this section we provide some more information about how the routing module can represent common components.

The routing module, ( M → P M R ), puts a prior on certain types of operations inside our architectures. Apatch embedding, such as those found in many transformers, is achieved by the following derivation: ( M → im2col linear identity ), while a convolution can be obtained by ( M → im2col linear col2im ). In terms of the process required to sample such combinations, the first is easy since there are no dependencies between the operations. The second, however, is more complicated and requires some discussion.

Let x be a tensor in R 3 × 32 × 32 andlet us consider a routing module containing the functions: im2col , linear , col2im . The functions will be applied in order to the input tensor, giving us

The shapes of the intermediate and final tensors { x ′ ,x ′′ ,y } depend on several function hyperparameters. These are listed below.

The input tensor has height dimension h in and width w in . The im2col operation will extract column vectors from this space a number of times depending on the kernel size k in , stride s in and padding p in

values in the table above. The number of column vectors equals

which in our case gives l =256 . The shapes of all intermediate tensors can therefore be written as in Table 3.

Table 3: Tensor shapes in the forward pass of our convolution component; c in =3 ,h in =32 ,w in =32 in this example.

Therefore, to successfully apply the col2im function, the constraint l = h out × w out must be satisfied. From Equation 30 we can see that the output height and width can be predicted by the im2col parameters

So, in practice, the im2col operation fully defines the behaviour of the convolution-apart from the number of output channels defined by the linear layer-and the col2im only rearranges the tensor back into its correct 3D form. This is trivial in the case where h in = w in since h out = w out = √ l . However, if h in = w in , then the predicted output shapes must be remembered until the col2im operation is applied.

̸

Thus, in our sampling and mutation algorithm, whenever an im2col operation is sampled, we must store the predicted output shape until a corresponding col2im is applied. Additionally, the dimensionality of l must not change in the connecting branch as it would break the constraint l = h out × w out .

A.4 Branching Rate of the CFG

If a PCFG is consistent, the probabilities of all finite derivations sum to one, or equivalently, it has a zero probability of generating infinite strings or derivations [8]. For us, that means a sampled architecture can not be infinitely large, and that the sampling algorithm will halt with probability one. In order to check if a CFG is consistent, we can inspect the spectral radius ρ of its non-terminal expectation matrix [8]. If ρ< 1 , then the PCFG is consistent. This expectation matrix is indexed by the non-terminals in the grammar (both the columns and the rows), and at each cell it provides the expected number of instances the column non-terminal being generated from the row non-terminal by summing the probabilities of the row non-terminal multiplied by the count of the column non-terminal in each rule.

Wecan also solve a (simple, in our case) set of linear equations in order to compute the expected length of an architecture string, ℓ , as a function of the rule probabilities. More specifically, denote by E [ A ] the expected length of string generated by a nonterminal in the grammar A . Then ℓ = E [ S ] , where:

In the above, α is the right hand side of a rule and α i varies over the nonterminals in that right hand side.

A.5 Adjustments for One-Dimensional Tasks

We have presented our new search space primarily with the application to two-dimensional data in mind-where, confusingly, the input tensor is of the shape (C, H, W) , i.e. two spatial dimensions and one channel dimension. In order to search for architectures on 1D datasets-with input tensors of shape (S, D) , i.e. one sequence dimension and one token dimension-we need to adjust the einspace CFG to make it compatible.

Wereplace the routing functions im2col and col2im with operations that perform the full 1D convolution directly-without decomposing the operation-and place them into the computation function group. Below are the new conv1d functions in the 1D variant of einspace

Some versions of the group , concat and permute operations are also removed as they operate on a dimension that now doesn't exist. Below is the adjusted grammar for this variant.

In the NASBench360 results presented in Table C.1, this 1D variant of einspace is used for the datasets ECG, Satellite and Deepsea. The baseline WideResNet-16-4 architecture is also adjusted to handle the 1D data, as described in [55].

A.6 Size of the Search Space

In this section we will discuss the size of the introduced search space.

Our einspace grammar contains recursive rules, e.g. ( M → M M ). Due to this recursion the grammar generates a language with an infinite number of strings. We know from Section A.4 that since our grammar is consistent, the architecture sampling process always terminates. This means every architecture derivation tree is finite in size. So, while einspace covers an infinite number of architectures, each such architecture is finite. Let the architecture string denote the left-to-right sequence of leaves of a derivation tree. We define S n to be the set of all architecture strings of length n . As we will show next, the size of S n grows exponentially with n . To do this, we first introduce a few new concepts.

A Dyck language [25] is a formal language consisting of strings that represent balanced and properly nested sequences of pairs of symbols, typically parentheses, where each opening symbol must have a corresponding closing symbol, and at no point in the string can the number of closing symbols exceed the number of opening symbols. A Dyck language can be generated by a context-free grammar defined over a set of terminal symbols Σ= { ( , ) } . Generalising this, Dyckk denotes a Dyck language with k distinct pairs of matching symbols, e.g. Dyck2 has Σ= { ( , ) , [ , ] } . The growth of a Dyck1 language is described by the Catalan numbers, reflecting the exponential increase in valid strings as the length of the input increases, i.e. the number of strings with m matching pairs of symbols is the m th Catalan number

C m = 1 m +1 ( 2 m m ) . Asymptotically, the number of such strings grows as O (4 m ) . Dyck-2 languages do not follow the Catalan numbers but still grow as O (4 m ) .

Wecan rewrite (and somewhat simplify) our einspace CFGfrom Section 3.3 in order to generate a Dyck language

The set of non-terminal symbols { B , A , P , R } has been replaced with the terminal set of parentheses and square brackets { ( , ) , [ , ] } and, for simplicity, the terminal symbols of C have been replaced by integers. This makes it a Dyck-2 language with the addition of the C symbol that can be expanded into several terminal options.

While a normal Dyck2 language grows as O (4 m ) , ours is more complex. Firstly, each non-terminal symbol M will eventually produce a C which leads to a terminal. Let the number of terminals derivable from C be χ , and the number of occurrences of the symbol C in the derivation of an architecture be c . The number of possible strings we can obtain from just the C symbols thus grows as O ( χ c ) . Secondly, the brackets we introduced can, in our original CFG, themselves be derived into terminal symbols for branching, aggregation and routing functions via { B , A , P , R }. Let the maximum number of terminals derivable from any of these be β . Wealready know that the number of balanced brackets is m , meaning the total number of terminals coming from these is 2 m . Therefore, the number of architecture strings in S n grows as

where m is the combined number of branching and routing modules, c is the number of computation functions, and n = 2 m + c . Some architectures will contain many nested branching or routing functions and be dominated by the first two factors, and some will contain many sequential modules and computation functions and be dominated by the third.

A.7 Comparison to Other Search Spaces

In Table 4 we compare einspace to other popular search spaces in the literature along several axes. Wehighlight some of the most important differences here:

einspace uniquely unifies multiple architectural families (ConvNets, Transformers, MLPonly) into one single expressive space while the CFG-based framework of Schrodi et al. [46] has variations of spaces centred around ConvNets only, and a separate instantiation focusing only on Transformers.

einspace extends to probabilistic CFGs. This enables a set of benefits that include (i) allowing experts to define priors on the search space via probabilities, and (ii) enabling a broader range of search algorithms that incorporate uncertainty estimates inside the search itself.

einspace contains recursive production rules (e.g. M → MM), meaning the same rule can be expanded over and over again, providing an infinite space and a very high flexibility in macro structures. In contrast, [46] instead focuses on fixed hierarchical levels that limits the macro structure to a predefined-though very large-set of choices.

einspace encodes architectures in the form of derivation trees. These allow for mutations that can effectively alter both the macro structure and the individual components of an architecture. Modifications of this class are more difficult if using e.g. the more rigid graph encodings.

Overall, we highlight that our experimental search results on einspace are competitive with previous work, even with far weaker priors on the search space.

B Implementation and Experimental Details

B.1 Networks

Baseline networks

Weuse convolutional baselines of ResNet18 [19] and WideResNet-16-4 [64]. Both stems use a 3 × 3 convolution instead of the standard 7 × 7 as most input shapes in the datasets we use are small. The former contains a max-pooling layer in the stem, which for simplicity we decide to not represent in our search space. Instead we modify the pooling operation and replace it with a 3 × 3 convolution with stride 2. This has been shown to be equally powerful [49] and in our experiments we find that it performs similarly. Our ViT model is a small 4-layer network with a patch size of 4, model width of 512, and 4 heads. The MLP-Mixer shares the same patch embedding stem with a patch size of 4. It has 8 layers and, similarly, a model width of 512. The channel mixer expands the dimension by 4 and the token mixer cuts it in half. The models have the following number of parameters (given an input image of shape [3, 64, 64]): Resnet18: 11.2M, WRN16-4: 2.8M, ViT: 4.4M and MLP-Mixer: 6.5M.

Network head

Every network that is instantiated contains a few common operations. For classification tasks, the network head takes the following form: the output features of the sampled backbone are reduced to their channel dimension via reduce(x, 'B C H W -> B C', 'mean') 4 or reduce(x, 'B C H -> B C', 'mean') , depending on if the input features x are in Im or Col mode. Second, a final linear layer that maps the channel dimension to the target dimension (i.e., the number of classes) is appended. For dense prediction tasks: the head contains an adaptive average pooling layer that upsamples the two final dimensions of the backbone features to the target image size. If the features are in the Col mode, we insert a new dimension after the batch size. Then regardless of mode, a linear layer adjusts the number of channels to the target channel number.

Network training

Each network is trained and evaluated separately with no weight sharing. During the search phase we minimise the loss on a train split and compute the validation metric on a validation set. To evaluate the final chosen module, we retrain on train+val splits and evaluate on test. To speed up our experiments, the inner loop optimisation of architectures uses fewer epochs compared to the evaluation phase. All networks are trained using the SGD optimizer with momentum of 0.9. The values for learning rate, weight decay, batch size and more can be found in Table 5.

B.2 Datasets

Our experimental evaluation covers 19 different datasets, with sizes ranging from thousands of data points, to a million (Satellite from NASBench360), and spatial resolutions of up to 256 × 256 (Cosmic from NASBench360). In this section we briefly describe these parts in detail.

Wefollowed the official instructions of Unseen NAS [16] to setup the datasets. Descriptions of each dataset follow:

AddNIST

This dataset is derived from the MNIST dataset. Specifically, each RGB image is computed by stacking three MNIST images in the channel dimension. Each image has the shape 3 × 28 × 28 . It has a total of 20 categories; the class label is computed by summing the MNIST labels in all three channels. Among the 70,000 images, 45,000 are used for training, 15,000 are used for validation, and 10,000 images are used for testing.

Language

Language consists of six-letter words extracted from dictionaries of ten Latin alphabet languages: English, Dutch, German, Spanish, French, Portuguese, Swedish, Zulu, Swahili, and Finnish. Words containing diacritics or the letters 'y' and 'z' are excluded, making an alphabet of 24 letters. Each sample consists of four words encoded into a binary image of shape 1 × 24 × 24 . The task is to predict the language of the sample. Along the y-axis are the letter positions in the concatenated 24-letter string, and along the x-axis are the letters in the alphabet. As an example, a one in the position (0, 0) indicates that the first letter in the string is an 'a'. The dataset is split into 50,000 training samples, 10,000 validation samples, and 10,000 test samples.

MultNIST

MultNIST is a dataset designed similarly to AddNIST, originating from the same research objective. Each channel of the 3 channel images contains an image from the MNIST dataset. Each image has the shape 3 × 28 × 28 . The dataset is divided into 50,000 training images, 10,000 validation images, and 10,000 test images. Unlike AddNIST, MultNIST comprises ten classes (0-9), the label for each MultNIST image is computed using the formula l =( r × g × b ) mod10 , where r , g and b are the MNISTlabels of the red, green, and blue channels, respectively.

CIFARTile

CIFARTile is a dataset where each image is a composite of four CIFAR-10 images arranged in a 2 × 2 grid, making each sample an image of shape 3 × 64 × 64 . The dataset is divided into 45,000

training images, 15,000 validation images, and 10,000 test images. CIFARTile has four classes (0-3), determined by the number of distinct CIFAR-10 classes in each grid, minus one.

Gutenberg

Gutenberg is a dataset sourced from Project Gutenberg. It includes texts from six authors, with basic text preprocessing applied: punctuation removal, diacritic conversion, and elimination of structural words. The dataset contains consecutive sequences of three words (3-6 letters each), padded to 6 characters and concatenated into 18-character strings. These strings are converted into images with size 1 × 27 × 18 , with the x-axis representing character positions and the y-axis representing alphabetical letters or spaces. The task is to predict the author of each sequence. The dataset is split into 45,000 training, 15,000 validation, and 6,000 test images.

Isabella

Isabella is a dataset derived from musical recordings of the Isabella Stewart Gardner Museum, Boston. It includes four classes based on the era of composition: Baroque, Classical, Romantic, and 20th Century. The recordings are split into five-second snippets and converted into 64-band spectrograms, resulting in images with dimensions 1 × 64 × 128 . The dataset is divided into 50,000 training images, 10,000 validation images, and 10,000 test images, ensuring no overlap of recordings across splits. The task is to predict the era of composition from the spectrogram.

GeoClassing

GeoClassing is based on the BigEarthNet dataset. It uses satellite images initially labeled for ground-cover classification but reclassified by the European country they depict. The dataset includes ten classes: Austria, Belgium, Finland, Ireland, Kosovo, Lithuania, Luxembourg, Portugal, Serbia, and Switzerland. Each image is of size 3 × 64 × 64 . The dataset is split into 43,821 training images, 8,758 validation images, and 8,751 test images. The task is to predict the country depicted in each image based on topology and ground coverage.

Chesseract

Chesseract is a dataset derived from public chess games of eight grandmasters. The dataset consists of the final 15% of board states from these games. Each position is one-hot encoded by piece type and color, resulting in 1 × 8 × 8 images. The dataset is divided into 49,998 training images, 9,999 validation images, and 9,999 test images, ensuring no positions from the same game appear across splits. Each image is classified into one of three classes: White wins, Draw, or Black wins. The task is to predict the game's result based on the given board position. We pad the input with 5 zero-valued pixels to make a 12 × 18 × 18 tensor.

Wefollow the official instructions of NASBench360 [55] to setup the datasets. Dataset descriptions follow:

CIFAR100

CIFAR100isawidelyknownimageclassification dataset with 100 fine-grained classes. Each image is of size 3 × 32 × 32 . The dataset is split into 40,000 training, 10,000 validation and 10,000 testing images. We preprocess the images by applying random crops, horizontal flips, and normalisation.

Spherical

Spherical dataset consists on classifying spherically projected CIFAR100 images. Specifically, CIFAR images are projected to the northern hemisphere with a random rotation. Each image is of size 3 × 60 × 60 . Spherical dataset uses the same split ratios as CIFAR100. In this case, there is no data augmentation nor pre-processing steps.

NinaPro

NinaPro is a dataset for classifying hand gestures given their electromyography signals. EMG data signals are collected with two Myo armbands as wave signals. Wave signals, along with their wavelength and frequency, are processed 2D signals of shape 1 × 16 × 52 . There are 18 classes, which are heavily imbalanced, with the neutral position amounting for 65% of all gestures. The dataset is split into 2,638 training samples, 659 validation samples, and 659 testing samples. No further data augmentation is applied.

FSD50K

Freesound Dataset 50k (FSD50K) is a collection of 51,197 sound clips, categorised into 200

categories, where each clip can receive multiple labels. The task is to classify the sound event from its log-mel spectrogram, and the performance is computed via the mean average precision (mAP). Weresample the raw audio files at a frequency of 22 , 050 Hz and convert them into 96-band log-mel spectrograms. From these longer audio files, we extract shorter, overlapping 1-second segments, resulting in an input size of 1 × 101 × 96 . During training, one randomly-selected segment per clip is used, rather than all segments. Background noise is added to 75% of the training data as part of data augmentation. The validation set consists of 4 , 170 clips and the test set of 10,231 clips.

Darcy Flow

Darcy Flow is a regression task for predicting the solution of a 2D PDE at a predefined later stage given some 2D initial conditions. The input is a 1 × 85 × 85 image describing the initial state of the fluid. The output should match the same dimensions. The dataset is split into 900 images for training, 100 for validation, and 100 for test. Input data is normalised.

PSICOV

This dataset concerns the use of neural networks in the protein folding prediction pipeline. It uses proteins from one source, DeepCov, for training and validation. DeepCov contains 3,456 proteins each with shape 57 × 128 × 128 . The validation set consists of 100 proteins, with the rest used for training. The test set comes from another source, PSICOV, with 150 proteins. These proteins come in features of a different shape, 1 × 512 × 512 . Due to this, the evaluated network takes each non-overlapping 1 × 128 × 128 patch as input. The labels represent pairwise distances between residues. The evaluation metric is mean absolute error (MAE) computed on distances below 8 ˚ A, denoted as MAE 8 .

Cosmic

Cosmic contains images from the F435W filter collected from the Hubble Space Telescope. It aims to identify cosmic rays (corrupted pixels) in the images. Inputs are images of 1 × 256 × 256 , and outputs are pixel binary predictions (artifact vs. no-artifact). The dataset is split into 4,347 images for training, 483 for validation, and 420 for test.

ECG

The ECG task concerns predicting irregularities in electrocardiograms. ECG recordings of 9-60 seconds are sampled at 300 Hz and labeled using four classes: normal, disease, other, or noisy rhythms. We process each recording with a fixed sliding window of 1,000 ms and stride of 500 ms. This transforms 2,186 single lead recordings into more than 330,000 segments. We use 261,740 of these for training, 33,281 for validation, and 33,494 for test. Each segment has the shape 1 × 1 , 000 . The evaluation metric is the F1-score.

Satellite

The goal of the Satellite task is to classify land cover maps for geo-surveying, for one million data points across 24 categories. Each data point is a single-channel satellite time-series of shape 1 × 46 , with standard normalisation augmentation applied. The data is split with 800,000 samples for the training set, 100,000 for validation, and 100,000 for test.

Deepsea

This dataset focuses on predicting the behaviour of chromatin proteins to aid in understanding genetic diseases. Each data point is a genome sequence of 1,000-base pairs (A, C, T or G), represented as a binary matrix of shape 1000 × 4 , categorised across 36 classes of chromatin features. The training set contains 71,753 data points, with 2,490 for validation and 149,400 for testing. The evaluation metric is the area under the receiver operating characteristic (AUROC).

Finally, we also report results on the classic CIFAR10 dataset:

CIFAR10

CIFAR10 is a widely known image classification dataset with 10 classes. Each image is of size 3 × 32 × 32 . The dataset is split into 40,000 training, 10,000 validation and 10,000 testing images. Wepreprocess the images by applying random crops, horizontal flips, and normalisation.

B.3 Compute Resources

All our experiments ran on our two internal clusters with the following infrastructure:

AMDEPYC755248-Core Processor with 1000GB RAM and 8 × NVIDIA RTX A5500 with 24GB of memory

AMDEPYC745232-Core Processor with 400GB RAM and 7 × NVIDIA A100 with 40GB of memory

Each experiment used a single GPU to train each architecture. Running 1000 iterations of RE(Scratch) onthe quickest datasets (Language and Chesseract) took around 2 days, while the slowest (GeoClassing) took 4 days. We had very similar training times for RE(RN18) and RE(Mix) which ran for 500 iterations.

C Additional Results

C.1 NASBench360

Wetest einspace on the diverse NASBench360 [55] to further showcase its potential. These results can be found in Table 6. In this setting the baseline network is a WideResNet16-4 (WRN)-which is

adapted for the 1D tasks ECG, Satellite and Deepsea as in [55]. We see that our regularised evolution with the baseline as the initial seed, RE(WRN), again consistently finds architectural improvements. It matches the performance of the GAEA search strategy on the DARTS search space, achieving the same average rank. On NinaPro, it even beats the Expert architecture, specifically designed for this task and to the best of our knowledge sets a new state-of-the-art. Note that we fail to exactly reproduce the WRN baseline network performance on several tasks, so we report both our own WRN values along with those from [55].

C.2 More Baseline Architectures

Weexplore another baseline architecture, DenseNet121, and use it as the initial seed to our evolutionary search in einspace . In Table 7, we can see that the DenseNet121 on its own performs comparably to the ResNet18 model. When seeding search from the model with RE(DN121), we observe performance boosts similar to the gaps found between RN18 and RE(RN18), highlighting the general applicability of initialising search from different architectures within einspace .

C.3 Finetuning

Wepresent further results comparing our method against finetuning a model from pre-trained weights. In the experiments presented in Table 7, a finetuned EfficientNet-B0 matches or beats the ResNet18 baseline on the Unseen NAS datasets. However, RE(Mix) on einspace still often outperforms this method, e.g. on Gutenberg by 12%, although finetuning dominates on the CIFARTile vision task.

C.4 CIFAR10

To complete our evaluation on common NAS benchmarks, we also report results on the CIFAR10 dataset in Table 8.

C.5 Runtime and Parameter Count

Table 9 summarises the runtimes and parameter counts of architectures found using DrNAS, PCDARTS, and RE on hNASBench201 and einspace . Notably, the gradient-based DrNAS and PCDARTSdemonstrate significantly shorter runtimes, with DrNAS achieving search times between 10 to 25 hours and PC-DARTS between 4 to 12 hours. In contrast, the black-box evolutionary methods show substantially longer runtimes, ranging from 9 to 80 hours, as they independently train a large number of networks. The search times for einspace are comparable to those for hNASBench201, though due to the potential increased complexity in candidate architectures, they can sometimes be significantly longer. In terms of model parameters, DrNAS and PC-DARTS consistently produce architectures

with parameter counts around 4 million, hNASBench201 is consistently more parameter efficient at around 1M, while einspace finds architectures that vary significantly in size from 1M-25M, showing flexibility in adaptation to task difficulty.

C.6 Empirical Architecture Complexity

For our experiments we set the computation module probability to p ( M → C | M ) = 0 . 32 using the branching rate method described in A.4. We next report empirical results for the architecture complexities as we vary this value. In Table 10 we can see that the complexity, as measured by the count of terminals and non-terminals in the derivation trees, grows as the probability decreases.

Whensearching for an architecture on a new unknown task, the flexibility of the search space is key. During our random searching on einspace wefound that we sampled networks with parameter counts ranging from zero up to 1 billion, and from as few as two operations up to as many as 3,000. The frequency of all functions in einspace that appear in these networks can be found in Figure 7.

sequential_module

branching_module

routing_module

computation_module

clone_tensor2

clone_tensor4

clone_tensor8

group_dim2s1d

group_dim2s2d

group_dim2s3d

group_dim4s1d

group_dim4s2d

group_dim4s3d

group_dim8s1d

group_dim8s2d

group_dim8s3d

dot_product

scaled_dot_product

add_tensors

cat_tensors1d2t

cat_tensors2d2t

cat_tensors3d2t

cat_tensors1d4t

cat_tensors2d4t

cat_tensors3d4t

cat_tensors1d8t

cat_tensors2d8t

cat_tensors3d8t

identity

permute21

permute132

permute213

permute231

permute312

permute321

im2col1k1s0p

im2col1k2s0p

im2col3k1s1p

im2col3k2s1p

im2col4k4s0p

im2col8k8s0p

col2im

linear16

linear32

linear64

linear128

linear256

linear512

linear1024

norm

leakyrelu

softmax

positional_encoding

Node type

10

2

10

4

Frequency

Distribution of node types

8

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer:[Yes]

Justification: The main contribution is our search spaceeinspace -which is described in detail in the paper, and can be explored using the code we have provided. We have claimed competitive results on the Unseen NAS datasets with our space using simple search algorithms which can be verified in Table 1. We have included aspirational goals as motivation, and made it clear that we believe this work is a step towards achieving them.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have a dedicated Limitations section in our paper. Please see Section 5 for more details.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our paper does not present any theoretical proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide implementation and experimental details in Appendix B. We also submit our codebase together with the paper. The codebase includes all configuration files to reproduce our results on Unseen NAS (Tab. 1) and NASBench360 (Tab. 6). It also includes detailed instructions on how to set up the environment and datasets.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) Werecognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some

way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We submit our codebase together with the paper for reviewers to check our results. The codebase includes all configuration files to reproduce our results on Unseen NAS (Tab. 1) and NASBench360 (Tab. 6). It also includes detailed instructions on how to set up the environment and datasets. We will make our codebase publicly available on acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide a detailed description of the experimental settings in Appendix B. Wealso submit the codebase, from which readers can find all the settings.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Unfortunately, the computational expense for running multiple runs of each experiment was too large. We focused on breadth of tasks instead of repeated runs on the same task.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In Appendix B.3 we give information on computer workers, memory and time.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read the code of ethics, and confirm that the research conducted conforms with it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This is foundational research which is application agnostic.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer:[NA]

Justification: This is foundational research which is application agnostic.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer:[Yes]

Justification: This work uses existing datasets and the work corresponding to these datasets is cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For commmon datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. NewAssets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Well-documented code is provided. Please see our answer to Question 5 of the checklist.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The research does not involve crowdsourcing or human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The research does not involve crowdsourcing or human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nNeural Architecture Search (NAS) often produces incremental improvements due to limited diversity in traditional search spaces. To address this, the paper introduces \"einspace,\" a versatile search space built from probabilistic context-free grammar (CFG). Einspace supports a wide range of architectures and operations, enabling the modelling of convolutions and attention mechanisms. Experiments on Unseen NAS datasets show that einspace can discover novel architectures from scratch and with grammars containing different foundation model architectures.\n\nSTRENGTHS:\n1. The paper is written very clearly and examples of CFGs aid understanding of the paper. \n2. The authors release code which improves the reproducibility of the work and potential future work \n3. While I do think that this paper opens up new opportunities and challenges for scaling NAS from scratch to larger spaces and datasets, I have questions (check questions) for the authors to improve the quality of their evaluations and make a stronger case for the practical applicability of einspace.\n\nWEAKNESSES:\n1. **Contribution**: Currently I fail to see the main contribution of the paper to the area to \"searching architectures from scratch\" [1]. I find the representation  properties of the search space to be same as [1] (except the use of probabilistic CFGs). Moreover, the search schemes proposed random and evolutionary search are not very sample efficient in such large spaces compared to [1], which uses bayesian optimization with graph kernels. \n2. **Evaluation**: The paper does not compare to natural baselines like [1]. Moreover I find the evaluation to be quite limited in terms of the datasets evaluated and the base models finetuned/evaluated on. \n\n[1] Schrodi, S., Stoll, D., Ru, B., Sukthanker, R.S., Brox, T. and Hutter, F., 2022. Towards discovering neural architectures from scratch.\n\nQUESTIONS:\nQuestions:\n\n1, The paper claims that it is the first one to represent foundation model architectures with a NAS search space. However [1] already represented a language model using CFGs. Also [1] can potentially present any architecture (ViT, MLPMixer). Is my understandng correct that this paper introduces a PCFG on [1] to  limit the search space size? I am also currently lacking a CFG which is a union of multiple foundation model architectures in a single grammar. Could the authors provide an example for this?\n\n2. The paper uses random search and evo search, both of which are sample inefficient and do not scale well with the size of the dataset and search space. How do more sample efficient bayesian optimization algorithms perform here?\n\n3. Currently the standard for small unseen datasets is finetuning a pretrained model on the smaller downstream task. Could the authors should ideally compare to for eg: simply finetuning a pretrained vision transformer or efficientnet on the unseen dataset?\n\n4. Evaluation on larger datasets is missing. Architectures which are able to exploit dataset patterns change depending on the scale of datasets available (eg: Vision transformers [2]). Are the architectures discovered significantly different from the existing ones even when the dataset is scaled up. How efficient is the search on this scale?\n\n5. Could you present parameter and FLOPs counts in Table 1?\n\n6. How is the branching rate hyperparameter set for a newer search space?\n\n\n\n\n\n[1] Schrodi, S., Stoll, D., Ru, B., Sukthanker, R.S., Brox, T. and Hutter, F., 2022. Towards discovering neural architectures from scratch.\n\n[2] Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C. and Dosovitskiy, A., 2021. Do vision transformers see like convolutional neural networks?. Advances in neural information processing systems, 34, pp.12116-12128.\n\nLIMITATIONS:\nI do think that the paper opens up newer avenues of research in neural architecture search, however, given the limited evaluation I do have concerns about the applicability of the method. Check weakness and questions for details on the limitations. I am happy to raise my score if each of my concerns are appropriately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new type of neural architecture search (NAS) search space that goes beyond standard, small NAS search spaces. Most popular search spaces in NAS are quite small, and include architectures that include known motifs because they are built around standard architectures. The authors point out that this is why NAS has failed to produce fundamentally new architectures, and instead, that most advancements in neural network architectures are from expert driven hand-design. Their search space, einspace, is a probabilistic context-free grammar (CFG) of fundamental operations.\n\nSTRENGTHS:\n- The paper is well-written, well-motivated, and easy to follow. \n- The paper is clear in that its innovation is related to search space design, with many suggestions for possible search algorithms. The search space itself is quite impressive, and includes both convolutions as well as self-attention operations (built-up from more fundamental building blocks). \n- The results of search seem impressive -- while einspace does not always yield stronger architectures compared to other methods or hand-designed baselines, it does sometimes yield significantly better architectures. Furthermore, the comparison to the random search baseline is appreciated. \n- The search space supports initialization using existing architectures -- this is important for setting priors for problems in which good architectures are already known.\n\nWEAKNESSES:\n- The resulting architectures in Table 1 do not always outperform SOTA architectures, but when they do, the improvement can be significant. On the other hand, there does not seem to be a comparison of the computational costs involved -- this would be helpful to include, especially as the search space is quite large. \n- The authors include results on NAS-Bench-360, which is great, however the results are incomplete and it is unclear why the authors only evaluate on 5 out of the 10 tasks. This seems particularly important as the motivation of einspace is similar to the line of work related to NAS-Bench-360. \n- In order to make search tractable, it seems like many assumptions are made on the priors on the search space. This makes sense, however, it would be interesting to explore variations on these choices more in-depth. \n\nIf the authors adequately address these concerns, I will gladly raise my score.\n\nQUESTIONS:\n- Out of curiosity, what do the authors view as the key reason for einspace not being able to express RNNs and SSMs? This is a valid design decision, but it also seems as though it should be fairly straightforward to design a version of einspace that supports recurrence (I could be mistaken about this). \n- Why did the authors choose the 5 NAS-Bench-360 tasks that were used in their evaluation? Does the method work on the other tasks?\n\nLIMITATIONS:\nThe authors clearly state the limitations, and even discuss avenues for future work. This includes extending the search space to handle recurrent architectures and state space models.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new search space named einspace based on a parameterized probabilistic context-free grammar for neural architecture search. In contrast to conventional search space composed of high-level operations and modules such as recurrent layer, convolutional layer, and activation layer, einspace consists of more fundamental operations, such as clone, summation, permutation, and activation. This einspace can be defined by a context-free grammar and can represent the neural architecture as the derivation tree. The experiments on Unseen NAS and NAS-Bench-360 demonstrate the advantage of this search space.\n\nSTRENGTHS:\n1. The paper is well-written and is easy to follow.\n2. The idea of fundamental operations and context-free grammar for search space is great and interesting.\n3. This search space has the advantage of designing new architectures based on existing SOTA architectures such as ResNet, Wide-ResNet, ViT, and MLP-Mixer.\n\nWEAKNESSES:\nDespite the strengths of this paper, I still have some concerns that prevent me from confidently recommending acceptance:\n1. The first problem is about the novelty. Since einspace is not the first search space based on CFG, and the fundamental operations of einspace are common in the existing search space, the novelty of einspace may not reach my expectations. Although the fundamental operations are different from and more expressive than the high-level operations and rigid structures of existing search space, these fundamental operations are also commonly used in existing search space. Furthermore, it is complicated to represent convolution and skip connection using these operations and it is impossible to represent recurrent computation, as the authors discussed in the Limitation section.\n2. The second problem is about the tradeoff between diversity and time complexity of the search space. It would be better to analyze the time complexity or compare the time consumption between einspace and others, such as the space of DARTS. \n3. From the experimental results, the architectures searched in einspace do not seem to be that competitive compared to other methods, as shown in Table 1. I think the results are also influenced by the search strategy. Is it possible to apply RL or differentiable search strategies to this search space? If possible, the author can give a table listing which search strategies are suitable for einspace and which search strategies are better.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nThe authors have discussed the limitations in their paper. Besides, there are no potential negative societal impacts of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript presents \"einspace,\" a novel neural architecture search (NAS) space based on a parameterized probabilistic context-free grammar (CFG). The authors aim to address the limitations of current NAS methods by proposing a highly expressive search space that supports diverse network operations and architectures of varying sizes and complexities.\n\nSTRENGTHS:\n1. The paper introduces a unique NAS search space, \"einspace,\" which is designed to be highly expressive, accommodating a wide range of architectures, including those not traditionally found in NAS literature.\n\n2. The work contributes to the ongoing discussion on the role of search space expressivity and strategic search initialization in NAS, potentially paving the way for future research in this direction.\n\nWEAKNESSES:\n1. While the authors claim to introduce a new search space, the manuscript's Method Section appears to describe a set of rules that break down operators into smaller elements, which could be misinterpreted as a mere decomposition rather than a novel search space construct. This raises questions about the actual size and scope of the proposed search space, which would benefit from further clarification.\n\n2. Section 3.7 is not clearly articulated, and the authors are encouraged to provide a simplified explanation of its main content to aid reader comprehension.\n\n3. The primary goal of the proposed method is to design a highly expressive yet constrained search space. To substantiate this claim, it would be beneficial to conduct searches and validating on larger-scale datasets, such as ImageNet, which could more effectively demonstrate the superiority of the proposed search space. The current experiments on smaller datasets may not fully showcase the advantages of the search space.\n\n4. The experimental settings are somewhat unclear. It appears that the authors search within the proposed space and then validate the discovered network structures on other datasets, such as those from NAS-Bench-360. However, NAS-Bench-360 imposes constraints on the design space that may not be compatible with the structures proposed by the authors. Further clarification on how the proposed space's network structures are adapted or validated on NAS-Bench-360 is needed to ensure the experiments are methodologically sound.\n\nQUESTIONS:\nNA\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces einspace, a hierarchical space of neural architectures based on parametric probabilistic context free grammar, which is expressive enough to accommodate various state-of-the-art architectures including ResNets and Transformers.\n\nThe Authors further perform Regularized Evolution (RE) search over einspace either searching from scratch or seeding the initial population with state-of-the-art architectures. In particular when seeded with ResNet18, RE generates novel architectures that significantly outperform ResNet18 on multiple datasets from Unseen NAS. Furthermore, RE both seeded with ResNet18 and Mix (a mixture of SOTA architectures) significantly outperform RE from scratch as well as random sampling and random search.\n\nSTRENGTHS:\nThis study extends previous research on hierarchical architecture spaces by introducing a more expressive framework capable of accommodating diverse structures including convolutional networks and transformers. \n\nNovelties include imposing minimal priors which facilitate successful search within a highly expressive architecture space. Additionally, grammar rules are equipped with parameters to ensure the generation of valid architectures through the combination of various components. Moreover, the complexity / depth of architectures are regulated by tuning the probabilities assigned to production rules.\n\nThe authors demonstrate through a number of experiments involving ResNet and WideResNet architectures that network performance can be significantly enhanced by utilizing einspace in conjunction with RE, when seeding the initial population with these SOTA architectures.\n\nWEAKNESSES:\nThe experiments are currently limited. While the paper does not introduce a search strategy tailored to this search space, it is crucial to emphasize experiments demonstrating the potential for RE, possibly seeded with SOTA networks, to enhance model performance. \n\nThe current evaluations focus on ResNet18 on the Unseen NAS datasets as well as WRN on datasets from NASBench360. Extending this analysis to include a broader range of models, e.g. those listed in Table.1 of Unseen NAS, such as AlexNet and DenseNet, would provide useful insights on the effectiveness of einspace. \n\nMoreover, it would be valuable to explore the application of RE on einspace to improve performance on widely used datasets like CIFAR10, e.g with ResNet and in particular ViT,  given that the paper highlights the capability of einspace to support transformer architectures as an advantage.\n\nQUESTIONS:\nHow does einspace compare with other hierarchical search spaces in the literature [1] and [2] in terms of performance? To support the advantages of einspace, including expressivity at a reasonable search cost, it would be beneficial to conduct comparative evaluations, at least with RE, across a number of tasks.\n\nDoes it make sense to apply black-box search methods other than RS, RE to einspace, for example the bayesian optimization-based methods BOHNAS and NASBOWL used in [2]?\n\nApproximate search times for RE on datasets of Table.1 are reported in appendix B.3. How do these search times compare with those of other NAS methods listed in Table.1? A similar comparison would also be valuable for the experiments in Table 5.\n\n[1] Hierarchical Representations For Efficient Architecture Search (Simonyan et.al 2018)\n[2] Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars (Schrodi et.al 2023)\n\nLIMITATIONS:\nThe limitations are adequately discussed in the final section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces einspace, a search space that is designed to hierarchically encode architectures using probabilistic context-free grammars (PCFG). It can encode various architectural components, such as convolutions, attention mechanisms, etc. The authors demonstrate the efficacy of simple blackbox optimizers in einspace to discover architectures that perform competitively on various tasks and datasets.\n\nSTRENGTHS:\nIn general the motivation to move beyond the conventional NAS spaces is valid and really important in my opinion. The paper is also very well-written, with simple examples followed by a more generic definition of the search space and interesting application to various tasks and datasets.\nDiscovering novel architectures is a very challenging problem for the NAS community and as far as I know, this has not been achieved yet. Having a complex and versatile search space is the first step towards this goal.\n\nSome other positive aspects of this submission:\n\n- Interesting experiments showing the complexity of the search space and the need of guided search methods (e.g. evolutionary strategies as the authors show), instead of random search, which on previous spaces (e.g. the DARTS one) was shown to perform already well.\n- Available code that fosters reproducibility and enables easier future research on this topic.\n- Extending the prior work of [1] to probabilistic CFG. This enables (1) easier expert prior definitions on the search space, (2) a broader range of algorithms applied on einspace, that can as well incorporate uncertainty estimates of their choices inside the search itself.\n\nWEAKNESSES:\nDespite the vast number of architectures that einspace includes, the major problem is how to search on these spaces in an efficient way. The prior work of [1] used BO with a hierarchical graph kernel (a novelty aspect of that paper), however that was still expensive, especially when moving to image classification tasks. I think defining search spaces that are very complex is very useful task, however, the NAS problem is not only solved by defining the search space alone. The search algorithm is a major component of the whole pipeline, and using blackbox methods to search in such search spaces will be computationally demanding and will still require company scale computation.\n\nBelow I list my major concerns about this submission:\n\n- *Novelty*: I think the paper has some novel aspects compared to [1], e.g. the fact that can encode both attention-mechanisms and convolution, or other operators, or the probabilistic extension of the CFG. However, as the authors mention in the limitation section, it would be great if accompanying the search space, there would be a new proposed search method that can efficiently search on this space by exploiting the PCFG.\n- *Limitations of the search space*: As the authors mention, there are various important architectures that cannot be encoded in einspace.\n- *Claims*: I think the claim (for instance in the abstract) that you find \"novel architectures\" can mean many things. Correct me if I am wrong, but in my opinion einspace still encodes most of the known architectures, and searching on it will only re-discover them or improve on top of those architectures, but it won't find a completely novel architectural component, as for instance ResNets introduced the residual connection back then. \n- *Limited experimental evaluation and results*: The empirical evaluations are not enough in my opinion. The used tasks are simple and one function evaluation is cheap enough to allow blackbox methods to run there. Evaluating on more expensive tasks would require the development of more efficient search strategies that are tailored to einspace. Moreover the results shown in Table 1 are not that impressive considering RE is typically a very strong baseline, so I would have expected it to outperform all other methods.\n\nQUESTIONS:\n- Could you please discuss further in more details on what are the main advantages of using CFG instead of graph-based or other encodings used in NAS [2]?\n\n\n-- References --\n\n[1] https://proceedings.neurips.cc/paper_files/paper/2023/file/4869f3f967dfe954439408dd92c50ee1-Paper-Conference.pdf\n\n[2] https://proceedings.neurips.cc/paper/2020/file/ea4eb49329550caaa1d2044105223721-Paper.pdf\n\nLIMITATIONS:\nThe authors have addressed the limitations adequately.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-28"}