STAR: Synthesis of Tailored Architectures
Authors: Armin W Thomas
Rom Parnichkun
Alexander Amini
Stefano Massaroli
Michael Poli
Venue: ICLR 2025 Oral (2025)
Keywords: alternative architectures
deep signal processing
language models
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a new approach for automatic model architecture optimization (STAR), which combines a novel search space based on the theory of linear input-varying systems with a hierarchical numerical encoding into architecture genomes.

Abstract: Iterative improvement of model architectures is fundamental to deep learning: Transformers first enabled scaling, and recent advances in model hybridization have pushed the quality-efficiency frontier. However, optimizing architectures remains challenging and expensive, with a variety of automated or manual approaches that fall short, due to limited progress in the design of search spaces and due to the simplicity of resulting patterns and heuristics. In this work, we propose a new approach for the synthesis of tailored architectures (STAR). Our approach combines a novel search space based on the theory of linear input-varying systems, supporting a hierarchical numerical encoding into architecture genomes. STAR genomes are automatically refined and recombined with gradient-free, evolutionary algorithms to optimize for multiple model quality and efficiency metrics. Using STAR, we optimize large populations of new architectures, leveraging diverse computational units and interconnection patterns, improving over highly-optimized Transformers and striped hybrid models on the frontier of quality, parameter size, and inference cache for autoregressive language modeling.

Published as a conference paper at ICLR 2025
STAR: SYNTHESIS OF TAILORED ARCHITECTURES
Armin Thomas', Rom Parnichkun Alexander Amini', Stefano Massaroli and Michael Poli
'iiuidd AI The University of Tokyo RIKENN "Stanford University
farmin, sommin..ww.geccaaddeeseemmmm.nce @ liquid.ai
ABSTRACT
Iterative improvement of model architectures fundamental deep learning: Transformers first enabled scaling, and recent advances in model hybridization have pushed the quality--fficieney frontier. However, optimizing architectures re- mains challenging and expensive. Current automated or manual approaches fall short, largely due to limited progress in the design of search spuces and due to the simplicity of resulting pattemns and heuristics. In this work, propose proach for the synthesis of tailored architectures (STAR). Our approach combines novel search space based the theory of linear input-varying systems, support- ing a hierarchical numerical encoding into architecture genomes. STAR genomes are automatically refined and recombined with gradient-free, evolutionary algo- rithms to optimize for multiple model quality and efficiency metrics. Using STAR, we optimize large populations of new architectures, leveraging diverse computa- tional units and interconnection patterns, improving over highly-optimizer Trans- formers and striped hybrid models on the frontier of quality, parameter size, and inference cache for autoregressive language modeling.
ap-
INTRODUCTION
Most domains of applications for Al have seen a gradual convergence towards similar model ar- chitecture designs, based on stacking multi-head attention and gated linear units (Transformers) (Vaswani et al., 2017: Shazeer, 2020; Brown, 2020) or combinations other basic computational units grounded in signal processing, such as recurrences convolutions (Martin & Cundy, 2017; Romero et al., 2021; Gu et al., 2022; Smith et al., 2023; Peng et al., 2023; Poli et al., 2023; Massaroli et al., 2023; Yang et al., 2024b).
Broadly, there are two prominent paths to improve model architectures: automated and manual. Automated design, leveraging optimization (e.g. evolutionary methods) within a predefined search space, has seen success in highly- targeted domains, such the refinement of convolutional neural networks for snoorce-constrainee applications (Pham et al., 2018; Liu et al., 2018; Howard et al., 2019; Li et al., 2021; Tan Le, 2021). Automated methods have also been utilized to identify candidate improvements to standard computational primitives (So et al., 2021) 1), e.g., depthwise con- volutions projections. Nevertheless, date, automated methods have fallen short of providing a unified framework that provides significant improvements in quality and efficiincy across domains and objectives over models using standard generalizable recipes. The homogeneity of architectures applied at scale during the Transformer era highlights this shortcoming.
The main challenge for automated methods lies in curating search space for computational units and architectures that is both (a) well-conditioned i.c., populations of model candidates can be trained effectively, without numerical instability unpredictable degradation in performance, and (b) comprehensive i.e., the design space includes candidates with significantly different properties from existing variants, expanding the range of potential improvements that can be identified.
Despite a wealth of automated approaches for the search and refinement of computational units and composition strategies (White et al., 2023), the current generation has been obtained mostly through an iterative manual process, guided by intuition and tuning on representative smaller-scale tasks via e.g., synthetics and scaling laws (Hoffmann et al., 2022; Arora al., 2023; Bi et al., 2024). Manual design has led to a variety of results, most notably in the introduction or improvement of computational units (Poli et al., 2023; Massaroli al., 2023; Yang et al., 2024a; Arora al., 2024),

---

Published as a conference paper at ICLR 2025
167
Optimizing: Quality & Size
Transformer+
StripodHlybrdd
STAR Synthesized
Syatboeised
Transformet
Params (Nom Eimbel)
8.0
Stripee Mannha
7.5
Generation
7.0
6.5
PPL
Population
2. Recombine
3. Mutate
Properties
Performance
Cachi:
SBM
SKMB
PPL Traior
Paaramm
122M
114
CPU
32.43
21.56mr
Figure 1.1: (Top Left]: Population architectures undergoing iterative STAR evolution to minimize number of parameters and maximize quality. (Top Right:] Baseline Transformer++, hybrid model, and representative architecture found via STAR. [Bottom]: ST AR evolution optimizes architectures using principles of evolution- ary optimization,, including assessment, recombination, and mutation.
modifications targeting smaller inference cache (Shazeer, 2019; Brandon et al., 2024), and the dis- covery of simple interconnection strategies e.g., e.g.. striped hybridization (Brown, 2020; Fu et al., 2022; Poli et al., 2024; Lieber et al., 2024), weight sharing (Liu al., 2024), and others (Liu al., 2022; Brandon et al., 2024). Yet, manual design is limited to finding relatively basic design patterns, com- pared to the total diversity of possible patterns, and requires significant investment in resources, expertise and time.
Given the wide range of possible applications of current Al systems, enabling systematic and auto- matic optimization of model architectures from the multitude of existing computational units is key to meeting the various demands these applications pose, in terms of efficiency (e.g., model size, in- ference cache size, memory footprint) and quality (e.g., perplexity, downstream benchmarks), and prerequisite on the path further, consistent improvements on the quality- Pareto frontier.
In this work, seek address limitations of existing automated architecture optimization methods by introducing an approach for the synthesis of tailored architectures (STAR). STAR based on the combination of novel hierarchical search space of computational units and their composition, as well as a numerical encoding compatible with evolutionary methods.
Hierarchical search spaces The design space for STAR is grounded in the theory of linear input- varying systems (LIVs), providing novel framework to design building blocks in architectures. LIVs generalize most computational units used in deep learning, such as attention variants, linear' recurrences, convolutions, and other structured operators. Notably, acterize model architectures at three hierarchical levels of resolution:
framework allows to char- (a) featurization, defining how the linear computation within the LIV is modulated by the input context; (b) operator structure, defining the token and channel mixing structure the LIV; and (c) backbone, defining the compo- sition structure between LIVs. contrast to previous search spaces (Pham al., 2018: Liu al., 2018; Howard et al., 2019; Li et al., 2021; Roberts et al., 2024), we show how the LIV search space both comprehensive and well-conditioned, as most sampled candidates train without instabilities.
Leveraging the modularity LIVs, we taxonomize the space and devise hierarchical numerical representation of model backbone, which we refer to the STAR genome. Due to its struc-
Here, by linear refer the linearity of the transition.
2

---

Published as a conference paper at ICLR 2025
ture, STAR genomes be optimized different levels of the hierarchy. We show how backbone genomes defining ordering and interconnection between LIVs can be automatically refined with evolutionary algorithms relying few key principles, such as evaluation, recombination, and mu- tation.
Exploring the efficiency-quality frontier We evaluate STAR autoregressive language mod- eling, domain historically dominated by Transformers and architecture improvements found via manual search. In particular, we optimize architectures for various combinations of metrics simul- taneously: quality (perplexity during pretraining). quality and size, and quality and inference cache size (KV cache and fixed state cache). When optimizing for quality and size, 7/8 evaluated STAR- evolved architectures improve over Transformer++ and striped hybrids of recurrences and attention across downstream evaluation benchmarks (Gao al., 2024), with reduction of up 13% in parameter counts. Similarly, optimizing for quality and cache size, 7/8 evaluated STAR STAR-evolved architectures achieve up to 37% smaller eache sizes than striped hybrids, and 90% smaller than Transformers, while performing at least well in quality. We also show that 125M-parameter chitectures optimized for quality and cache by STAR can scale to IB parameters and perform on par with parameter-matcher Transformer++ and striped hybrid architectures, while maintaining the same advantages in cache size reductions. When optimizing solely for quality, all evaluated STAR- evolved architectures outperform standard hybrids downstream benchmarks, achieving improve- ments twice as large as those of hybrids over Transformers. Finally, we showcase how STAR can be used to identify recurring architecture motifs emerging during evolution, driving the observed performance gains.
FOUNDATIONS OF THE SEARCH SPACE
We detail how the framework behind our search space is grounded in the theory of linear systems.
Linear input-varying systems The class of data structures under consideration are sequences of vectors {ro, ,1,..., where each element z; is referred as token. Each token z; is real- The individual components 2; of each valued vector IR represented as x; (x.) x.f.x!.... .........._.1
token are called channels.
The attention operator (Bahdanau et al., 2014; Vaswani al., 2017), provides valuable starting point to define a search space for model architectures, as it defines a prototype of what we call linear iaput-varying (LIV) operators. Attention, in its common formulation, can be expressed linear operator applied to the input, with the operator's 's action determined by the input itself:
yi Σ
(gi, k,) = (vp(x)), (x:))
BE(d JE(8]
attent Lon operator
where a :R R. is nonlinear function and V E Rdxd, The intermediate quantities g, h obtained through functions s,, v R2 R the input tokens and used to construct the linear operator T, are referred to as feature groups.
We extend this idea include broader family of L.VS, expressed in their most general form
v Σ Σ Ig (x)x}
JE(A)
The LIV framework decouples the (potentially) nonlinear and linear computation required to mate- rialize the operator T(x:) and apply obtain the outputs, y Tx. LIVs include and generalize diverse array of computational units commonly used in model architectures, whose class is defined by the structure of the operator: attention, convolutions, linear recurrences, and various forms of other structured layers:
Tú oCC,B,)
dense low-rank semi-separable scaled Toeplitz
attention [4; 45]
Tú C.B;
linear attention [21]
Tı = C..--1....
A...
linear recurrence [28; 41; 17; 49]
Ти = C,Kr,BB
gated convolution [12: 33]
(o(C) i=
Ty 0
diagonal
memoryless system [39]
otherwise

---

Published as a conference paper at ICLR 2025
where the structure" of the operator T induces a decomposition into feature groups, analogously to the attention example. To highlight the parallels between different LIVs and attention, we have adopted shared notation for the feature groups. Differentiating between LIV systems are two key factors, operator structure featurization
Featurization refers to the process with which feature groups are obtained, either via direct parametrization, reparametrization", or via parametric transformations of inputs as is the case in attention i... linear projections.
Structure We taxonomize the linear operators of LIVs by decoupling the analysis of token mixing and channel-mixing structures. To define structure, we look at two different slices of the operator:
Taß E R éxÈ highlights the token-mixing structure for each tuple of input and output channels, i.c the linear contribution of the ,th input channel x IR'' to the ath out- put channel y E R.. Loosely speaking, the choice of token-mixing structure determines the class of matrix multiplication algorithms that be utilized to apply the operator to the input (e.g. Fast Fourier Transform based convolution Toeplitz, or parallel prefix scan if semi-separable (Dewilde & Van der Veen, 1998)).
Try E Rdxd conversely reveals the channel-mixing structure of T, i.c. the (linear) contri- bution of the jth input token žj of the input sequence to the ith output token yi. In practice, the most common choice of channel-mixing structure is by far the diagonal one, as used in attention and most variants of linear recurrences". Diagonal Tay blocks allow maximum parallelization of the LIV operators as the linear computation reduces to d independent matrix multiplications.
Note that we use the same logic define the structure present in the featurizer itself. The structure of the featurizer and operator need not be the same.
Composition An architecture backbone can be decomposed into a set of LIVs with different com- position rules. Beyond sequential stacking of LIVs, as is common in standard deep architectures, we introduce other composition rules realized via the featurization: featurizer sharing, where the same featurizer weights are shared between different LIVs of backbone, and feature group sharing, where different LIVs share the same feature groups
Let T, S denote the operators at two different depths m, n (m < n) of the composition, respectively. If, for example, both LIV operators are chosen with similar low-rank (linear attention) structure Ti = C,B,, Sij- E,F; and the dimensions of the feature groups are compatible, i.e. C,, Es E Rdx1 and Bi, F3 E Te we can apply both featurizer and feature group sharing techniques:
* If the parametric featurizer of C, and E; has the same form C; = (p( :) E; ()
we can share the same set of parameters 0 between them:
Ti = (( (m), 0)B; Bj
Sij = (nx(());)))
Fenturizer sharing
where ((m)) (n) denote the input to the mith and nth LIV system, respectively.
Similarly, we can simply re-use one of the feature groups of the moth LIV system in the wihh one, e.g. By:
Tij C,B
Sij B,
feature group sharing
A prominent example of feature group sharing is the sharing of key-value caches between attention operators (Brandon al., 2024). Beyond featurizer interconnections, we explore other strategies of operator composition in Appendix A.7.
Note that in this list we refer to the sequence-mixing structure i.e., the structure of slices T of the operator T. We define the structure via entries 1:38 for convenience.
"Sometimes referred as implicit parametrization (Mescheder al., 2019; Romero al., 2021; Sitzmann al., 2020).
"This is brought to extreme by multi-headed architectures that only present number-of-heads distinet
values on the diagonal.
4

---

Published as a conference paper at ICLR 2025
Jesturizer sharing strategy
Jeature group sharing straley
Backbone:
1122021212233311222221222
Aelrro Fead groegs
LIV system
fecturiaer sharing
feature group sharing
Operator:
12121
y == Tx
Featurizer:
T(x; 0)
Hicrarchical structure of the STAR genome. Each sequence lower levels is summarized into a single value at higher levels, enabling its treatment as discrete variable. We leverage this property extensively when optimizing backbones directly.
Figure 3.1:
DESCRIBING OPERATORS AND BACKBONES WITH STAR GENOMES
The new design space of LIVs and their compositions serves as the foundation for the automated synthesis of lailored architectures (STAR). In the following, we will describe how we map the three hierarchical levels of the LIV representation suitable for optimization: the STAR genome. Each level of the hicrarchy can be sum- marized into single integer, yielding a numerical representation that can be optimized at different levels of granularity (see Fig. 3.1). this work, we focus on backbone optimization but are report- ing the full description of the genome for completeness. See Appendix A.6 for full description of all genome values considered this work.
structure, and composition- a numerical
3.1 BACKBONE GENOME
We begin by describing the highest abstraction level of the STAR genome, the backbone genome, which defines the composition LIV's in the backbone. We recall that under the LIV framework, LIVs can be connected with featurizer and feature group sharing, as described in Section 2. Specifi- cally, the backbone genome represents set of integer-values valued sequences of length five, one for each LIV of the backbone. Each of these 5-number segments defines the following properties of the LIV:
1. LIV class: integer summary of lower levels of the STAR genome, i.e., operator and featurizer genomes (see Section 3.2).
Featurizer sharing: determines the weight-sharing structure between featurizers of LIVs at different depth the backbone. LIVs with the same value at this position share featurizer weights, as defined by the featurization sharing strategy.
3. Featurization sharing strategy: defines how featurizer sharing is implemented for the LIV class. Featurizer weights can be shared partially, for example, only those responsible for com- puting (xx), in contrast to also sharing weights that compute C'xx). We explore all combinations of sharing strategies based on the number of feature groups of the LIV class.
4. Feature group sharing: LIVs with the same index share feature groups directly, instead of featurizer weights, for example, by using the exact same Bxx) and C(x).
5. Feature group sharing strategy: describes which feature groups, of all available feature groups of the LIV class, are shared.
These 5-number segments are then repeated in the order at which the encoded operators occur in the backbone (Figure 3) Note that outside of the compositions defined by backbone genome, LIV' are sequentially stacked using pre-norm residuals i.., y T(norm (z) (no))()))))))))) z.
32112 is the backbone a genome with 4 LIVs. The first and third LIV belong to the same class "2", and are part of the same featurizer sharing group "1" thus sharing featurizer weights according to the indexed featurizer sharing strategy "2" (e.g.. only sharing the weights responsible for the first feature group). The second and fourth LIV belong to class "3", they do not share any featurizer weights (they have different featurizer sharing indices, "1" and "2"), and instead share feature groups directly, with strategy "2" (sharing all groups).
Example:
5

---

Published as a conference paper at ICLR 2025
1 Initial Population
ii) Assess
iii) Pair
iv) Recombine
v) Mutate
vi) Repair
Cache:
PPL Train:
1032043010
= 80IMB
11.1
1032011022
032011022
403201 1022
2010320110 = 632MB
12.2
2010320501
201031 0301
201031 0001
4050320501
1122MB
9.9
4050320110
40203201 22
4010320112
0102011022
987MB
10.5
0102043010
1102033001
1102033001
Figure 4.1: Fundamental operations of STAR evolution (akin to other evolutionary optimization algorithms).
3.2 OPERATOR AND FEATURIZER GENOMES
In the backbone genome, LIVs are summarized into single number, indicating the specific fea- turizer and structure of the LIV. Unrolling this encoding reveals an additional level, the operator genome, which identifies a specific LIV in 5 numbers: (1) fcaturization, indicating the specific fea- turizer class; (2) linear voken-mixing structure of the LIV (T*9); (3) possible structured sparsity masks (e.g.. banded) for token-mixing (4) any nonlincarity applied to the token-mixing structure; (5) the LIV's channel-mixing structure (Tiy)-
The featurizer class can be similarly unrolled. In STAR, each featurizer genome is a sequence defin- ing for each of the feature groups: token and channel mixing structure (akin the operator genome); expansion factor, defining the ratio of the feature group channel dimension over the input channel dimension, encoded as one number; repeat factor, defining how many times the feature groups are replicated across the channel dimension. encoded as one number. For detailed description of all genome entries, and the respective values considered in this work, see Appendix A.6.
SYNTHESIZING ARCHITECTURES BY EVOLVING GENOMES
We have devised the STAR genome hierarchical numerical representation that encodes a specific LIV backbone, suitable for gradient-free frree optimization. In the following, will outline how STAR genome can be optimized via evolutionary methods (Beyer & Schwefel, 2002); a process subsequently referred to as STAR evolution. To allow for the application of evolutionary optimization methods to the STAR genome, we adapted methods commonly used to iteratively evolve an initial population of genomes.
4.1 KEY STEPS OF STAR EVOLUTION
Assessment STAR evolution begins by evaluating the quality of each genome in the initial popula- tion. This involves realizing the model encoded in each genome and scoring it against the objective functions of interest, either by training and assessing its performance through static analysis for efficiency objectives, such as the total number of trainable parameters in the model. Notably, STAR evolution can incorporate multiple and diverse objectives.
Pairing After assessing the quality of each genome, STAR evolution selects parent genomes for generating the next generation of offspring through tournament selection. Parents are chosen by randomly selecting set of & genomes from the population and then picking the one with the highest typiilyyyy based on criteria such as predictive accuracy or the lowest parameter count.".
Recombination Next, STAR evolution generates new candidate solutions by applying k-point crossover to the selected parent genomes. Here, genetic material from two parents is exchanged between I randomly chosen points, resulting in offspring that inherit traits from both parents. All random sampling in STAR is performed with a uniform probability across all valid options.
Mutation Finally, STAR evolution introduces random mutations to the offspring. These mutations essential for maintaining diversity in the population and promoting exploration of the search space. In STAR evolution, random mutations are implemented as alterations to the numbers in a genome, where values randomly replaced by others from predefined set of possible choices. As
As we will explore later, STAR evolution also takes into account solution diversity in the selection process maintain variety within the population (see Appendix A.4)

---

Published as a conference paper at ICLR 2025
discussed earlier (Sec. 3), these choices vary depending the genome position. To ensure that all genomes encode models capable of being trained stably and showing smooth quality improvements, STAR enforces several constraints on these random mutations, as outlined below (Sec. 4.2).
4.2 GUIDING EVOLUTION WITH HIBRARCHICAL MUTATIONS
To ensure compatibility with evolutionary optimization, the STAR genome must remain robust to random edits such as recombination, mutation, or initialization. The backbone genome is composed of 5-number segments. When mutating the first entry (LIV class), mutation is restricted to valid LIV classes. For the second and fourth entries (defining featurizer and feature group sharing), only LIVs within the same class can connect. The third and fifth entries mutated by randomly sampling valid sharing strategies for the corresponding LIV class. If mutations or recombinations result in invalid configurations, such as incompatible sharing strategies, these are detected and repaired by either removing the invalid connection for entries 2 and 4 of the operator genome, or re-sampling the respective genome value from the set of valid choices for entries 1, 3, and 5.
Controlled improvements Combining the LIV search space, genome encoding, and guidelines for mutation and recombina- tion, leads to stable training runs for most candidates obtained during the course of STAR evolution, as shown in Figure 4.2.
Optimizing: Quality
20
15
5 EXPERIMENTS
6
Experimental setup The goal of our experiments is to test whether STAR is suitable for synthesizing architectures tailored to diverse objectives, such as predictive quality and efficiency. If not noted otherwise, STAR evolutions presented are performed at model scale, where backbones contain 24 LIVs at a width of 768 dimensions, with populations of 16 genomes that are evolved for 18 generations. During each STAR evolution, we keep the depth and width of the backbone fixed. Experiments are performed in autoregressive language modeling 4096 token sequences from the RedPajama dataset (Weber et al., 2024).
5
5
2000 4000
Training Steps
Figure 4.2: Training perplexity for all runs during STAR evolution of a population.
Training details During STAR evolution, models are trained from scratch for 1.3B tokens using AdamW (Loshchilov et al., 2017) with a peak learning rate of 0.0008, batch size of 0.25M tokens, and a cosine learning rate schedule with a 130M-token linear warmup. The resulting synthesized backbones are evaluated by training them from scratch for 5B tokens under the same setup but with an extended warmup of 400M tokens. Additionally, we train select 1 IB-parameter models (48 LIVs at width of 2048) for 40B tokens, increasing the batch size to 0.75M tokens and the warmup to 2.6B tokens. See Appendix A.2 for details.
Evaluation We use two-stage evaluation process. During STAR evolution, performance metrics are computed on a 500M-token evaluation set from RedPajama (Weber et al., 2024) Post-evolution, we select the 8 models with the lowest perplexity among those with lower parameter counts (for qual- ity and quality-size optimizations) or smaller cache size (quality- models. These models are trained further and evaluated on downstream tasks: HellaSwag (Zellers et al., 2019), ARC-e (Clark et al., 2018), Winogrande (Sakaguchi et al., 2019), PiQA (Bisk et al., 2020), and SeiQ (Welbl et al., 2017); and ARC-c (Clark et al., 2018) for IB- 1B-parameter models.
optimization) than baseline
Option pool To improve initialization during STAR evolution, we incorporate genomes of com- mon backbone types (see Sec. 4). Unless stated otherwise, initial populations include simple hybrid backbones without special interconnections, combining memoryless LIVs (e.g.. SwiGLU (Shazeer, 2020)) with baseline LIVs such as convolutions, recurrences, or attention. Other backbones are ran- domly initialized, with random LIV class choices and compositions. As the focus is on backbone optimization, the pool of LIV classes is limited to subset of systems encodable through the operator and featurizer genomes, including voken-mixing structures from Section 2. This includes common dense channel-mixing featurizers (linear projections), more advanced Toeplitz loken-mixing featur- izers, and "differential" variants of all LIV classes except memoryless ones, which use two identical, parallel LIVs and output their difference. See Appendix A.6 for all included LIV classes
'e used two different randomly- evaluation datasets for our ablation and main experiments.
7

---

Published as a conference paper at ICLR 2025
FA GA
NSCA--2
967708
2400
2407008
7.
455
6.00
20
0.5
10.5
10.0
10.5
191 PP5
PPL
PPU
197
Figure 5.1: [Left]] Evolutionary algorithms: Final populations evolved with the Firefly Algorithm (FA), Genetic Algorithm (GA), and Non-dominated Sorting Genetic Algorithm II (NSGA-2). [Right]: Backhone synthesis scales (left right): synthesized reduced depth ("motif," LIVs at 768), reduced width (24 LIVs 256), or full depth and width (24 LIVs at 768). Models are scaled to the same LIV count and width via stacking or width extension.
5.1 IDENTIPYING A SYNTHESIS PROTOCOL
Evolutionary algorithm We compare three gradient-free evolutionary algorithms- -ireffl rithm (FA) (Yang, 2009), Genetic Algorithm (GA) (Bremermann et al., 1966), and NSGA-2 (Deb al., 2002)-for optimizing STAR genomes toward quality and parameter count. Each algorithm evolves population of 16 genomes (8 LIVs, 768 dimensions) over 8 iterations (details in Appendix A.4)². Results show GA and NSGA-2 outperform FA, achieving lower parameter counts while maintaining comparable predictive quality. GA slightly surpasses NSGA-2 in performance but pro- duces larger models, whereas NSGA-2 achieves greater solution diversity (Fig. 5.1). We thus use NSGA-2 in subsequent experiments. Hyperparameter tuning indicates optimal performance with population size of 16, a 10% mutation probability, and 2 crossover points (Appendix A.4)*
Firefly Algo-
Synthesis scale Automated architecture optimization in language modeling faces the challenge of high compute costs for training and evaluating large-scale models. We explore two paths to reducing this cost: (a) optimizing smaller backbone motifss (groups of fewer LIVs in deeper mod- els) and stacking them to build deeper models; and (b) optimizing full-depth backbones at reduced widths. For both approaches, we evolve 16 genomes over iterations, optimizing for parameter count and quality, and compare the resulting models to those synthesized at full width and depth under identical settings. From each evolution, we select backbones smaller in parameter count than Transformer++ and StripedMamba baselines (Appendix A.3) and with the lowest evaluation perplexity. Selected backbones are scaled to the same LIV count and width (via motif stacking or width extension) and trained for 5B tokens before downstream evaluation.
Finding Synthesizing backbones full width and depth yields consistent improvements, while reduced-widd synthesis achieves similar results with fewer successful candidates. Motif synthesis underperforms both approaches (Fig. 5.1).
5.2 SYNTHESIZING HIGHI-QUALIT LANGUAGE MODELS
We apply identified protocol to synthesize high- quality backbones for language modeling by evoly- ing a population using perplexity the only objec- tive. When optimizing for quality, STAR evolution achieves a reduction of the average quality of an ini- tial population by 1.0 PPL. point without changes to model depth and width (Figs. 5.2, B.34).
le?
Optimizing: Quality
9.0
Synthosized
Treddumee+++
Preraas (Noa-Eenbeel.)
8.5
Stripei3Lumha..
ag
12
Geseration
8.0
Finding 2: STAR backtoones-optimized for qullity-ou parameter-matched Trans- former++ and StripedMamba backbones in RedPajama eval. PPL as well as on Hellaswag, ARC-Easy, Winogrande, PiQA, and SciQ. Im- provements of STAR backbones over standard hybrids on benchmark averages is 2 times larger than the improvement of hybrids over Transformers (Tables 5.1 and A.4).
7.5
8.5
9.0
9.5
10.0
10.5
PPL
Figure 5.2: Genome scores during S STAR evolu-
tion. when optimizing for quality.
"FA and GA optimize a single objective, using the sum normalized loss and parameter count P: U(L) + UP)), where (xx) 3--min X) "The 2 best-performing genomes per population carried over prevent performance regression.
m for X.
8

---

Published as a conference paper at ICLR 2025
Backbone Optimized for
Size
Cache (bytes 4K)
RedPj. ppl |
Hella. acc. norm. f
ARC-e
Wino,
PIQA
SciQ
Avg.
acc. f
acc.
acc. f
acc,
Transformer++
85M
150MB
28.9
38.8
$1.2
61.2
64.1
StripedMamba
80M
25MB
7.2
28.6
39.3
51.1
60.9
67.4
49.5
STAR-1/ Quality
79M
100MB
7.0
29.8
39.3
51.2
62.2
72.5
51.0
STAR-2/ Quality
80M
82MB
7.1
29.2
40.5
51.1
61.6
72.4
51.0
STAR-3 / Quality
78M
120MB
7.1
29.7
40.0
50.9
62.0
71.2
51.0
STAR-4/ / Quality
79M
94MB
7.1
29.3
39.7
51.0
61.5
72.6
50.8
STAR-1 Q,+Size
74M
63MB
7.2
28.9
39.3
51.0
61.8
67.6
49.7
STAR-2/ Q.+Size
74M
64MB
7.2
28.7
37.5
52.8
61.0
68.9
STAB-3/ / Q,+Size
70M
151MB
7.2
29.2
39.5
51.9
61.5
69.4
50.3
STAR--4 Q-+Size
70M
114MB
7.2
29.2
40.0
52.7
61.4
68.9
50.4
STAR-1 / Q.+Cache
77M
16MB
7.2
28.9
40.0
51.3
61.0
66.4
49.5
STAR-2/ Q +CCahe
83M
22MB
7.2
28.7
40.1
50.3
62.2
66.0
49.5
STAR-3 Q.+Cache
75M
23MB
7.2
28.9
40.6
50.2
61.3
67.2
49.6
STAR-4/ C.+Cc
78M
22MB
7.2
29.1
53.0
62.2
66.7
50.2
Table 5.1: Evaluation of backbones optimized for quality (upper third), quality and size (middle third), and quality and cache (lower third). We L.M-Faal-Harness (Gao al., 2024), reporting Transformer++ and StripedMamba baselines trained on the same data. Size indicates trainable parameter count w/o embeddings.
Backbone
Size
Cache
RedPj. ppl
ARC-c acc.
Hella.
ARC-e
Wino.
PiQA acc.
SeiQ
Avg.
(bytes 4K)
norm. t
Transf++
1.2B
805MB
5.9
27.3
49.3
58.9
51.3
71.2
86.3
57.4
StripedMb.
1.IB
136MB
5.7
28.3
52.8
59.8
54.1
72.9
86.0
59.0
STAR-IB
1.IB
86MB
5.7
27.9
52.6
53.9
71.8
87.0
59.0
Table 5.2: Evaluation of IB STAR backbone (48 LIVs, 2048 width) optimized quality and cache (LM- Eval-Hamess (Gao et al., 2024)). Results are compared to parameter parameter-matched matched Transformer++ and Striped- Mamba baselines trained on 40B RedPajama tokens. Size excludes embedding layers.
5.3 SYNTHESIZING PARAMETER-EPFICHEE HIGH- QUALITY LANGUAGE MODELS
We observed that STAR can synthesize high-quality language models. Next, we ask whether it can likewise synthesize language models of high quality and smaller parameter counts. To this end, we evolve a population using evaluation perplexity and parameter count as objectives. Optimizing for quality and size, STAR evolution improves an initial population by 0.5 PPL points at aver- age reduction 10% in trainable parameter count, as shown in Figure 1.1. We also evaluate the performance of representative synthesized backbones when training them longer.
Finding 3: STAR backbones- oottmmize for quality and site-outperform Transformer++ and match StripedMamba backbones in RedPajama eval PPL, while surpassing both on Hellaswag, ARC-Easy, Winogrande, PiQA, and SciQ, with reduction in parameter count by 13% and 8% respectively (Table 5.1 and A.4).
5.4 SYNTHESIZING CACHE-EFFICIENT EFFICIENT HIGH HIGH-QUALEA QUALITY LANGUAGE MODELS
High inference costs limit the widespread of language models. To address this, test whether STAR can synthesize architectures with reduced inference cache size without sacrificing predictive quality. By optimizing for perplexity and cache size (Fig. 5.3), STAR evolution improves initial population by 0.4 PPL. points and a 40% cache size reduction.
Finding 4: STAR backbones nackbones-optimizzedi for quality and cache size- outperform Transformer++ and match StripedMamba in RedPajama perplexity while surpassing both on HellaSwag, ARC-Easy, Winogrande, and PiQA, with cache size reductions of 90% and 36%, respectively, at sequence length of 4096 tokens (Tables 5.1 and A.4).
Our previous experiments have shown that STAR synthesis smaller scales yields suboptimal re- sults (see Fig. 5.1). Nevertheless, when scaling a 24-LIV backbone at 768 dimensions, optimized for quality and cache size, to 48 LIVs 2048 dimensions, through stacking and width extension,

---

Published as a conference paper at ICLR 2025
1ch Optimising: Quality & Cache
Inforeace Cache Size Scaling (1892008)
1.50
Senthwelsed
10-1
Tambormees+
15
Cache (byses 9 (is))
1.25
Strppeikaamo
12
Clecke Sita (boted)
1.00
Geserniio
10.
0.76
0.50
Trnedimmnt+
Stoppedhaatt
0.25
8.6
650-6 100072
262144
524225
PPL
Seguence Length
Figure 5.3: [Left]]: Genome scores during STAR evolution when optimizing for quality and cache size. Cache size is computed at a fixed sequence length of 4096 tokens. [Right]: Cache size scaling with increasing input sequence length for the models show in Table 5.2.
we find that it matches the performance of a parameter- matchhe StripedMamba baseline and outper- forms parameter-matchit Transformer++ baseline when all are trained for 40B tokens:
Finding 5: Scaling a synthesized STAR backhone from 125M to 1B parameters (Fig. B.25) out- performs wrrameter-matched Transformer++ baseline and matches StripedMamba baseline in RedPajama evaluation perplexity and performance on ARC-Challenge, HellaSwag, Winogrande, PiQA, and SciQ, while reducing cache size by 90% and 37%, respectively (Table 5.2).
Finding Overall, synthesized STAR backbones outperform Transformer++ and StripedMamba baselines with hit rates of 8/8, 7/8, and 7/8 when optimizing for quality, quality and size, and quality and cache, respectively (Tables 5.1 and A.4).
5.5 COMPARING SYNTHESIZED BACKBONES
STAR is well-suited for the synthesis of backbones optimized for various objectives. In addition, it provides tool for the automated discovery of backbone motifs that drive these performance improvements, as it evolves populations towards using those combinations and compositions of LIVs that perform best. Figure 5.4 demonstrates this for the evolution targeting model quality and size. We observe that STAR favors gated short convolutions (GConv-1), grouped query (Ainslie et al., 2023) attention variants (SA-3), and differential variants of input-varying recurrences (Rec- I-Diff), as well as SwiGLUs (Shazeer, 2020) (GMemless). more detailed analysis of the motifs resulting from all STAR evolutions, well as visualizations of all evaluated backbones, can be found in Appendix B.
Optimizing: Quality & Size
6 CONCLUSION
This work presents STAR, a framework for the auto- mated evolution of tailored architectures. Unlike other approaches, STAR explores a hierarchical, general de- sign space encompassing attention, recurrences, convo- lutions, and other input-dependent units. Its design space well-conditioned, with most architecture candidates training stably. Using evolutionary methods on nu- merical backbone encoding, STAR achieves significant improvements in perplexity, downstream benchmarks, model size, and inference cache compared to optimized striped hybrid and Transformer baselines.
RAS
M4-3
Bro2
RewiDE
BoaDE
#'
Woighe
Foiticn
Reproducibility statement To aid in reproducibil- ity, we run optimization and training on open-source datasets (RedPajama). We report full training and STAR evolutionary algorithms details in Appendix and a full description of the S STAR genome in Appendix A.6.
Figure 5.4: Evolution of backbones opti- mized for quality and size, averaged per population. Distance measures the num- ber of other LIVs between two connected LIVs.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Joshua Ainslie, James Lee-Thorp. Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head check- points. arXiv preprint arXiv:2305. 222223305.124555 13245, 2023. 10
Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language mod- els. arXiv preprint arXiv:23/2.04922) 2023. 1, 15
Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré. Simple linear attention language models balance the recall-throughee tradeoff. arXiv preprint arXiv:2402. 18668, 2024.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly
learning to align and translate. 2014. 3
Hans-Georg Beyer and Hans-Paul Schwefel. Evolution strategies-a comprehensive introduction.
Natural computing, 1:3-52, 2002.
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint 1xXi::2401.02944 2024.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020. 7
William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan Kelly. Reducing transformer key-value cache size with cross-layer attention. arXiv preprint 2rXiv:2405.12981, 2024. 2, 4
Hans Bremermann, M Rogson, and S Salaff. Global properties of evolution processes. Natural automata and useful simulations, pp. 3-41, 1966.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005. 2020. 1,
2
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1803.0:80030550 003055577 2018.
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International conference on machine learning, pp. 933-941. PMLR, 3017. 3
Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. A fast and elitist mul- tiobjective genetic algorithm: Nsga-ii. IEEE transactions on evolutionary computation, 6(2): 182-197, 2002. 8
Patrick Dewilde and Alle-Jan Van der Veen. Time-varying systems and computations. Springer Science & Business Media, 1998. 4
Daniel Y Fu, Tri Dao, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. arXiv preprint arXiv:2212. 14052, 2022.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL https : zenodo. arg/records/ 66008.0.. 3, 22
11

---

Published as a conference paper at ICLR 2025
Albert Gu and Tri Dao. Mamba: Linear-ting sequence modeling with selective state spaces, 2023. 3, 16, 20
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces, 2022.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Train- ing compute-optimal large language models. arXiv preprint arXiv: 15556, 2022. 1
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Pro- ceedings of the IEEE/CVF international conference on computer vision, pp. 1314-1324, 2019. 1,
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention, 2020. 3
Sheng Li, Mingxing Tan, Ruoming Pang, Andrew Li, Liqun Cheng, Quoc Le, and Norman Jouppi. Searching for fast model families on datacenter accelerators. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8085-8095, 2021. 1,2
Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwarke et al. Jamba: A hybrid transformer- mamba language model. arXiv preprint arXiv:240. 200::247..1887, 2024. 2
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv
preprint 1rXiv:1806.0900 2018. 1,2
Zechun Liu, Changsheng Zhao, Forrest landola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernic Chang, Yangyang Shi, Raghuraman Krishnamoorthi, et al. Mobilellm: Optimizing sub-billion parameter language models for on-device use cases. arXiv preprint arXiv:2402.14900 2024. 2
Zhuang Liu, Hanzi Mao, Chao- Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 11976-1 1986, 2022. 2
Ilya Loshchilov, Frank Hutter, et al. Fixing weight decay regularization in adam. arXiv preprint
arXiv: 5, 2017. 7
Eric Martin and Chris Cundy. Parallelizing linear recurrent neural nets over sequence length. arXiv preprint /ttl::/70004444 2017. 1,3
Stefano Massaroli, Michael Poli, Daniel Y. Fu, Hermann Kumbong, Rom N. Parnichkun, Aman Timalsina, David W. Romero, Quinn McIntyre, Beidi Chen, Atri Rudra, Ce Zhang, Christopher Re, Stefano Ermon, and Yoshua Bengio. Laughing hyena distillery: Extracting compact recur- rences from convolutions, 2023. 1
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Oc-
cupancy networks: Learning 3d reconstruction in function space. In Proceedings the IEEE/CVF conference on computer vision and paltern recognition, pp. 4460-4470, 2019. 4
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, the transformer era. arXiv preprint arXiv:2305.1 13048, 2023.
Rwkv; Reinventing mns for
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International conference on machine learning, pp. 4095-4104 PMLR, 2018. 1,2
Michael Poli, Stefano Massaroli, Eric Nguyen. Daniel Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. In International Conference on Machine Learning, pp. 28048.28078. PMLR, 2023. 3, 20, 21
12

---

Published as a conference paper at ICLR 2025
Michael Poli, Armin W Thomas, Eric Nguyen, Pragaash Ponnusamy, Bjöm Deiseroth, Kristian Kersting, Taiji Suzuki, Brian Hie, Stefano Ermon, Christopher Ré, et al. Mechanistic design and scaling of hybrid architectures. arXiv preprint arXhv:400..78444, 2024. 2, 15, 16
Nicholas Roberts, Samuel Guo, Zhiqi Gao, Satya Sai Srinath Namburi GNVV, Sonia Cromp, Chengjun Wu, Chengyu Duan, and Frederic Sala. Pretrained hybrids with mad skills. arXiv preprint arXiv:2406.00894 2024. 2
David W Romero, Anna Kuzina, Erik J Bekkers, Jakub Tomezak, and Mark Hoogendoorn. Ck- conv: Continuous kernel convolution for sequential data. arXiv preprint arX1v:21022022111.0000 2021.
1.4
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. arXiv preprint arXiv: 10641, 2019. 7
Noam Shazeer. Fast transformer decoding: One write-head all you need. arXiv: 19.102/50 2019. 2, 20
arXiv preprint
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020. 3, 10, 16, 21
Vincent Sitzmann, Julien NP Martel, Alexander W Bergman, David B Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. arXiv preprint 0rXiv:2006.09661 2020. 4
Jimmy T. H. Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for sequence modeling, 2023. 1,3
David So, Wojciech Manke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc Le. Searching for efficient transformers for language modeling. Advances in neural information processing systems, 34:6010-6022, 2021. 1, 20
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127:127063, 2024. 16
Mingxing Tan and Quoc Le. Ffficientnet22: Smaller models and faster training. In International
conference on machine learning, pp. 10096-10106. PMLR, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017. 3, 16
Maurice Weber, Daniel Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, et al. Redpajama: an open dataset for training large language models. arXiv preprint arXiv:2411. 2024.
Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In EMNLP, 2017. 7
Colin White, Mahmoud Safari, Rhea Sukthanker, Binxin Ru, Thomas Elsken, Arber Zela, De- badeepta Dey, and Frank Hutter. Neural architecture search: Insights from 1000 papers. arXiv preprint arXiv:2301.00 088027...77...... 2023.
Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-effiiiet training, 2024a. 3, 20
Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transform- ers with the delta rule over sequence length. arXiv preprint arXiv:2406.06484 2024b.
Xin-She Yang. Firefly algorithms for multimodal optimization. In International symposium on stochastic algorithms, pp. 169-178. Springer, 2009. 8
Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei. Differential transformer. arXiv preprint arXiv:2410.05258 2024. 18, 21
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- 2019. 7
chine really finish your sentence? arXiv preprint
13

---

Published as a conference paper at ICLR 2025
Supplementary Material
CONTENTS
Introduction
2 Foundations of the Search Space
3
Describing Operators and Backbones with STAR Genomes
3.1
Backbone Genome
3.2
Operator and Featurizer Genomes
Synthesizing Architectures by Evolving Genomes
4.1
Key Steps of STAR Evolution Guiding Evolution with Hierarchical Mutations
4.2
5
Experiments
5.1
Identifying Synthesis Protocol
8
5.2
Synthesizing sigh-Quality Language Models
5.3
Synthesizing Parameter- Purameter-Efficient High Quality Language Models
5.4
Syathesizing Mache-Efficien High-Quality Language Models
5.5
Comparing Synthesized Backbones
10
6
Conclusion
10
Experimental Details
15
A.I
Discussion
................................
15
A.2
Training
15
A.3
Baseline models
16
A.4
Evolutionary Optimization Algorithms
16
A.5
Genome Scores
17
A.6
Linear Varying Systems and Featurizers: Option Pools
17
A.6.1 Backbone Genome
18
A.6.2
Operator Genome
19
A.6.3 Featurizer Genome
A.7
Extending the backbone genome for variable residual connections
21
A.8
Evaluation of synthesized backbones
Visualization and Analysis of STAR Backbones
22
B.I Recurring Motifs
23
14

---

Published as a conference paper at ICLR 2025
A EXPERIMENTAL DETAILS
A.1 DISCUSSION
Limitations and extensions of STAR While the LIV search space is general in the context of sequence modeling primitives, does not include classes of functions that can be embedded in backbone, potentially missing options for further optimization.
Relation to scaling laws and mechanistic design Since STAR evolution can target any objective, the methods presented in this paper are suited to integration within scaling laws protocols. Another options is optimizing efficiently computed metrics that correlate with performance at scale e.g., average accuracy on curated synthetic tasks (Arora et al., 2023; Poli et al., 2024).
Optimizing variable depth backbones Currently, STAR optimizes fixed-length genomes, lim- iting architectures to fixed depth and width. Optimizing variable-depth and variable- wiith archi- tectures challenging due to the hierarchical and modular design space. Shallower genomes are computationally cheaper and converge faster but may lack the complexity needed for difficult tasks. Deeper genomes offer greater representational power but expand the search space, risking subop- timal convergence due to overfitting or inefficient sampling. Future extensions of STAR could ad- dress these challenges with adaptive mechanisms like depth-aware sampling or dynamic penalties to improve scalability. Testing these methods tasks requiring deeper architectures will be key to enhancing STAR's robustness and versatility.
Multi-level optimization STAR streamlines the search by treating the genome as unified en- tity, enabling efficient pelloration-especiiane beneficial for rapid iteration limited resources. However, this approach may not fully exploit the hierarchical design space, potentially overlook- ing dependencies or key subspaces across genome levels. contrast, a multi-stage optimization strategy could systematically refine each hierarchical level-featurization, operator structure, and backbone uusionnee--inggggg tailored evolutionary algorithms. This could improve convergence, especially in complex task settings, by leveraging the genome's modularity and addressing interactions incremen- tally, but would increase algorithmic complexity and computational costs.
A.2 TRAINING
Tables A.1, A.2, and A.3 provide an overview of the training settings used during STAR evolution and when evaluating the resulting synthesized backbones.
Table A.1: Training setting during STAR evolution.
OPTIMIZER
ADAMW 31, B2 0.9, 0.95 0.25M TOKENS 5000 COSINE DECAY 500 STEPS 0.0008
OPTIMIZER MOMENTUM
BATCH SIZE
TRAINING STEPS
LEARNING RATE SCHEDULE
LINEAR LEARNING RATE WARM-UP
BASE LEARNING RATE
WEIGHT DECAY
0.1
DROPOUT
NONE
GRADIENT CLIPPING
1.0
Table A.2: Training setting for evaluation of synthesized backbones.
OPTIMIZER
ADAMW
OPTIMIZER MOMENTUM
81, 82 0.9, 0.95 0.25M TOKENS 20000 COSINE DECAY 1500 STEPS 0.0008
BATCH SIZE
TRAINING STEPS
LEARNING RATE SCHEDULE
LINEAR LEARNING RATE WARM-UP
BASE LEARNING RATE
WEIGHT DECAY
0.1
DROPOUT
NONE
GRADIENT CLIPPING
1.0
15

---

Published as a conference paper at ICLR 2025
Table A.3: Training setting for models with 48 LIVs at width of 2048 dimensions (see Table 5.2).
OPTIMIZER
ADAMW B1, 0.9, 0.95 0.75M TOKENS 50000 COSINE DECAY 3500 STEPS 0.0008
OPTIMIZER MOMENTUM
BATCH SIZE
TRAINING STEPS
LEARNING RATE SCHEDULE
LINEAR LEARNING RATE WARM-UP
BASE LEARNING RATE
WEIGHT DECAY
0.1
DROPOUT
NONE 1.0
GRADIENT CLIPPING
A.3 BASELINE MODELS
baseline models are trained according to the recipe described Table A.2. We train the two baseline models each at two depths widths to match the parameter counts of our synthesized models (Tables 5.1 and 5.2): 24 operators at 768 dimensions and 48 operators at 2048 dimensions.
Transformer++ A Transformer (Vaswani et al., 2017) with an improved architecture, namely rotary positional encodings (Su et al., 2024), SwiGLU MLP (Shazeer, 2020), RMSNorm instead of LayerNorm, and no linear bias term. We use a head dimension of 64 for all Transformer++ baselines trained in this work, resulting in 12 and 32 heads for models with 768 and 2048 width.
Manip A striped hybrid backbone (Poli et al., 2024) that combines Mamba (Gu & Dao, 2023), SwiGLU MLP (Shazeer, 2020), and self-attention (Vaswani et al., 2017) operators. The 24 operator backbone is composed of interlcaved Mamba and SwiGLU operators, with the exception of operators 6 and 18, which are softmax attention. The Mamba operators have state size of 32, while we use a head dimension of 64 for the attention operators. The 48 operator StripedMamba backbone obtained by stacking two 24 operator backbones in depth and increasing the overall width to 2048.
A.4 EVOLUTIONARY OPTIMIZATION ALGORITTMS
In this section, we present an overview of the three variants of commonly used gradient-free evo- lutionary optimization algorithms applied in this work. These algorithms have been adapted as necessary to be compatible with the STAR genome, Before discussing their individual differences, will first describe several core operations shared across all variants.
Tournament selection selects parent candidates by randomly sampling a subset of individuals from the population and choosing the highest-performing individual from this subset. This method promotes the propagation of strong candidates while preserving diversity through its inherent ran- domness.
K-point crossover recombines the genomes of two parents by exchanging segments of genetic material at k randomly selected points, creating offspring that inherit a mix of traits from both parents.
Elitism balances exploration and exploitation by preserving subset the top-performing indi- viduals from the current population and carrying them over directly to the next generation. This approach ensures that bigh-quality solutions are not lost and generally accelerates convergence, while reducing the risk of premature convergence to suboptimal regions of the solution space.
Mutation maintains population diversity by introducing randomness through random alterations to genome, helping the algorithm explore new regions of the solution space.
Firefly Algorithm (FA) The Firefly Algorithm (FA) is inspired by fireflies' attraction to brighter (fitter) individuals based on their light intensity. FA assigns a light intensity a T++ to each genome, inversely related its fitness score In each iteration, FA pairs genome i with genome via tournament selection. a attraction strength 8 fo ((-rr),, where 3o is bascline attraction, 7 is the light absorption coefficient, and is the similarity ratio of matching LIVs, and (2) replacing LIV of with of with probability B. If az 2 aj. 9" remains unchanged. Finally, g undergoes mutation.
FA updates g to resemble g through two steps: (1) computing
16

---

Published as a conference paper at ICLR 2025
Genetic Algorithm (GA) In each iteration, the Genetic Algorithm (GA) generates genomes by: (i) selecting two parents via tournament selection; (ii) recombining them using k- k-point crossover; and (iii mutating the recombined genomes.
Non-dominated Sorting Genetic Algorithm II (NSGA-2) NSGA-2 extends GA for multi- objective optimization by maintaining a diverse set of Pareto-optimal solutions through non- dominated sorting and crowding distances. It first segregates genomes into fronts, with the first front containing the most optimal, non-dominated solutions. Genome g¹ dominates g outper- forms y in at least one objective without being worse in others. Within each front, genomes are sorted by objective scores, such as predictive quality, and assigned crowding distances 9 Boundary solutions, with extreme objective scores, receive infinite crowding distances ensure preservation, while non-boundary solutions are assigned crowding distances based on differences from adjacent neighbors. Selection then favors genomes with higher front rank and crowding distance.
Determining parametesseeters for NSGA-2 We found that NSGA-2 performed the best in our comparison (Section 5). We also investigate the optimal hyper-parameter settings for NSGA-2, specifically population size mutation probability p. and number of crossover points k. To do this, we evolved two population sizes (16 and 32), optimizing for quality and parameter count, while varying the number of crossover points (1 2) and mutation probabilities (10% 20%) (Fig. I. To keep to overall number of sampled genomes constant, we evolve populations containing 16 genomes for 8 iterations and populations containing 32 genomes for 4 iterations. All genomes contain 24 LIVs at a width of 64 dimensions. Our results indicate that NSGA-2 performs best with a population size of 16, 2 crossover points, and 10% mutation probability.
a:36, sx0.1.
po0.1, 1:2
p:0.,,
36, p:0.2, luz2
px0.1. kel
po0.1, p.2.Z lel R2 p:2.
400000
TBt.
P9C.
PP1
FPT.
Figure A.1: Comparison of different paperr-arrmettr settings for NSGA-2.
A.5
GENOME SCORES
We provide visualizations of all genome scores in all STAR evolutions:
Evolutionary algorithm ablations: figures B.28 and B.29.
ii Comparison of synthesis scales: figures B.30, B.31, and B.32.
iii. Direct quality optimization: figure B.34.
iv. Quality and size optimization: figure B.33.
Quality and cache optimization: figure B.35.
A.6 LINEAR INPUT-VARYING SYSTEMS AND FEATURIZERS: OPTION POOLS
Modifying the STAR genome the numerical encoding enabled by the LIV framework - allows exploration model architectures with substantial differences multiple levels, including in the type of LIVs they are composed of, as well as their ordering and interconnection.
Recall that the STAR genome is structured hierarchically: At the highest level, the backbone genome specifies the composition of LIVs in the backbone, with each LIV represented by a single inte- ger. Expanding this integer reveals the operator genome, which identifies the LIV. At the operator genome level, the specific featurizer of the LIV is similarly encoded as a single integer, which can
"Crowding distance in multi objectivee optimization is a measure of the density of solutions surrounding particular solution, caleulated the of the normalized distances between adjacent solutions across all objectives: d, EM
f++1 fi-1
where d, is crowding distance for solution i, and fm objective 7 Taaa Fagm
value the m objective.
17

---

Published as a conference paper at ICLR 2025
be further expanded into the featurizer genome, which specifies the particular featurizer used by the LIV.
Below, we provide an overview of the specific integer values and their corresponding meanings considered in this work for each level of the STAR genome. Note that this definition of the genome is not exhaustive and can be extended further.
A.6.1 BACKBONE GENOME
The basic formulation of the backbone genome (without extensions such as residual composition explored in A.7) consists of sequences five integers, where each sequence corresponds to one of the LIVs contained in the backbone, defining (a) the individual operator and (b) composition rules with other LIVs.
For each integer in the genome, we detail the set of options considered in this work. While we present specific set of choices here, the pool of options can be readily expanded, provided the results compile to valid operators and backbones within the LIV framework.
Tthe first integer specifies the LIV class and can take on the following values in our experiments:
1-4. Softmax attention variants (SA) 1-4
5-6. Recurrences (Rec) 1-2
7-8. Gated convolutions (GConv) 1-2
9. Gated memoryless unit (GMemless)
10-17. Differential variants LIV classes 1-8 (akin to the "Differential Transformer" (Ye al., 2024))
The second integer defines the weight-sharing structure of the LIVs' featurizers. Specifically, all LIVs within backbone that share featurizer weights will have the same value in this position. The mapping of integer values to the weight-sharing structure thereby depends on the number of occurrences of each LIV class (N) in the backhone. If all LIVs of the same class share featurizer weights, they all assigned value of 1 at this position. Conversely, if none of these LIVs share featurizer weights, each LIV is assigned unique integer value from 1 to N.
The third integer defines the strategy for sharing featurizer weights. In this work, we are consider- ing only two possible featruizer sharing strategies:
1. No weights are shared
2. All weights are shared
The fourth integer establishes the feature group sharing structure of the LIVs. Similar to the featur- izer weight-sharing structure (second integer), all LIV's within a backbone that share feature groups will have the same value at this position. The assignment of integer values to feature group sharing follows the same logic described for the second integer.
The fifth integer specifies the strategy used for sharing feature groups. Since feature groups are unique to each LIV class, the possible values for this integer vary depending range is between (indicating no shared feature groups) and N + 1, where N represents the number of unique feature groups in the given LIV class. For example, in the case of softmax attention, the possible values are:
the LIV class. The
1. No shared feature groups
2. Shared key cache
3. Shared value cache
4. Shared key and value cache
For clarity, we provide examples of backbone genomes below:
18

---

Published as a conference paper at ICLR 2025
11111 91111 12121 92121
This genome consists of four LIVs arranged in an interleaved order. The first and last LIVs belong to the SA-1 class, while the second and fourth LIVs belong to the GMemless class, None of the LIVs share featurizer weights or feature groups, as each occurrence of a LIV class has distinct integer values for featurizer sharing (integer 2) and feature group sharing (integer 4). Both the featurizer sharing strategy (integer 3) and feature group sharing strategy (integer 5) are set to indicating no sharing.
11111
91111 51111 92121
This genome represents a variation of the genome shown above, where the third LIV has been switched from class SA-1 to class Rec-l. As before, none of the LIVs share feature groups of featurizer weights.
11111 91111 51111
92121
11221
91131
This genome is comprised of six LIVs. The first and fifth belong to the SA-1 class. The first and fifth belong to the SA-1 class, the second, fourth, and sixth to the GMemless class, and the third to the Rec-I I class. Notably, the two SA-1 LIVs share the weights of their featurizers, as both have value at the second integer position and a value of 2 at the third integer position.
A.6.2 OPERATOR GENOME
The operator genome specifies a particular LIV and consists of five integer values. Integer I sum- marizes the LIV's featurizer, integers 2-4 define the token-mixing structure of the LIV, and integer 5 determines the channel-mixing structure.
The first integer specifies the featurizer class. In this work, we consider the following featurizer classes:
1. Dense channel mixing structure with diagonal token mixing structure all feature groups (3 groups e.g., SA-I).
2. Dense channel mixing structure with Toeplitz token mixing structure on all feature groups (3 groups e.g., in SA-2).
3. Variant of 1. where a repeat factor of is applied to the last two feature groups (e.g., used for SA-3).
4. Variant of 1. where repeat factor of 2 is applied to the last two feature groups (e.g., used for SA-4).
5. Dense channel mixing and Toeplitz token mixing structure. An expansion factor of 16 is applied the last two feature groups (e.g., used for Rec-1).
6. Dense channel mixing and Toeplitz token mixing structure. An expansion factor of is applied to the last two feature groups (e.g., used for Rec-2).
7. Diagonal channel mixing structure with Toeplitz token mixing structure for all feature groups. One of the groups is explicitly parametrized (e.g., short convolutions of length 3 used for GConv-1).
8. Variant 5. Where the short convolution kernel feature group is replaced with implicitly parametrized feature group (e.g., long convolutions used in GConv-2).
9. Dense channel mixing structure with diagonal token mixing structure with feature groups (e.g., used for GMemless). Variant of I with one fewer feature group.
The second integer defines the linear token- mixing structure of the LIV, before any final nonlinear- ity, and can take on the following values in this work:
1. Diagonal (e.g., GMemless)
2. Low rank (e.g., SA)
19

---

Published as a conference paper at ICLR 2025
3. Scaled Toeplitz (e.g., GConv)
4. Sequentially semi-separable (e.g., Rec)
The token-mixing structure determines the class of matrix multiplication algorithms that can be used to apply the operator to the input. For instance, if the LIV is sequentially semi-separable, it supports an (l) algorithm implemented as linear recurrence.
The third integer determines whether any structured sparsity mask is applied to the token-mixing structure. We consider the following in this work:
1. No sparsity
2. Banded (e.g., as used for short convolutions)
Note that all models trained in this work are causal, and as such upper-triangulat sparsity masks are introduced whenever needed e.g., in LIVs wrapped by nonlinearities.
The fourth integer describes whether any final nonlinearity is applied to the token-mixing structure. We consider the following set of static and normalization non- non-lincarities this work:
1. None
2. Softmax
3. ReLU
4. Swish
The fifth integer deseribes the LIV channel mixing structure, for which consider the following two possible structures in this work:
1. Diagonal
2. Dense
Below provide the specific operator genomes for each LIV class considered in this work:
SA-1 12121 refers to the standard attention operator using a featurizer with a dense channel mixing and diagonal token mixing structure with 3 feature groups, low-rank moken-mixing structure, sparsity mask, softmax non-linearity, and a diagonal channel-mixing structure.
SA-2 22121 represents a variant SA-1 whose featurizer has dense channel mixing and Toeplitz token mixing structure. This is realized in practice by adding depth- wiihh convolutions to the featur- izer, in line with the findings (So et al., 2021; Poli et al., 2023).
SA-3 32121 represents variant of SA-1 where repeat factor of is applied to the last two feature groups. This is akin to variants multi-query (MQA) and grouped-ques attention (GQA) (Shazeer, 2019).
SA-4 42121 represents a variant of SA-3 with a lower repeat factor of for the last two feature groups
Rec-1 54111 is characterized by featurizer with Toeplitz token mixing structure and dense chan- mixing structure with feature groups where an expansion factor of 16 is applied the last two feature groups, semi-separable token mixing structure, no sparsity, no non-linearity, and a diag- onal channel mixing structure. Rec-1 is representative of variety of modern input-varying linear recurrent layers (Gu & Dao, 2023; Yang et al., 2024a).
Rec-2 64111 is the same as Rec-1, with the exception of an expansion factor of 2 for the last two feature groups.
GConv-1 73111 is characterized by a featurizer with diagonal channel mixing structure and Toeplitz token mixing structure (using a short convolution filter) applied to all feature groups in
20

---

Published as a conference paper at ICLR 2025
addition to explicitly parametrized feature group for short convolution, Toeplitz token mixing structure, no sparsity, no non-linearity, and a diagonal channel mixing structure.
GConv-2 83111 has the structure GConv-1, except for the of implicitly parametrized feature group for long convolutions. GConv-2 represents modern operators in the gated long convo- lution family (Poli et al., 2023).
GMemless 91142 is characterized by featurizer with dense channel mixing structure and diagonal token mixing structure with two feature groups, diagonal token mixing structure, no sparsity, swish non-linearity, and dense channel mixing structure. GConv-1 thereby represents SwiGLU (Shazeer, 2020).
In addition, we include differential variants of LIV classes 1-8 (SA, Rec, and GConv), where two identical LIV's applied in parallel, outputting their difference, similar to the "Differential Trans- former" (Ye et al., 2024).
A.6.3 FEATURIZER GENOME
The featurizer genome composed of sequences of seven integers, one sequence per feature group of the featurizer. The first five integers are akin to integers 2-5 of the operator genome, respec- tively indicating the linear token-mixing structure, whether any sparsity applied, whether any non-linearity is applied, and the channel mixing structure. The sixth integer indicates an expansion factor of the feature group channel dimension over the input channel dimension. The seventh inte- ger indicates repeat factor for how many times the feature groups are replicated across the channel dimension.
Note that we restrict the featurizer genome to a maximum of 5 feature groups (ie, 35 integers in total). If the featurizer takes less than feature groups, we the sequences of all excess feature groups to O.
A.7 EXTENDING THE BACKBONE GENOME FOR VARIABLE RESIDUAL CONNECTIONS
In our main experiments, constrain backbone topologies pre-norm residual structure, where the outputy nM of LIV 7 at backbone depth m is defined as: y" = T(norm(y"=)) norm(y" y"
However, the backbone genome can be extended to support more flexible residual streams. This can be achieved by introducing sixth entry to each subsection of the backbone genome, corresponding its respective LIV. Recall that the backbone genome consists of sequences of five integers, where each sequence encodes the characteristics of an LIV and its integration within the composition struc- ture. If two LIVs, at depths m and n (where m < n),, share the same value at this new genome position, the residual stream is extended such that: y" T(norm(u)) norm(u) u, where y"1 + ym
We evaluate this extended backbone genome in an ablation study by comparing the outcomes of two STAR evolutions: one incorporating this extension and the other using the standard backbone genome. In both conditions, we evolve population of 16 genomes, each consisting of 24 LIVs with a width of 768 dimensions, for generations. The results indicate that the extension allows STAR to synthesize architectures of even smaller parameter counts while maintaining the same level of quality (see Fig. A.2).
Based on these promising findings, we plan to further investigate composition strategies via residuals and inputs to LIVs in future work, in addition to improved featurizer interconnections e.g., sharing inputs to the system, the featurizer, and the residual stream itself.
A.8 EVALUATION SYNTHESIZED BACKBONES
Table A.4 provides overview of the evaluation performances of the remaining 4 synthesized backbones that were selected from each STAR evaluation and trained for longer (for comparison, Table 5.1).
21

---

Published as a conference paper at ICLR 2025
1x7 Gruendion
Generstion
Generation
Geoeration
Generation
40maa
Figure A.2: Comparison of two STAR evolutions with and without an extension of the backbone genome allowing for more flexible residual connections.
Backbone Optimized for
Cache
RedPj. ppl +
Hella. acc. norm. f
ARC-e acc. 1
Wino, acc.
PiQA acc. f
SciQ acc.
Avg.
(bytes 4K)
Transformer++
85M
150MB
7.3
28.9
38.8
51.2
61.2
64.1
48.8
StripedMamba
80M
25MB
7.2
28.6
39.3
51.1
60.9
67.4
49.5
STAR-5 Quality
78M
94MB
7.1
29.2
39.1
52.1
62.1
72.7
$1.0
STARR-6 / Quality
79M
94MB
7.1
29.0
39.9
50.9
61.7
71.1
50.5
STAR-7 Quality
79M
107MB
7.1
29.3
38.2
51.5
61.6
70.2
50.2
STAR-8 Quality
79M
94MB
29.1
50.8
62.0
70.3
50.6
STAR-5 Q.+Size
78M
64MB
7.2
29.2
40.0
52.7
61.0
67.8
50.1
STAR-6 / Q.+Size
73M
104MB
7.2
27.7
39.5
53.1
61.6
69.4
50.3
STAR-7 / Q.+Size
69M
170MB
7.2
27.8
39.2
49.9
61.2
69.5
49.5
STAR-8/ Q.+-iiee
72M
92MB
7.2
27.5
39.2
51.7
61.8
64.1
48.9
STAR-5 Q.+Cache
79M
22MB
7.2
28.9
40.0
50.2
61.1
69.1
49.9
STAR-6 Q++Cache
68M
25MB
7.2
29.1
40.0
51.3
60.9
68.7
50.0
STAR-7 Q.+Cache
75M
22MB
28.6
39.4
52.6
61.0
66.6
49.6
STAR-8 Q++Cache
74M
16MB
51.2
61.0
67.0
49.4
Table A.4: Evaluation of backbones optimized for quality (lower half) and quality and size (upper half). We test on L.M-vallHarneee (Gao et al., 2024), reporting parameter parameter-matched matched Transformer++ and StripedMamba baselines trained on the same data. Size indicates trainable parameter count, excluding embeddings layers. All models were trained for 5B tokens from Redpajama.
B VISUALIZATION AND ANALYSIS OF STAR BACKBONES
We provide visualization of the STAR backbones presented in Tables 5.1, 5.2, and A.4. Featurizer sharing between operators is indicated as solid black arrows on the right, feature group sharing as dashed black arrows on the left.
Direct quality optimization: figures B.1 (STAR-1), B.2 (STAR-2), B.3 (STAR-3), B.4 (STAR-4), B.5 (STAR-5), B.6 (STAR-6), B.7 (STAR-7), B.8 (STAR-8)
ii. Quality and size optimization: figures B.9 (STAR-1), B.10 (STAR-2), B.11 (STAR-3), B.12 (STAR-4), B.13 (STAR-5), B.14 (STAR-6), B.15 (STAR-7), B.16 (STAR-8)
iii. Quality and cache optimization: figures B.17 (STAR-1), B.18 (STAR-2), B.19 (STAR-3), B.20 (STAR-4).B.A (STAR-5), B.22 (STAR-6), B.23 (STAR-7), B.24 (STAR-8), and B.25 (STAR-IB)
We also provide overviews of the average count at which each LIV type occurs in population over the course of STAR optimization, the average count of LIVs that share featurizer weights or groups, and the average distance between LIVs sharing featurizer weights or feature groups" '0:
Direct quality optimization: figure B.26
Distance is indicated by the number of LIVs between two LIVs with connected through featurizer feature group sharing.
22

---

Published as a conference paper at ICLR 2025
ii. Quality and size optimization: figure 5.4
iii. Quality and cache optimization: figure B.27
We have observed that STAR can evolve populations of architectures optimize their quality (per- plexity, accuracy, downstream performance), size (number of parameters), and efficiency (inference cache). The basis for this is laid out by the flexibility of the LIV design space, which allows con- structing computational units tailored to these various objectives. STAR leverages evolutionary opti- mization methods search the design space and converge on those solutions performing best under the given set of objectives.
For example, a key mechanism for STAR to reduce parameter counts is to identify which LIV' can be connected through featurizer or feature group sharing without degrading performance. Likewise, STAR reduce parameter counts by purposefully placing MLPs only at those positions of the backbone (as observed in Figs. B.9, 1.10, B.11, B.12), instead of at every other depth index as is otherwise common.
By contrast, STAR can reduce cache size by deliberately placing LIVs with large cache sizes in the backbone and connecting these through featurizer or feature group sharing, while increasing the overall amount of MLPs in the backbone (as observed in Figs. B.27, B.17, and B.23).
B. RECURRING MOTIFS
Feature group sharing in softmax attention When optimizing solely for quality, notable pat- tern in STAR is that the first LIV in the model is typically variant of softmax attention (SA), connected via feature group sharing to other SA-LIVs positioned toward the end of the model (Figs. B. B.3, B.5, and B.7).
Dominance of softmax attention and memoryless LIVs when optimizing for quality Softmax attention and memoryless LIVs are foundational the Transformer architecture. When optimiz- ing for quality, STAR tends to favor these LIV classes (Fig. B.26). Their performance further enhanced by strategically placed recurrences and differential variants of short gated convolutions (Figs. B. B.4, and B.5).
Sparsely placed differential gated convolutions with featurizer sharing Backbones optimized for quality often include two differential variants of short gated convolutions connected through fcaturizer sharing (as illustrated in Figs. B.2, B.4, B.5, B.6, and B.8).
Reduced connectivity when optimizing for quality and size Interestingly, backbones synthe- sized by STAR for both quality and size exhibit significantly fewer LIVs connected through featur- izer and feature group sharing compared those optimized for quality alone (compare Figs. B.26 and 5.4).
Connected gated convolutions recurring motif from the evolutionary process involves LIVs with block-Toeplits token-mixing structure (e.g., convolutions, gated convolutions). In these cases, earlier LIVs in the model are connected through feature group sharing to later LIVs (Figs. B. and B.22).
23

---

Published as a conference paper at ICLR 2025
SA-2
SA-4
Rec-2
GConv-1
GConv-2
GMemless
SA-I-Diff
SA-2-Diff
SA-3-Diff
SA-4-Diff
Ree-1-Dit
Rec-2-Diff
GConv-I-Di
GConv-2-Dif'
Figure B.I: STAR-I I optimised for quality (see Table 5.1). Dashed lines the left indicate feature group sharing while solid lines on the right indicate featurizer sharing.
24

---

Published as a conference paper at ICLR 2025
SA-1
SA-2
SA-3
SA-4
Rec-1
Rec-2
GCony-1
GConv-2
GMemless
SA-1-Diff
SA-2-Diff
SA-3-Diff
Siff-Diff
Rec-I-Diff
Rec-2-Diff
GConv-1-Diff
GConv-2-Diff
Figure B.2: STAR-2 optimised for quality (see Table 5. Dashed lines on the left indicate feature group sharing while solid lines on the right indicate featurizer sharing.
25

---

Published as a conference paper at ICLR 2025
SA-1
SA-2
SA-3
SA-4
Rec-1
Rec-2
GConv-1
GConv-2
GMemless
SA-1-Diff
SA-2-Diff'
SA-3-Diff
SA-4-Diff
Rec-I-Diff
Rec-2-Diff
GConv-1-Did
GConv-2-Diff
Figure B.3: STAR-3 optimised for quality (see Table 5.1). Dashed lines on the left indicate feature group sharing.
26

---

Published as a conference paper at ICLR 2025
SA-1
SA-2
SA-3
SA-4
Rec-1
Rec-2
GCony-1
GConv-2
GMemless
SA-1-Diff
SA-2-Diff
SA-3-Diff
Siff-Diff
Rec-I-Diff
Rec-2-Diff
GConv-1-Diff
GConv-2-Diff
Figure B.4: STAR-4 optimised for quality (see Table 5. Dashed lines the left indicate feature group sharing while solid lines on the right indicate featurizer sharing.
27

---

Published as a conference paper at ICLR 2025
SA-1
SA-2
SA-3
SA-
Rec-1
Rec-2
GConv-1
GConv-2
GMemless
SA-1-Diff
SA-2-Diff
SA-3-Diff
SA-4-Diff
Rec-I-Diff
Rec-2-Diff
GConv-I-Din
GConv-2-Dift
Figure B.5: STAR-5 optimised for quality (see Table A.4). Dashed lines on the left indicate feature group sharing while solid lines the right indicate featurizer sharing.
28

---

Published as a conference paper at ICLR 2025
SA-1
SA-2
SA-3
SA-4
Rec-1
Rec-2
GCony-1
GConv-2
GMemless
SA-1-Diff
SA-2-Diff
SA-3-Diff
Siff-Diff
Rec-I-Diff
Rec-2-Diff
GConv-1-Diff
GConv-2-Diff
Figure B.6: STAR-6 optimised for quality (see Table A.4). Dashed lines on the left indicate feature group sharing while solid lines on the right indicate featurizer sharing.
29

---

Published as a conference paper at ICLR 2025
SA-1
SA-2
SA-3
SA-4
Rec-1
Rec-2
GConv-1
GConv-2
GMemless
SA-1-Diff
SA-2-Diff'
SA-3-Diff
SA-4-Diff
Rec-I-Diff
Rec-2-Diff
GConv-1-Did
GConv-2-Diff
Figure B.7: STAR-7 optimised for quality (see Table A.4). Dashed lines on the left indicate feature group sharing.
30

---

Published as a conference paper at ICLR 2025
SA-1
SA-2
SA-3
SA-4
Rec-1
Rec-2
GCony-1
GConv-2
GMemless
SA-1-Diff
SA-2-Diff
SA-3-Diff
Siff-Diff
Rec-I-Diff
Rec-2-Diff
GConv-1-Diff
GConv-2-Diff
Figure B.8: STAR-8 optimised for quality (see Table A.4). Dashed lines on the left indicate feature group sharing while solid lines on the right indicate featurizer sharing.
31

---

Published as a conference paper at ICLR 2025
SA-1
SA-2
SA-3
SA4
Rec-1
Rec-2
GConv-1
GConv-2
GMemless
SA-1-Diff
SA-2-Diff
SA-3-Diff
SA--DDiff
Rec-1 leifff
Rec-2-Diff
GCony-1-Diff
GConv-2- 2-fov
Figure B.9: STAR-1 optimised for quality and size (see Table 5.1). Dashed lines on the left indicate feature group sharing while solid lines on the right indicate featurizer sharing.
32

---

Published as a conference paper at ICLR 2025
SA-1
SA-2
SA-3
SA-4
Rec-1
Rec-2
GConv-1
GConv-2
GMemless
SA-1-Diff
SA-2-Diff
SA-3-Diff
SA-4-Diff
Rec-1-Diff
Rec-2-Diff
GConv-1-Diif
GConv-2-Diff
Figure B.10: STAR-2 optimised for quality and size (see Table 5.1). Dashed lines the left indicate feature group sharing while solid lines on the right indicate featurizer sharing.
33

---

Published as a conference paper at ICLR 2025
SA-1
SA-2
SA-3
SA-
Rec-1
Rec-2
GConv-1
GConv-2
GMemless
SA-I-Diff
SA-2-Diff
SA-3-Diff
SA-4-Diff
Rec- Rec-I-Diff
Rec-2-Diff
GConv-1-Diff
GConv-2-Dire
Figure B.11: STAR-3 optimised for quality and size (see Table 5. Solid lines on the right indicate featurizer sharing.
34

---

Published as a conference paper at ICLR 2025
SA-1
SA-2
SA-3
SA-
Rec-1
Rec-2
GConv-1
GConv-2
GMemless
SA-1-Diff
SA-2-Diff
SA-3-Diff
SA-4-Diff
Rec-I-Diff
Rec-2-Diff
GConv-1-Diff
GConv-2-Diff
Figure B.12: STAR-4 optimised for quality and size (see Table 5.1). Dashed lines on the left indicate feature group sharing.
35

---

Published as a conference paper at ICLR 2025
SA-1
SA-2
SA-3
SA-4
Rec-1
Rec-2
GCony-1
GConv-2
GMemless
SA-1-Diff
SA-2-Diff
SA-3-Diff
SA-4-Diff
Rec-I-Dif
Rec-2-Diff'
GConv-1- I-Diff
GConv-2-Din
Figure B.13: STAR-5 optimised for quality and size (see Table A.4), Solid lines on the right indicate featurizer sharing.
36

---

Published as a conference paper at ICLR 2025
SA-1
SA-2
SA-3
SA-4
Rec-1
Rec-2
GCony-1
GConv-2
GMemless
SA-1-Diff
SA-2-Diff
SA-3-Diff
SA-4-Diff
Rec-I-Diff
Rec-2-Diff
GConv-1-Diff
GConv-2-Difl
Figure 14: STAR-6 optimised for quality and size (see Table A.4).
37

---

Published as a conference paper at ICLR 2025
SA-I
SA-2
SA-3
SA-4
Rec-1
Rec-2
GConv-1
GConv-2
GMemless
SA-1-Diff
SA-2-Diff
SA-3-Diff
SA-4-Diff
Rec-I-Diff
Ree-2-Diff
GConv-1-Dit
GConv-2-Diff
Figure B.15: STAR-7 optimised for quality and size (see Table A.4). Dashed lines on the left indicate feature group sharing while solid lines on the right indicate featurizer sharing.
38

---

Published as a conference paper at ICLR 2025
SA-1
SA-2
SA-3
SA-4
Rec-1
Rec-2
GCony-1
GConv-2
GMemless
SA-1-Diff
SA-2-Diff
SA-3-Diff
SA-4-Diff
Rec-I-Diff
Rec-2-Diff
GCony-
GConv-2-Diff
Figure 16: STAR-8 optimised for quality and size (see Table A.4).
39

---

Published as a conference paper at ICLR 2025
SA-1
SA-2
SA-3
SA-4
Rec-2
GConv-1
GConv-2
GMemless
SA-1-Diff
SA-2-Diff
SA-3-Diff
SA-4-Diff
Rec-I-Diff'
Rec-2-Diff
GConv-I-Diff
GConv-2-Dii
Figure B.17: STAR-1 optimised for quality and cache (see Table 5.1). Dashed lines on the left indicate feature group sharing.
40

---

Published as a conference paper at ICLR 2025
SA-1
SA-2
SA-3
SA-4
Rec-1
Rec-2
GCony-1
GConv-2
GMemless
SA-1-Diff
SA-2-Diff
SA-3-Diff
SA-4-Diff
Rec-I-Diff
Rec-2-Diff
GCony-
GConv-2-Diff
Figure B.18: STAR-2 optimised for quality and cache (see Table 5.1).
41

---

Published as a conference paper at ICLR 2025
SA-1
SA-2
SA-3
SA-4
Rec-2
GConv-1
GConv-2
GMemless
SA-I-Diff
SA-2-Diff
SA-3-Diff
SA-4-Diff
Rec-I-Diff'
Rec-2-Diff
GConv-I-Diff
GConv-2-Dii
Figure 1919; STAR-3 optimised for quality and cache (see Table 5.1). Dashed lines on the left indicate feature group sharing.
42

---

Published as a conference paper at ICLR 2025
SA-1
SA-2
SA-3
SA-4
Rec-1
Rec-2
GConv-1
GConv-2
GMemless
SA-I-Diff
SA-2-Diff
SA-3-Diff
SA-4-Diff
Rec-I-Diff
Rec-2-Diff"
GConv-1-Diff
GConv-2-Difl
Figure B.20: STAR-4 optimised for quality and cache (see Table 5.1).
43

---

Published as a conference paper at ICLR 2025
SA-1
SA-2
SA-3
SA-4
Rec-1
Rec-2
GConv-1
GConv-2
GMemless
SA-I-Diff
SA-2-Diff
SA-3-Diff
SA-4-Diff
Rec-I-Diff
Rec-2-Diff
GConv-I-Diff
GConv-2-Diff
Figure B.21: STAR-5 optimised for quality and cache (see Table A.4). Dashed lines on the left indicate feature group sharing.
44

---

Published as a conference paper at ICLR 2025
SA-1
SA-2
SA-3
SA-4
Rec-1
Rec-2
GConv-1
GConv-2
GMemless
SA-1-Diff
SA-2-Diff
SA-3-Diff
SA-4-Diff
Rec-1-Diff
Rec-2-Diff
GConv-I-Diff
GConv-2-Diff
Figure B.22: STAR-6 optimised for quality and cache (see Table A.4), Dashed lines on the left indicate feature group sharing.
45

---

Published as a conference paper at ICLR 2025
SA-1
SA-2
SA-3
SA-4
Rec-1
Rec-2
GConv-1
GConv-2
GMemless
SA-1-Diff
SA-2-Diff
SA-3-Diff'
SA-4-Diff
Rec-1-Diff
Rec-2-Diff
GConv-I-Dilf'
GCony-2-Diff
Figure B.23: STAR-7 optimised for quality and cache (see Table A.4). Dashed lines on the left indicate feature group sharing.
46

---

Published as a conference paper at ICLR 2025
SA-1
SA-2
SA-3
SA-4
Rec-2
GConv-1
GConv-2
GMemless
SA-I-Diff
SA-2-Diff
SA-3-Diff
SA-4-Diff
Rec-I-Diff'
Rec-2-Diff
GConv-I-Diff
GConv-2-Dii
Figure B.24: STAR-8 optimised for quality and cache (see Table A.4), Dashed lines the left indicate feature group sharing.
47

---

Published as a conference paper at ICLR 2025
SA
SA-4
Ras2
OCum-1
OCanr
GMemless
SA Dut
SA-2-D0F
SA-4-DLI
Rai--Diil
CCony Dxff
Figure B.25: STAR backbone optimized for quality and cache, consisting of 48 LIVs with a width of 2048 dimensions (see Table 5.2). This backbone generated by duplicating backbone from the STAR evolution for quality and cache and increasing its width from 768 to 2048 dimensions. Dashed lines the left indicate feature group sharing.
48

---

Published as a conference paper at ICLR 2025
Optinizing: Quality
8.0
BA-1
10
SA-2
543
Besl
BA-SDIE
Ste- 1-me
Re
GCavd1DD
GCav mue
#Cuneeted
Stornd
LIV.
Wealns
Shered
Gooupe
Dotaane Cuanded 1.8%
8
10
16
Figure B.26
Carbi (bytes a dl))
Optimizng Quality & Cacbe
Reed
+LE
GCae1
Sh-LDE
DA
Re-2-Di
GGm--1-3
Staned
Prsturion
L.V
Brotine
Grnpp
Didtazo lafe Chanected
LiVes
10
100
15
Figure B.27
49

---

Published as a conference paper at ICLR 2025
Ieeration
Itreation
Breation
Breation
Ioreation
Eoration
Toraaiin
Iomation
K
20
23
30
23
30
20 Brail PPC.
20
10
20
10
Eval PPL
Bhvd POL.
Bivd PeL.
Erai PPL.
Rival P0C.
Bival PPL.
Eval PPC.
Figure B.28
Nleration
Ieraaion
Dremation
Tiention
Rieestion
Donetiin 7
100003
X000
3000
30000
200
300000
120
340
120
140
:28
140
120
540
340
222
140
220
140
120
340
Eval PPL
Eval PPL.
Biv PPC
Evd P9C.
Figure B.29
50

---

Published as a conference paper at ICLR 2025
Optinising fir Quality &e Size (867768)
Deration
Eirration
Itenation 2
Turation
Beration 5
Becstion
Turdaion
Itrrtionn
Esesstion
Rerstio 11
Figure B.30
Optimising Quality (2901256)
Beratton 0
Irecation
Iteration
Inceation 3
Iterotioon
Itematioe:
1.0
0.8
Beraaton 6
Ireeation
Iteration
Ineeation
Ieeration
Ineeation
1.0
Brai PPL
Real PPA
Eval PVL
Sbaal PSL.
18val P95.
Evai P9LL
Figure B.31
Optimizing for Quality & Siac
Tlaradion
Iienation
Iteration 2
Dierstiun
Bunatio
Beralian 5
1.0
0.8
Treestion
Eiseation
Ienation s
Direttion
Itenation 1n
Beration 11
1.0
20
20
20 PPl
Bnd PST.
Eval PoL
Eval PPL
Erdl PPC
Eval
Figure B.32: Note that we used different evaluation datasets for our ablation and main experiments.
51

---

Published as a conference paper at ICLR 2025
Opsimizing for Quality & Size
Wftam Eaboll
Generation
Grorration
Georration
Gevration 3
Geoeration
Grorration 5
Whona Enioll
Georration
Generation
Georration
Gmeration
Generation
Georaaton
Ph Eaboll
Generation
Generation IS
Gevenaion
Geveraiion I5
Gennration
Geoerttio 17
0.2
0.5
10.0
90
9-5 PP1
10.0
0.0
9.5 FFL
10.0
163
0.5 PP1.
10.
9.5
10.0
90
9.3
10.0
PP1
FP1.
FF1.
Figure B.33
Optimizing for Quality
Hhmm (Nea Ea
Generdiio
Generation
Generalion
Ganestion 3
Geoeration
Groerstion 5
Munan Aatal))))
Generation
Geveration
Gemration
Ganeztion
Genaration
Genestion
Genertioo
Generation 18
Geveraion 34
Gearndion 15
Geneaiion 16
Genssation 17
10
10
10
PPL
PPL
PFL
PPL
IPL
PPL
Figure B.34
52

---

Published as a conference paper at ICLR 2025
Optimizing for Quality & Cache
Generation
Generation
Generation
Genrention
Geoeration
Generation
Stap
Generation
Geveration
Generation $
Groreation
Geveration 10
Generatio 11
1.0
10
a.
Genernttion
Geperation
Geneextin 14
Generatin
Geveration
Gonertion 17
(bto 431
5o
86
Qb
4.75
1.00 EPI
9.35
675
9:00
8:35
8.75
3.00 PPC
9.55
6:35
500
935
6.75
$.55
6.75
9.00 PPC.
9:33
PPS
PPL
PEL
Figure B.35
53


Reviews:
[{"review_id": "n12U2lUvMb", "reviewer": "Reviewer_GcDr", "summary": "This work presents an iterative improvement of model architectures to achieve quality-efficiency frontier called STAR. It leverages the theory of  linear input-varying systems to obtain architecture genome encoding and optimize the model through evolutionary algorithms. The experiments reveal optimized models compared to highly optimized transformers and striped hybrid models.", "questions": "See the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "--It shows promising results.\n-The encoding for architecture genome seems to be novel.", "weaknesses": "-The idea of using evolutionary algorithm for the architecture design space exploration  is not new.\n\n-The mutation mainly happens on the backbone and  it is not clear if will affect the underlying components.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RjSo6rHIe2", "reviewer": "Reviewer_hz6S", "summary": "The authors propose a new approach for automated deep learning architecture search based on evolutionary optimization. To realize effective optimization, the work introduces 'linear input-varying systems' (LIVs), a general formulation of input-dependent token vector transformations. With this formalism, a wide range of commonly used model operators such as attention, recurrence, and convolutions can be expressed as special LIV-cases, that compose into model architecture backbones via a set of compositional rules. Notably, the set of LIVs and their composition can be expressed in a hierarchical coding scheme that describes a specific network architecture as a sequence of integers. This 'genome' lends itself to evolutionary optimization via recombination and random mutation of the genome integer elements. Using the LIV genome representation with multi-objective evolutionary algorithms such as NSGA-2, the authors demonstrate that they can automatically discover network architectures that offer improvements over highly optimized Transformer architecture baselines.", "questions": "I might have missed it, but in Table 5.1, what is the difference between STAR-1 to STAR-8? Presumably, the optimization finds a Pareto-front of solutions, but I couldn't find any details on how you picked the evaluated solutions. Could you please provide details on the selection criteria for these eight models and explain how they differ architecturally? This would give readers a clearer picture of the diversity of solutions found by the method.\n\nThe experiments used a fixed number of LIVs to form a backbone. How did you choose this default backbone length of 24 LIVs? Could you discuss the potential challenges and benefits of allowing a variable-length search space with an arbitrary number of LIV modules, and whether this would be something to consider for future work?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The work proposes a straightforward method for constructing a well-conditioned and comprehensive model architecture search space that is amenable to evolutionary optimization under multiple objectives. The hierarchical construction of the LIV building blocks and the coding schemes ensures that a wide variety of candidate architectures can be expressed and searched efficiently. Notably, the genome representation has a clear interpretation which makes it easy to apply constraints to ensure robustness to random edits.\n\nThe paper is well-written and provides a clear presentation of the motivation and context of this work. The experimental evaluation is logically structured and presents the key findings in an accessible way. \n\nThe extensive experimental results provide convincing evidence that the STAR method is effective in finding high-performing architectures under various optimization objectives.", "weaknesses": "By construction, the method appears to be geared towards finding tweaks of transformer-based architecture stacks and does not aim to discover fundamentally new architecture designs. For instance, genomes are constructed assuming a pre-norm residual structure that cannot be varied by the optimizer. This is a reasonable assumption but it arguably relies on the manual design expertise that automated architecture search is ultimately aiming to overcome. Can you think of ways to extend the genome to parametrize the residual connection and normalization schemes?\n\nOn a similar note, the mutation constraints described in Section 4.2 -- while well motivated and reasonable -- suggest that an unconstrained STAR genome is not sufficiently robust to random edits, meaning that expert intuition is still required to 'guide' the mutation process. Can you quantify the impact of unconstrained edits on the search outcomes, e.g. through an ablation study that compares constraint vs unconstrained mutations? \n\nFinally, minor points, I found the notation in Eq. L128 and L138 a little confusing. \n- The underbrace in Eq. L128 suggests that the summation is part of $T_{ij}^{\\alpha \\beta}$, but in Eq. L138 the summation is repeated. \n- Additionally, as noted in L155 $T_{ij}$ is used to denote a whole range of possible operators, so it seems that $T_{ij}^{\\alpha \\beta}$ in L128 is overloaded. It may be clearer to use the underbrace in Eq. L128 to annotate 'attention $T_{ij}$' and 'attention $T^{\\alpha \\beta}$' explicitly as the special case and then use $T_{ij}^{\\alpha \\beta}$ to define the generalized version in Eq. L138.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HdFuSFnZ7O", "reviewer": "Reviewer_uJi5", "summary": "The paper tackles a crucial research problem in the ML community—Optimizing the design of neural network architectures beyond predefined and typical search spaces. The authors posit important research questions on the limitations of existing automated neural network design approaches—mainly relying on predefined backbones and limited search parameters. Also, existing works on NAS methods overlook the inherent hierarchy of the NN design at the level of operators and architecture topology. The authors propose STAR, a novel framework to address the issues above, and build a synthetic, multi-pattern, hierarchical search space-optimized with evolutionary algorithms. A comprehensive evaluation has demonstrated the effectiveness of STAR, showing significant improvements in perplexity, downstream benchmarks, model size, and inference caching compared to optimized hybrid Transformer baselines.", "questions": "- Could you provide additional explanation for certain design choices within the hierarchical search space in Section 2, specifically regarding how they relate to existing implementations and the empirical rationale for their inclusion?\n- Since STAR operates within a hybrid design space that may involve either convolution or attention operations, could you discuss how each design choice applies to different LIV operators, including any constraints involved?\n- Could you elaborate on the encoding strategy used in the evolutionary process, particularly in terms of parameter ranges and the significance of each genome gene?\n- You mention in the abstract and introduction that STAR’s optimization framework is gradient-free, yet some parameters of the explored architectures are trained during the evolutionary process. Could you clarify this point?\n- Could you provide more detailed information on the mutation (e.g., uniform, polynomial) and recombination (e.g., uniform, fixed-point) operators used within STAR?\n- How does the STAR framework handle invalid designs generated through evolution, and could you provide information on the likelihood and extent of these occurrences?\n- Given that STAR operates within a hierarchical design space with a one-genome representation, there are some concerns about optimization convergence. Could you discuss the pros and cons of using a one-shot optimization approach versus a multi-stage optimization strategy that involves separate evolutionary algorithms for different levels of the search space?\n- Without access to the original source code, reproducing STAR is challenging. Would you be open to sharing the source code to allow for further investigation into the technical details of your implementation?\n- The current evaluation seems limited to only two architectures (Striped Mamba and Transformer++). Since this paper aims to explore neural network design beyond typical search spaces, would you consider evaluating STAR models on state-of-the-art architectures that perform well on benchmarks such as RedPajama, LAMBADA, HellaSwag, ARC-e, and PiQA?\n- A comparison between STAR models and other SOTA models, such as LLama, BitNet, and Falcon-Mamba, trained under consistent configurations, could offer valuable insights. Would you consider including performance and efficiency metrics to assess STAR’s effectiveness relative to these models?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper presents an important research problem for the ML design automation community: The limitations of existing search spaces, especially for hybrid and transformer models in language processing applications, which typically require significant computational resources to evaluate different design choices.\n- The hierarchical search space design is grounded in the theory of linear input-varying systems, providing a solid theoretical foundation. Characterizing neural network architectures through featurization, structure, and backbone offers a clear and logical approach, supporting the creation of a more generalizable search space that extends beyond predefined backbones.\n- The authors' choice of an evolutionary algorithm to explore the hierarchical design space is well-justified, given the complexity of the architectures being searched. This approach facilitates the evolution of large populations of architectures using mutation and recombination operators.\n- The evaluation section is thorough and detailed, with promising results that demonstrate the effectiveness of the STAR framework. Overall, the paper is well-written and flows smoothly.", "weaknesses": "- While Section 2 provides a thorough overview of the hierarchical search space, certain design choices would benefit from additional explanation, particularly in relation to existing implementations and empirical justification for their inclusion. For example, the distinction between channel and token mixing could be clarified, along with an explanation of the LIVs to which each can be applied. Since STAR operates within a hybrid design space that may incorporate either convolution or attention mechanisms, it would be helpful to discuss how each design choice applies to different LIV operators and whether any constraints are involved.\n- Section 3 explains the STAR genome and the role of each gene. However, it lacks a detailed description of the encoding strategy used in the evolutionary process, particularly regarding parameter ranges and the significance of each genome gene. For example, it would be helpful to clarify which integers correspond to each LIV class and the specific meaning behind each integer value. Additionally, further explanation is needed on how the evolutionary algorithm processes the genome encoding. To enhance clarity, the authors are encouraged to include a table (in addition to Figure 3.1) listing all search parameters (genes), their definitions, possible values, and how they translate into the neural network architecture.\n- In the abstract and introduction, the authors claim that STAR’s optimization framework is a gradient-free evolutionary algorithm. However, this statement is somewhat misleading, as certain parameters of the explored architectures are indeed trained during the STAR evolution process. To clarify, the authors should provide a detailed schematic of the optimization process, including a clear figure, and add specifics regarding the training and evaluation times within the STAR evolution. This would allow readers to better understand how much STAR improves upon native GA frameworks. For a gradient-free optimization, it would be interesting to see how zero-shot approaches (e.g., ZiCo [1]) can be integrated into the STAR framework to accelerate the fitness evaluation.\n- Although Section 4.1 introduces the evolutionary operators, such as mutation and recombination, these are not explained in sufficient detail. Specifically, the authors do not clarify which types of mutation (e.g., uniform, polynomial) and recombination (e.g., uniform, fixed-point) operators are employed, nor do they discuss how the STAR framework manages invalid designs generated after evolution. Additionally, it would be helpful to understand the likelihood and extent to which STAR produces invalid designs during evolution. To address this important aspect, the authors should consider including a figure or algorithm that illustrates the evolution process, detailing how mutation and recombination are applied and explaining how invalid neural network designs are managed—whether through replacement, alteration, correction, or removal.\n- Given the hierarchical nature of the design space and STAR's apparent use of a single-genome representation, concerns arise regarding optimization convergence. Specifically, what are the pros and cons of a one-shot optimization approach versus a multi-stage optimization strategy, where multiple evolutionary algorithms are employed to optimize different levels of the search space? The authors are encouraged to discuss this comparison and to evaluate STAR's scalability on complex tasks, where achieving convergence toward optimal architectures may be challenging.\n- The evaluation of STAR is somewhat limited, as it is only applied to two architectures (i.e., Striped Mamba and Transformer++). Since this paper aims to push neural network design beyond typical search spaces, it would be beneficial to evaluate STAR models on state-of-the-art (SOTA) architectures that have demonstrated strong performance on benchmarks such as RedPajama, LAMBADA, HellaSwag, ARC-e, and PiQA. A comparison with SOTA models like LLama [2], BitNet [3], and Falcon-Mamba [4] would provide valuable insights. To ensure a fair comparison, all models could be trained under the same conditions, with both performance and efficiency metrics used to assess the effectiveness of STAR’s design approach.\n- Reproducing STAR without access to the original source code proved challenging. Despite my best efforts to replicate the work using the information provided in the appendix, I was unable to do so. To facilitate further investigation into the technical implementation details, it is recommended that the authors share the source code.\n\n**References:**\n- [1]: Zuo, Jingwei, et al. \"Falcon Mamba: The First Competitive Attention-free 7B Language Model.\" arXiv preprint arXiv:2410.05355 (2024).\n- [2]: Touvron, Hugo, et al. \"Llama: Open and efficient foundation language models.\" arXiv preprint arXiv:2302.13971 (2023).\n- [3]: Ma, Shuming, et al. \"The era of 1-bit llms: All large language models are in 1.58 bits.\" arXiv preprint arXiv:2402.17764 (2024).\n- [4]: Zuo, Jingwei, et al. \"Falcon Mamba: The First Competitive Attention-free 7B Language Model.\" arXiv preprint arXiv:2410.05355 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hn9K9c26a4", "reviewer": "Reviewer_mX7E", "summary": "This paper proposes the approach for the synthesis of tailored architectures (STAR). STAR genomes are automatically refined and recombined with gradient-free, evolutionary algorithms to optimize for multiple model quality and efficiency metrics. In the experimental results, the proposed method is compared with Transformer++ and StripedMamba.", "questions": "1.\tHow about the performance compared with other LLMs, such as Llama series?\n2.\tHow about the performance compared with KV cache optimization methods ?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe proposed method combines a novel search space based on the theory of linear input-varying systems, supporting a hierarchical numerical encoding into architecture genomes.\n2.\tThe proposed method STAR optimizes architectures for various combinations of metrics simultaneously: quality (perplexity during pretraining), quality and size, and quality and inference cache size (KV cache and fixed state cache). \n3.\tThe experimental results are well analyzed.", "weaknesses": "The proposed method is only compared with Transformer++ and StripedMamba. Comparison with additional LLMs (such as Llama series) would be preferred.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)