LiteTransformerSearch: Training-free Neural Architecture Search for Efficient Language Models
Authors: Mojan Javaheripi
Gustavo Henrique de Rosa
Subhabrata Mukherjee
Shital Shah
Tomasz Lukasz Religa
Caio Cesar Teodoro Mendes
Sebastien Bubeck
Farinaz Koushanfar
Debadeepta Dey
Venue: NeurIPS 2022 Accept (2022)
Keywords: Neural Architecture Search
AutoML
Transformers

Abstract: The Transformer architecture is ubiquitously used as the building block of largescale autoregressive language models. However, finding architectures with the optimal trade-off between task performance (perplexity) and hardware constraints like peak memory utilization and latency is non-trivial. This is exacerbated by the proliferation of various hardware. We leverage the somewhat surprising empirical observation that the number of decoder parameters in autoregressive Transformers has a high rank correlation with task performance, irrespective of the architecture topology. This observation organically induces a simple Neural Architecture Search (NAS) algorithm that uses decoder parameters as a proxy for perplexity without need for any model training. The search phase of our training-free algorithm, dubbed Lightweight Transformer Search (LTS), can be run directly on target devices since it does not require GPUs. Using on-target device measurements, LTS extracts the Pareto-frontier of perplexity versus any hardware performance cost. We evaluate LTS on diverse devices from ARM CPUs to NVIDIA GPUs and two popular autoregressive Transformer backbones: GPT-2 and Transformer-XL. Results show that the perplexity of 16-layer GPT-2 and Transformer-XL can be achieved with up to 1.5×, 2.5× faster runtime and 1.2×, 2.0× lower peak memory utilization. When evaluated in zero and one-shot settings, LTS Pareto-frontier models achieve higher average accuracy compared to the 350M parameter OPT across 14 tasks, with up to 1.6× lower latency. LTS extracts the Pareto-frontier in under 3 hours while running on a commodity laptop. We effectively remove the carbon footprint of hundreds of GPU hours of training during search, offering a strong simple baseline for future NAS methods in autoregressive language modeling.

Training-free Neural Architecture Search for Efficient Language Models
Mojan Javaheripi', Gustavo H. de Rosa", Subhabrata Mukherjee", Shital Shah', Tomasz L. Religa', Caio C.T. Mendes", Sebastien Bubeck?, Farinaz Koushanfar', Debadeepta Dey
University of California San Diego, Microsooft Research, Microsoft moj mojan@ucsdd edu, dedey@microsoft.co
Abstract
The Transformer architecture is ubiquitously used as the building block of large- scale autoregressive language models. However, finding architectures with the optimal trade-off between task performance (perplexity) and hardware constraints like peak memory utilization and latency is non-trivial. This is exacerbated by the proliferation of various hardware. We leverage the somewhat surprising empirical observation that the number of decoder parameters autoregressive Transformers has a high rank correlation with task performance, irrespective of the architecture topology. This observation organically induces a simple Neural Architecture Search (NAS) algorithm that uses decoder parameters as a proxy for perplexity without need for any model training. The search phase of our training-frres algorithm, dubbed Lightweight Transformer Search (LTS can be directly target devices since does not require GPUs. Using on-target-device measurements, of perplexity versus any hardware performance cost. We evaluate LTS on diverse devices from ARM CPUs to NVIDIA GPUs and two popular autoregressive Transformer backbones: GPT-2 and Transformer-XL Results show that the perplexity of 16-layer GPT-2 and Transformer-X XI. can be achieved with up 1.5x, 2.5x faster runtime and 1.2x, 2.0x lower peak memory utilization. When evaluated in zero and one-shot settings, LTS Pareto-frontier mod- els achieve higher average accuracy compared to the 350M parameter OPT across 14 tasks, with up to 1.6x lower latency. LTS extracts the Pareto-frontic in under 3 hours while running on a commodity laptop. We effectively remove the carbon footprint of hundreds of GPU hours of training during search, offering a strong simple baseline for future NAS methods in autoregressive language modeling.
LTS extracts the
Introduction
The Transformer architecture 12 has been used the de-facto building block of most pre-trained language models like GPT (5) A common problem arises when one tries to create smaller versions of Transformer models for edge or real-time applications (c.g. text prediction) with strict memory and latency constraints: it is not clear what the architectural hyperparameters should be, e.g., number of attention heads, number of layers, embedding dimension, and the inner dimension of the feed forward network, etc. This problem is exacerbated if each Transformer layer is allowed the freedom have different values for these settings. This results in a combinatorial explosion of architectural hyperparameter choices and large heterogeneous search space. For instance, the search space considered this paper consists of over 10 possible architectures.
Neural Architecture Search (NAS) is an organic solution due to its ability to automatically search through candidate models with multiple conflicting objectives like latency vs. task performance. The central challenge in NAS is the prohibitively expensive function evaluation, i.c., evaluating each architecture requires training it the dataset at hand. Thus it often infeasible to evaluate more
code available at http://ppw..cccooooooo.ccccccoca
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Figure 1: High-level overview of LTS. We propose a training- zero-cost proxy for evaluating the val- idation perplexity of candidate architectures. Pareto- frontier search is powered by evolutionary algo- rithms which use the proposed proxy along with real latency and memory measurements on the tar- get hardware to evaluate sampled architectures.
Target Hardware
Search-space
Evalustion
sample
Validation Perplexity
Evolutionary Search
Proxy
Model
Latency and Memory Measurement
Ranking alri
3i1 have emerged as dominant paradigm NAS which combine all possible architectures into a single graph and jointly train them using weight-sharing. Nevertheless, supernet training imposes constraints
than handful of architectures during the search phase. Supernets
the expressiveness of the search space [29]] and is often memory-hunger [52] & 51 as creates large networks during search. Additionally, training supernets is non-trivial as children architectures may interfere with each
other and the ranking between sub- architecturees based on task performance is not preserved 22
We consider different approach by proposing training- free proxy that provides a highly accurate ranking of candidate architectures during NAS without need for costly function evaluation or supernets. Our scope is NAS for efficient autoregressive Transformers used in language modeling. We design a lightweight search method that is target aardware-wward and outputs a gallery of models on the Pareto-fronticr of perplexity versus hardware metries. We term this method Lightweight Transformer Search (LTS). LTS relies on our somewhat surprising observation: the decoder parameter count has high rank correlation with the perplexity of fully trained autoregressive Transformers.
Given a set of autoregressive Transformers, one can accurately rank them using decoder parameter count the proxy for perplexity. Our observations are also well-aligned with the power laws in [2] shown for homogeneous autoregressive Transformers, i.c., when all decoder layers have the same configuration. We provide extensive experiments that establish high rank correlation between perplexity and decoder parameter count for both homogeneous and heterogeneous search spaces.
The above phenomenon coupled with the fact that candidate architecture's 's hardware performance can be measured on the target device leads a training- free search procedure: pick one' 's favorite discrete search algorithm (e.g. evolutionary search), sample candidate architectures from the search space; count their decoder parameters as a proxy for task performance (i.e., perplexity); measure their hardware performance (e.g-, latency and memory) directly on the target device; and progressively create a Pareto-frontie estimate. While we have chosen a reasonable search algorithm in this work, one can plug and play any Pareto-frontier search method such as those in 200
Building upon these insights, Figure T shows a high-level overview of LTS. We design the first training-free Transformer search that is performed entirely on the target (constrained) platform. As such, LTS easily performs a multi -ujti--obeective NAS where several underlying hardware performance metrics, e.g., latency and peak memory utilization, are simultaneously optimized. Using our training- free proxy, we extract the 3-dimensional Pareto-frontier of perplexity versus latency and memory in a record-breaking time of < 3 hours on commodity Intel Core i7 CPU. Notably, LTTS eliminates the carbon footprint from hundreds of GPU hours of training associated with legacy NAS methods.
To corroborate the effectiveness of our proxy, we train over 2900 Transformers on three large language modeling benchmark datasets, namely, WikiText-103 177. One Billion Word 77], and Pile [[7]. We use LTS to search for Pareto-optimal architectural hyperparameters in two popularly used autoregressive Transformer backbones, namely, Transformer-X XL [CC and GPT-2 [32]. We believe decoder parameter count should be regarded as competitive baseline for evaluating Transformer NAS, both in terms of ranking capabilities and easy computation. We open-source our code along with tabular information of our trained models to foster future NAS research on Transformers.
2 Related Work
Here, we discuss literature on automated search for Transformer architectures in the language domain. We refer to extensive surveys on NAS [4] for a broader overview of the field.
Decoder-only Architectures. So et al. 3 search over TensorFlow programs that implement an autoregressive language model via evolutionary search. Since most random sequences of programs
"See 20 for comprehensive treatment of the difficulties of training supernets.

---

either have errors or underperform, the search has to be seeded with the regular Transformer architec- ture, termed "Primer". As opposed to "Primer" which uses large computation to search a general space, we aim to efficiently search the "backhone" of traditional decoder-only Transformers. Addi- tionally, the objective in "Primer" is to find models that train faster. Our objective for NAS, however, is to deliver Pareto-fronties for inference, with respect to perplexity and hardware constraints.
Encoder-only Architectures. Relative decoder-only autoregressive language models, encoder-only architectures like BERT I|| have received much more recent attention from the NAS community. NAS-BERT 50 trains supernet to efficiently search for masked language models (MLMs) which are compressed versions of the standard BERT, Such models can then be used in downstream tasks as standard practice. Similar search with the aim of finding more efficient BERT variants. They find interesting empirical insights into supernet training issues like differing gradients at the same node from different child architectures and different tensors as input and output at every node in the supernet. The authors propose fixes that significantly improve supernet training. Tsai et 41 Yin et al.
Xu et al. [5]] train a supemett conduct architecture
53 Gao et al. [T also conduct
variants of supernet training with the aim of finding more efficient BERT models.
x approach Transformer search Encoder- Encoder-Decoder Related: Applying the well-known DARTS
spaces leads to memory-hungge supernets. To mitigate this issue. Zhao et al. [6]] propose a multi-split reversible network and a memory- -efeorrefffiiient backpropagation algorithm. One of the earliest papers that applied discrete NAS to Transformer search spaces was [36]] which uses a modified form of evolutionary search. Due to the expense of directly performing discrete search on the search space, this work incurs extremely large computation overhead. Follow-up work by [46] uses the Once-For For- All K approach to train a supernet for encoder-decoder architectures used in machine translation. Search is performed on subsamples of the supernet that inherit weights to estimate task accuracy. For each target device, the authors train a small neural network regressor on thousands of architectures to estimate latency. As opposed to using latency estimator, LTS evaluates the latency of each candidate architecture on the target hardware. Notably, by performing the search directly on the target platform, LTS can easily incorporate various hardware performance metrics, e.g., peak memory utilization, for which accurate estimators may not exist. To the best of our knowledge, such holistic integration of multiple hardware metrics in Transformer NAS has not been explored previously.
3 Lightweight Transformer Search
We perform an evolutionary search over candidate architectures to extract models that lie on the Pareto-frontier. In contrast to the vast majority of prior methods that deliver a single architecture from the search space, our search is performed over the entire Pareto, generating architectures with a wide range of latency, peak memory utilization, and perplexity with one round of search. This alleviates the need to repeat the NAS algorithm for each hardware performance constraint.
To evaluate candidate models during the search, LTS uses a training- free proxy for the validation perplexity. By incorporating training-free evaluation metrics, LTS, for the first time, performs the entire search directly on the target (constrained) hardware. Therefore, we can use real mea- surements of hardware performance during the search. Algorithm outlines the iterative process performed in LTS F for finding candidate ar- chitectures in the search space (D), that lie on the 3-dimensional Pareto- Pareto-frontier (IF) of perplexity versus latency and memory. At each iteration, a set of points (F') are sub- sampled from the current Pareto-frontics A new batch of architectures (Sx) ) are then sampled from IF' using evolutionary algo- rithms ( E A-)). The new samples are eval- uated in terms of latency (L), peak memory utilization (JM), and validation perplexity
Algorithm 1: LTS's training-free NAS
Input: Search space D. Wliter Output: Perplexit--ntency--memo Pareto-frontier F C, M,P,P. P, F 0, 0,0, 0,
while N S Aatte do
F' + Subsample(H))
Syy < EA(F"
// hardware profiling
C - L Latenc(SSS))
M - MM Memory(Sw)
1/ estimate perplexity
(P). Latency and memory are measured directly on the target hardware while the perplexity is indirectly estimated using our accurate and training-free ffee proxy methods.
P t Prox((x))
// update the Pareto-frontice
F + L=LowerComeeHHll(PP..CC.
'The freeto-rrontie search method in Algorithm is inspired by 1 and 21. Other possibilities include
variations proposed in (20). evaluation of which is orthogonal to contributions in this work.
3

---

Finally, the prreto-fronterio is recalibrated using the lower convex hull of all sampled architectures. In the context of multi-objeetive NAS, Pareto-frontier points are those where no single metric (e.g.. perplexity, latency, and memory) can be improved without degrading at least one other metric [20]. To satisfy application-specific needs, optional upper bounds can be placed on the latency and/or memory of sampled architectures during search.
Search Space. Figure 2 shows all elastic parameters in LTS search space, namely, number of layers (Mayer), number of attention heads (Mhead), decoder output dimension (dmodel), inner dimension of the feed forward network (dimner), embedding dimension (demted), and the division factor (k) of adaptive embedding 3. These architectural parameters are compatible with popularly used autoregressive Transformer backbones, e.g., GPT. For prelimi- naries on autoregressive Transformers, please see Appendix A We adopt heterogeneous search space where the backbone parameters are decided on a per-layer basis. This is in contrast to the homogeneous structure commonly used in Trans- formers IC 1 which reuses the same configu- ration for all layers. Compared to homogeneous models, the flexibility associated with heteroge- Figure 2: Elastic parameters in LTS search space. neous architectures enables them to obtain much better hardware performance under the same perplexity budget (see Section 4.4)
Mayer
Dhead
Dirad
modddel
mmoddl
danner
dunner
Adaptive Embedding
Decoder Layer
Decoder
Layer
Heterogeneous search space was previously explored in 146. However, due to the underlying supernet structure, not all design parameters can change freely. As an example, the dimensionality of the Q. K, V vectors inside the encoder and decoder layers is fixed to large value of 512 to accommodate inheritance from the supernet. Our search space, however, allows exploration of all internal dimensions without constraints. By not relying on the supernet structure, our search space easily encapsulates various Transformer backbones with different configurations of the input/output embedding layers and elastic internal dimensions.
LTS searches over the following values for the architectural parameters in our backbones: nlayer E { 16) 1 E {128..... 1024]64), dinner {256, .... 4096 64], and Ahead {2, 4, 8). Additionally we explore adaptive input embedding 3 with dembed E (128, 256, 512} and factor (1, Once a dmooell is sampled, we adjust the lower bound of the above range for dimer to 2xdmndee-- Encoding this heuristic inside the search ensures that the acquired models will not suffer from training collapse. Our heterogeneous search space encapsulates more than 10 different architectures. Such high dimensionality further validates the critical need for training-free NAS.
3.1 Training-free Architecture Ranking
Recently, Abdelfattah et al. I utilize the summation of pruning scores over all model weights the ranking proxy for Convolutional Neural Networks (CNNs), where a higher score corresponds to higher architecture rank in the search space. White et al. [48] analyze these and more recent proxies and find that particular proxy performs consistently well over various tasks and baselines, while parameter and floating point operations (FLOPS) count proxies are quite competitive. However, they did not include Transformer-based search spaces in their analysis. To the best of our knowledge, low-cost (pruning-based proxies have not been evaluated on Transformer search spaces in the language domain. Note that one cannot naively apply these proxies to language models. Specifically, since the embedding layer in Transformers is equivalent to a lookup operation, special care must be taken omit this layer from the proxy computation. Using this insight, we perform the first systematic study of low-cost proxies for NAS on autoregressive Transformers for text prediction.
Low-cost Ranking Proxies.
We leverage various pruning metrics, namely, grad norm, snip 23 grasp
fisher 40
and synflow 38). We also study jacob_cov 26 and relu_log det P which are low-cost scoring mechanisms proposed for NAS on CNNs in vision tasks. While these low-cost techniques do not perform model training, they require forward and backward passes over the architecture to compute the proxy, which can be time-consuming on low-end hardware. Additionally, the aforesaid pruning techniques, by definition, incorporate the final softmax projection layer in their score assessment. Such an approach seems reasonable for CNNs dealing with few classification labels,
WWe use the notation ( (Vmin..... Vsplstep size] show the valid range of values.

---

WikiText-103
LM1B
WikiText-103
LM1B
Ground-truth Ranking
700
Ground-truth Ranking
150
150
100
50
50
SRC 0.97
SRC = 0.99
SRC = 0.97
SRC 0.99
100
200
Proxy Ranking
Proxy Ranking
Proxy Ranking
Proxy Ranking
Transformer-XL
GPT-2
Figure 3: Our training-free zero-cost proxy based decoder parameter count is highly correlated with the (ground truth) validation perplexity after full training. Each plot contains 200 architectures sampled randomly from the search space of Transformer-XL or GPT-2 backbone.
however, it can skew the evaluation for autoregressive Transformers dealing with large output vocabulary space. overcome these shortcomings, we introduce a zero-cost architecture ranking strategy in the next section that outperforms the proposed low-cost proxies in terms of ranking precision, is data free, and does not perform any forward/backward propagation.
Decoder Parameter Count as Proxy. We empirically establish strong correlation between the parameter count of decoder layers and final model performance in terms of validation perplexity. We evaluate 200 architectures sampled uniformly at random from the search space of two autoregressive Transformer backbones, namely. Transformer-Xe and GPT-2. These architectures are trained fully on WikiTeex- 103 and One Billion Word (LM1B) datasets, which consumes over 25000 GPU-hours on NVIDIA A100 and V100 nodes. We compare the ranking obtained using decoder parameter count proxy and the ground truth ranking after full training in Figure 3 On WikiTexxt-109 1033, zero-cost ranking using the decoder parameter count obtains a Spearman's 's Rank Correlation (SRC) of 0.97 with full training. SRC further increases to 0.99 for the more complex LM1B benchmark on both backhones. This validates that the decoder parameter count is strongly correlated with final model performance, thereby providing a reliable training-free frree proxy for NAS.
4 Experiments
We conduct experiments to seek answers to the following critical questions:
How well can training-free ffree proxies perform compared to training- bbsed methods for estimating the performance of Transformer models?
How does model topology affect the performance of the proposed decoder parameter proxy?
Can our training-free decoder parameter count proxy be integrated inside search algorithm to estimate the frontirrr How accurate is such an estimation of the Pareto?
Which models are the Pareto-frontier of perplexity, latency, and memory for different hardware?
How well do LTS models perform in zero and one-shot settings compared to hand-designed variants when evaluated on downstream tasks?
We empirically answer questions further address question 3 in Appendix where we show the Pareto-frontier models extracted by the decoder parameter count proxy are very close the ground truth of 0.6% perplexity difference. Additionally, we show the efficacy of the decoder parameter count proxy when performing search on different ranges of model sizes Appendix C Figure 12
and in Sections 4.2
and 4.5 respectively. We
with an average
4.1 Experimental Setup
Please refer to Appendix B for information about the benchmarked datasets, along with details of our training and evaluation setup, hyperparameter optimization, and evolutionary search algorithm.
Backbones. We apply our search on two widely used autoregressive Transformer backbones. namely, Transformer-XL [10] and GPT-2 [32 that are trained from scratch with varying architectural hyperparameters. The internal structure of these backbones are quite similar, containing decoder blocks with attention and feed-forward layers. The difference between the backbones lies mainly in their dataflow structure; the Transformer-X XI. backbone adopts a recurrence methodology over past states coupled with relative positional encoding which enables modeling longer term dependencies.
5

---

Topk 10 %
Topk 30 %
Topk 50 %
1.00
1.0
1.0
0.75
0.8
CR-Partis) Treining
CR-Partis Training
CR-Partial Trsining SRC-Partisl Treining De-dddeeeefaaammm
0.50
SRC-Partel Training
CRDDeccdessarams
0.6
SRC--ariial Tiiining
0.6
0.25
SRC-DecoderPraang
0.4
SRC-DecoderParans
0.4
SRC-DecodePParems
2500
5000
7500
2500
5000
7500
2500
5000
7500
Time (s)
Time (s)
Time (s)
Figure 4: Comparison between partial training and our zero-cost proxy, i.e., decoder parameter count, in terms of ranking performance and timing overhead. Each subplot corresponds to a topk% of the randomly sampled models, based on their validation perplexity after full training.
Performance Criteria. To evaluate the ranking performance of various proxies, we first establish ground truth ranking of candidate architectures by training them until full convergence. This ground truth ranking is then utilized to compute two performance criteria as follows:
Common Ratio (CR): We define CR as the percentage overlap between the ground truth ranking of architectures versus the ranking obtained from the proxy. CR quantifies the ability of the proxy ranking to identify the topk% architectures based on their validation perplexity after full training.
Spearman's 's Rank Correlation (SRC): We use this metric to measure the correlation between the proxy ranking and the ground truth. Ideally, the proxy ranking should have high correlation with the ground truth over the entire search space as well as high- performiffog candidate models.
4.2 How do training-free proxies perform compared to training- based methods?
In this section, we benchmark several proxy methods for estimating the rank candidate architectures. Specifically, we investigate three different ranking techniques, namely, partial training, low-cost methods, and number of decoder parameters.
Partial Training. We first analyze the relationship between validation perplexity after a shortened training period versus that of full training for ranking candidate models. We stop the training after T E [1.25% 87 755 of the total training iterations needed for model convergence. Figure demonstrates the SRC and CR of partial training with various rS, evaluated on 100 randomly selected models from the Transformer-X XL backbone, trained on WikiText-1033 The horizontal axis denotes the average time required for J iterations of training across all sampled models. Intuitively, higher number of training iterations results in a more accurate estimate of the final perplexity. Nevertheless, the increased wall-clock time prohibits training during search and also imposes the need for GPUs. Interestingly, very few training iterations, i.e., 1.25%, provide very good proxy for final performance with an SRC of >> 0.9 on the entire population. Our training-free free proxy, i.e., decoder parameter count, also shows competitive SRC compared to partial training.
Low-cost Proxies. We benchmark various low-cost methods introduced Section B.1 on 200 randomly sampled architectures from the Transformer-XL backbone, trained on WikiText- 103. Figure 5 shows the SRC between low-cost proxies and the ground truth ranking after full training.
We measure the cost of each proxy in terms of FLOPs. As seen, the evaluated low-cost proxies have a strong correlation with the ground truth ranking (even the lowest perform- ing relu_log.e.. det has > 0.8 SRC), validating the effectiveness training-free NAS on autore- gressive Transformers. The lower performance of relu_log_ det can be attributed to the much higher frequency of ReLU activations in CNNs, for which the method was originally developed, architectures.
5pcarman's Correlation
snip
0.95
grad norm
fisher
0.90
jacob cov
* grasp
relu log det
synflew
DecoderParams
0.80
FLOPs
lel1
Figure 5: SRC between low-cost proxies and the ground truth ranking after full training of 200 ran- domly sampled Transformers. The decoder param- eter count obtains the best SRC with zero cost.
compared to Transformer-basee Our analysis of randomly selected models with homogeneous structures also shows strong cor- relation between the low-cost proxies and validation perplexity, with decoder parameter count outperforming other proxies (see Appendix D)
Parameter Count. Figure Th demonstrates the final validation perplexity versus the total number of model parameters for 200 randomly sampled architectures from GPT-2 and Transformer-Xi

---

Transformer-XL
Validctoon PRL
DecaderParams
TotalParams SRC- TotalParams
CR Decoderarams --- CR TotalParams
SRC DecoderParams
DecoderParams
SRC TotaiParama
1.0
2.g
0.8
Vallidetion PPL
GPT-2
Am
0.6
0.8
0.4
0.2
0.6
Transformer-XL
GPT-2
20
2 Total nTarams
50
100
30
70
80
Ze7
Topk (%)
Topk (%)
(a)
(b)
Figure 6: (a) Validation perplexity after full training versus total parameters for 200 randomly sampled architectures trained on WikiText-103. The clear downward trend suggests a strong correlation between parameter count and perplexity. (b), (c) Performance of parameter count proxies for ranking the randomly sampled architectures with Transformer-XL and GPT-2 backbones.
backbones. This figure contains two important observations: (1) the validation perplexity has downward trend as the number of parameters increases, (2) The discontinuity is caused by the dominance of embedding parameters when moving to the small Transformer regime. We highlight several example points in Figure Gja where the architectures are nearly identical but the adaptive input embedding factor i is changed. Changing & E (1, 4] (shown with different colors in Figure (a)) varies the total parameter count without much influence on the validation perplexity.
The above observations motivate us to evaluate two proxies, i.e., total number of parameters and decoder parameter count. Figures 5 and 6 demonstrate the CR and SRC metrics evaluated on the 200 randomly sampled models divided into topk% bins based on their validation perplexity. As shown, the total number of parameters generally has a lower SRC with the validation perplexity, compared to decoder parameter count. This due the masking effect of embedding parameters, particularly in the Transformer-XL backbone. The total number of decoder parameters, however, provides highly accurate, zero-cost proxy with an SRC of 0.97 with the perplexity over all models, after full training. We further show the high correlation between decoder parameter count and validation perplexity for Transformer architectures with homogeneous decoder blocks in the supplementary material, Appendix D While our main focus is on autoregressive, decoder-only, Transformers, we provide preliminary results on the ranking performance parameter count proxies for encoder-only and encoder-decoder Transformers in Appendix
4.3
How does variation in model topology affect decoder parameter count as a proxy?
The low-cost proxies introduced in Section 3.1 rely on forward and backward passes through the network. As such, they automatically capture the topology of the underlying architecture via the dataflow. The decoder parameter count proxy, however, is appology-agnosii investigate the effect of topology on the performance of decoder parameter count proxy. Specifically, we seek to answer whether for a given decoder parameter count budget, the aspect ratio of the architecture, i.e., trading off the width versus the depth, can affect the final validation perplexity.
this section, we
We define the aspect ratio of the architecture as dmodel (=width), divided by nlayer (=depth). This metric provides a sense of how skewed the topology is and has been used in prior works which study scaling laws for language models [ For a given decoder parameter count budget, we generate several random architectures from the GPT-2 backbone with wide range of the width- te-thp aspect ratiosfl The generated models span wide, shallow topologies (e.g.. =1m6 Dlyer=2) to narrow, deep topologies (e.g., 112 =100) Figure 7a| shows the validation perplexity of said architectures after full training on WikiText-103 versus their aspect ratio. The maximum deviation (from the median) of the validation perplexity is < 12.8% for a given decoder parameter count, across wide range of aspect ratios E [1, 630). Our findings on the heterogeneous search space complement the empirical results by 22 where decoder parameter count largely determines perplexity for homogeneous Transformer architectures, irrespective of shape (see Figure [22]
'ee control the aspect ratio by changing the width, i.c., dundel while keeping = dmadel and D The number of layers is then derived such that the total parameter count remains the same.
7

---

DecoderParams--30
Validition PPL
45
DecoderParams-
123
140
120
183
200
220
240
240
150
200
230
300
350
400
400
500
250
40
PecoderParoms-225
25M
Validation PPL
Validation PPL
Decoder Params- SCM
33-
150
230
300
350
402
520
100
000
100
25-
20
10'
10 DindedDiager
10a
550
750
400
550
1200
1200
Latency (ms)
Memory (MB)
(a)
(b)
(c)
Figure Validation perplexity after full training versus the (a) width-to-depth aspect ratio, (b) latency, and (c) peak memory utilizzation. Models are randomly generated from the GPT-2 backbone and trained on WikiText-103, For a given decoder parameter count, we observe low variation in perplexity across different models, regardless of their topology. The topology, however, significantly affects the latency (up to 2.8x) and peak memory utilization (up to 5.5x) for models with the same perplexity.
We observe stable training when scaling models from the GPT-2 backbone up to 100 layers, with the perplexity increasing only when the aspect ratio nears 1. Nevertheless, such deep models are not part of our search space as they have high latency and are unsuitable for lightweight inference. For the purposes of hardware-awar and efficient Transformer NAS, decoder parameter count proxy holds a very high correlation with validation perplexity, regardless of the architecture topology as shown in Figure 7a We further validate the effect of topology on decoder parameter count proxy for the Transformer-X XL backbone in Figure 14 of Appendix E Our results demonstrate less than 7% deviation (from the median) in validation perplexity for different aspect ratios E [8, 323].
Note that while models with the same parameter count have very similar validation perplexities, the topology in fact affects their hardware performance, i.e., latency (up to 2.8x) and peak memory This motivates the need for incorporating
utilization (up to 5.5x), as shown in Figures 7b and 7c
hardware metrics in NAS to find the best topology.
4.4 Pareto-frontier models for various hardware platforms
We run LTS on different target hardware and obtain range of Pareto-optimal architectures with vari- ous latency/memorry/prlleee. to search models that are lightweight while ensuring nearly on-par validation perplexity with non-adaptive input embedding. As the baseline Pareto, we benchmark the Transformer-X -L. (base) and GPT-2 (small) models with homogeneous layers E [1, 16). This because the straightforward way to produce architectures of different latency/memory is varying the number of layers (layer- We compare our NAS -AS-generated architectures with layer-scaled backbones and achieve better validation perplexity and/or lower latency and peak memory utilization. All baseline and -AS-generated models are trained using the same setup enclosed in Table 2 of Appendix B
characteristics. During search, we fix the adaptive input embedding
factor
scaling) 22 [0O)
architectures found by LTS versus the layer-scaled baseline. Here, all models are trained on the LMIB dataset (See Figure in Appendix for results on WikiText- 103). Note that the Pareto-frontier search is performed in 3-dimensional space, an example of which enclosed in Appendix F Figure 15 For better visualization, in Figure X we plot 2 2-dimensional slices of the Pareto-frontier frontier with validation perplexity on the y-axis and one hardware performance metric (either latency or memory) on the x-axis.
Figure g shows the
As seen, in the low-latency regime, LTS consistently finds models that have significantly lower perplexity compared naive scaling of the baseline Transformer-Xl. GPT-2. On the Transformer- XI. backbone, LTS finds architectures with an average of 19.8% and 28.8% lower latency and memory, while achieving similar perplexity compared the baseline on ARM CPU. Specifically. the perplexity of the 16-layer Transformer-Xy XL base can be replicated on the ARM device with a lightweight model that is 1.6x faster and utilizes 1.9x less memory during execution. On the Corei7 CPU, the Pareto-frontier models found by LTS are on average 25.8% faster and consume 30.0% less memory under the same validation perplexity constraint. In this setting, LTS finds a model that replicates the perplexity of the 16- 16-layer Transformer-X base while achieving 1.7% faster runtime and 1.9x less peak memory utilization. The savings are even higher on the GPU device, where
'The best reported result in the literature for GPT-2 or Transformer-xx XL might be different based on the specific training hyperparameters, which is orthogonal to our investigation.
8

---

Transtormer-XL Backbone
GPT-2 Backbone
ARM CIPU
LTS
Scaled Transformer
UTS
Scaled Transformer
Velidatton ove
11.5
LTS
Scaled Transfomerr
80
7.6
Scaled Transoomer
LT5
ddetonn
-50
50
1,69
1.6%
60
00
13x
50-
50
1000
2000
3000
4000
100
200
300
2000
4000
6000
200
400
600
Corel7 CPU
valiozton PRL
60
8.9
Transformer
60
8.9
Scaled Transformer
Velidation
(136
70
Scaled Transformer
(6.8%)
LTS
Transfomer
50
15k
60
13k
00
40
40
300
500
200
300
200
800
300
400
500
600
TITAN Xp GPU
Velitetion
60
Scaled Transfommer
<celed Transformer
Volldatton DR.
70-
80-1
7700
LTS
Scaled Transformer
70
80
16.5
D'
Scaled Transformer
50
2.58
40
40
1:28
50-
50
1000
600
Latency (ms)
Memory (MB)
Latency (ms)
Memory (MB)
Figure 8: 2D visualization of the perplexity versus latency and memory Pareto-frontier found by LTS, versus the scaled backbone models with varying number of layers, trained on the LMIB dataset. Architectural parameters for models shown here are detailed in Appendix
the NAS MAS-generated models achieve the same perplexity as the baseline with average 30.5% lower latency and 27.0% less memory. Specifically, LTS model with the same perplexity as the 16-layer Transformer-XI. base has 2.5x lower latency and consumes 2.0x less peak memory on TITAN Xp.
On the GPT-2 backbone, NAS HAS-generated models consume average 11.8% less memory while achieving the same validation perplexity and latency on an ARM CPU. The benefits are larger on Corei7 and TitanXP where the latency savings are 13.8% and 11.9%, respectively. The peak memory utilization also decreases by 9.7% and 12.9%. on average, compared to the baseline GPT-2s on Corei7 and TITAN Xp. Notably, NAS finds new architectures with the same perplexity as the 16-layer GPT-2 with 1.3x, x.5x faster runtime and 1.2x lower memory utilization on Corei7 and TITAN Xp.
Our heterogeneous search space allows us to find a better parameter distribution among decoder layers. Therefore, LTS delivers architectures with better performance in terms perplexity, while reducing both latency and memory when compared to the homogeneous baselines. We provide the architecture of all baseline and LTS models shown Figure in Tables of Appendix !ا
Search Efficiency. The main component in LTS search time is the latency/peak memory utilization measurement for candidate architectures since evaluating the model perplexity is instant using the decoder parameter count. Therefore, our search finishes in a few hours on commodity hardware, e.g.. taking only 0.9, 2.6, and 17.2 hours on TITAN Xp GPU, Corei7 CPU, and an ARM core, respectively. To provide more context into the timing analysis, full training of even one 16-layer Transformer-Xi XL base model on LMIB using a machine with 8x NVIDIA V100 GPUs takes 15.8 hours. Once the Pareto-frontier models are identified, the user can pick a model based on their desired hardware constraints and fully train on the target dataset. LTS an alternate paradigm to that of training large supernets; our search can run directly on the target device and GPUs are only needed for training the final chosen Pareto-frontier model after search.
In Table we study the ranking performance of partial training (500 steps) versus the decoder parameter count proxy for evaluating 1200 architectures from the Transformer-XL. backbone during LTS search. Astonishingly the decoder parameter count proxy gets higher SRC compared to partial training, while effectively removing training from the inner loop of search for NAS.
Train
GPU
COse
Table 1: Ranking abilities of full and partial
SRC
Iter
Hours
(lbs)
training versus our proxy for 1200 models sam-
Full Training
40,000
19,024
5433
1.0
pled during LTS search. Training time is re-
Partial Training
500
231
0.92
ported for Wiki'eex- 103 and NVIDIA V100
5,000
2690
768
0.96
GPU. Decoder parameter count proxy obtains
# Decoder Params
0
0.98
an SRC of 0.98 using zero compute.

---

Figure 9: Average and one-shot curacy obtained by LTS models (dots) and the baseline OPT-350M (triangle) across 14 NLP tasks. Latency is measured on A6000 NVIDIA GPU. Architectural parameters for all models shown here are detailed in Appendix
Avg Accuracy (%)
zero-shot
one-shot
39
30
39
150
200 Latency (ms)
225
150
200 Latency (ms)
225
4.5
Zero and one-shot performance comparison of LTS models with OPT
Zhang et al. [58] open-source a set of pre-trained decoder-only language models, called OPT, which can be used for zero or few-shot inference on various NLP tasks. Below, we compare the performance of LTS Pareto-frontier models with the hand-designed OPT architecture in zero and one-shot settings.
We LTS to search for language models with GPT-2 backbone which have 300M to 500M total parameters to compare with the 350M parameter OPT. Our search space is detailed in Appendix H The search is conducted with latency as the target hardware metric and decoder parameter count as a proxy for perplexity. Once the search concludes, We train 20 models from the Pareto-frontier along with OPT-350M on 28B tokens from the Pile [17]. The pretrained models are then evaluated on 14 downstream NLP tasks, namely, HellaSwag 57 PIQA 4 ARC (easy and challenge) 19]. OpenBookQA (28), WinoGrande 34. and SuperGLUE
benchmarks BooiQ, CB, COPA, WIC, WSC, MultiRC, RTE, and ReCoRD. The training hyperparameters and the evaluation setup are outlined Appendix B Figure 2 shows the overall average accuracy obtained across all tasks versus the inference latency for LTS models and the baseline OPT. As shown, NAS-generated models achieve a higher average accuracy with lower latency compared to the hand-designed OPT-350M model. We provide per-task breakdown of zero and one-shot accuracy in Appendix H Figure 77
4
Zero-shot Performance. Figure 17a demonstrates the zero-shot accuracy obtained by LTS and OPT-350M on the benchmarked tasks. Compared to the OPT-350M architecture, LTS finds models that achieve higher accuracy and lower latency in the zero-shot setting on all evaluated downstream tasks. Specifically, the maximum achievable accuracy of our NAS generated models is 0.2 - 8.6% higher than OPT-350M with an average speedup of 1.2x. If latency is prioritized, LTS delivers models which are, on average, 1.5x faster and up to 4.6% more accurate than OPT-350M.
One-shot Performance. Similar trends can be observed for one-shot evaluation shown for different tasks in Figure 17b LTT Pareto-frontier models improve the per-task accuracy of OPT-350M on 12 out of 14 tasks by 0.1 8.0%. while achieving an average speedup of 1.2x. On the same tasks, LTS Pareto-frontier includes models that enjoy to 1.6x speedup over OPT-350M with an average 1.5% higher accuracy. On the RTE task, the best LTS model has 0.4% lower accuracy but 1.6x faster runtime. On the WSC task, the best performing LTS model obtains a similar one-shot accuracy as OPT-350M, but with 1.5x faster runtime.
5 Limitations and Future Work
Decoder parameter count provides a simple yet accurate proxy for ranking autoregressive Trans- formers. This should serve as a strong baseline for future works on Transformer NAS. Our focus is mainly on autoregressive, decoder-only transformers. We therefore, study perplexity as the com- monly used metric for language modeling tasks. Nevertheless, recent literature on scaling laws for Transformers suggest similar correlation between parameter count and task metrics may exist for encoder only (BERT-style) Transformers or encoder-decoder models used in neural machine translation (NMT) 19. Additionally, recent findings (39 show specific scaling laws exist between model size and downstream task metrics, e.g., GLUE
Inspired by these observations, we provide preliminary studies that suggest parameter count proxies may be applicable to Transformers in other domains. Detailed investigations of such zero-cost proxies for NAS on heterogeneous BERT-style or NMT models with new performance metries is an important future avenue of research.
Acknowledgements
Professor Farinaz Koushanfar's effort has been in parts supported by the NSF TILOS Al institute, award number CCF-2112665.
10

---

References
[1] Mohamed S Abdelfattah, Abhinav Mchrotra, Fukasz Dudziak, and Nicholas Donald I. Lane. Zero-cost proxies for lightweight nas. In International Conference on Learning Representutions, 2020.
[2] Meta AI. Codebase for open pre-trained transformers. me taseq
httpes///ttaabbccm//ccooooererre
[3] Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. In International Conference Learning Reprexentations, 2018.
[4] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqu: Reasoning about physical commonsense natural language. Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439, 2020.
[5] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
[6] Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. In International Conference on Learning Representations, 2019.
(7) Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Kochn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling, arXiv preprint arXiv:1312.3000 2013.
[8] Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12270-12280, 2021.
[9] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1803.05457, 2018.
[10] Zihang Dai, Zhilin Yang. Yiming Yang. Jaime G Carbonell, Quoe Le, and Ruslan Salakhutdinov. Transformer-xi xl: Attentive language models beyond fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pages 2978-2988, 2019.
[11] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In NAACL-HLL (I), 2019.
[12] Xuanyi Dong and Yi Yang. Nas-bench-201 Extending the scope of reproducible neural architecture search. In International Conference on Learning Representations. 2019.
[13] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Efficient multi-objective neural architecture search via lamarckian evolution. In International Conference Learning Representations, 2019.
[14] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: survey, 2019.
[15] Hugging Face. Openai gpt2 by hugging face. model, doc/ ggpt/
[16] Jiahui Gao, Hang Xu, Han shi, Xiaozhe Ren, Philip L. H. Yu, Xiaodan Liang, Xin Jiang, and Zhenguo Li. Autobert-zero: Evolving bert backbone from scratch, 2021.
[17] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.000 00027, 2020.
[18] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeftrey Hsu, Kyle McDonell, Niklas Mucnnighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. framework for few-shot language model evaluation. https September 2021.
//gthub.. http://aacccoooeeeeeecaeaaaeeeee
[19] Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. In International Conference on Learning Representations, 2021.
11

---

[20] Julia Guerrero- Viu, Sven Hauns, Sergio Izquierdo, Guilherme Miotto, Simon Schrodi, Andre Biedenkapp. Thomas Elsken, Difan Deng, Marius Lindauer. and Frank Hutter. Bag of baselines for multi-objecttvve joint neural architecture search and hyperparameter optimization. arXiv preprint arXiv:2105.01015. 2021.
[21]
Hanzhang Hu, John Langford, Rich Caruana, Saurajit Mukherjee, Fric J Horvitz, and Debadeepta Dey. Efficient forward architecture search. In H. Wallach, H. Larochelle, A. Beygelzimer. F. d' d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL http://e////ocengggs.//.. wttp:/pacc/c/ppppe/////aaa. ///l////pppe/////fllee///e 6c468ec5a41d658 2eee 1008a7951 Paper pdf
[22] Jared Kaplan, Sam McCandlish, Tom Henighan. Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.088 00000 2020.
[23] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. SNIP: SINGLE-SHOT NETWORK PRUNING BASED ON CONNECTION SENSITIVITY. In International Conference on Learning Representations, 2019. URL htttps://operreviee net/ wttp://orummii-iiigiiea
[24] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search, In
International Conference on Learning Representations, 2019. URL BtLLittts://oppeeevieet wew. forumm
Ed=S1eYHoC5FX
[25] Joc Mellor, Jack Turner, Amos Storkey, and Elliot J Crowley. Neural architecture search without training. In International Conference on Machine Learning, pages 7588-7598, PMLR, 2021.
[26] Joseph Mellor, Jack Turner, Amos Storkey, and Elliot J. Crowley. Neural architecture search without training, 2021. URL httppp/s/ppppaarrreeeeeeee////rp
[27] Stephen Merity, Caiming Xiong, James Bradhury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint (xXiv:600.07843 2016.
[28] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint 0rXiv:1809.007788 2018
[29] Xuefci Ning, Changeheng Tang, Wenshuo Li, Zixuan Zhou, Shuang Liang, Huazhong Yang, and Yu Wang. Evaluating efficient performance estimators of neural architectures. Advances in Neural Information Processing Systems, 34, 2021.
[30] NVIDIA. Transformer-xl for pytorch.
htttp:://githu..comm/ com/ cwtp://VWDD///DDeeeeeeaaaaaaaaaa tree/ httph//twee/yoooe////////aaaaaaa
XL.
[31] Hieu Pham. Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International Conference on Machine Learning, pages 4095-4104, PMLR, 2018.
[32] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)99, 2019.
[33] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1-67, 2020.
[34] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
[35] Julien Siems, Lucas Zimmer, Arber Zela, Jovita Lukasik, Margret Keuper, and Frank Hutter. Nas-bench bench- and the case for surrogate benchmarks for neural architecture search. arXiv preprint arXiv:2008.0977 097777 2020.
[36] David R. So, Chen Liang, and Quoc V. Le. The evolved transformer, 2019.
[37] David R. So, Wojeiech Manke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V. Lc. Primer: Searching for efficient transformers for language modeling, 2021.
[38] Hidenori Tanaka, Daniel Kunin, Daniel Yamins, and Surya Ganguli. Pruning neural networks without any data by itcratively conserving synaptic flow. Advances in Neural Information Processing Systems, 33, 2020.
[39] Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang. Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale efficiently: Insights from pre- pre-training and fine- tne-tuning transformers. arXiv preprint arXiv:2109.10668 2021.
12

---

[40] Lucas Theis, Iryna Korshunova, Alykhan Tejani, and Ferenc Huszár. Faster gaze prediction with dense networks and fisher pruning, arXiv preprint arXiv: 0tfl:::8010005758 2018.
[41] Henry Tsai, Jayden Ooi, Chun- Sung Femg, Hyung Won Chung. and Jason Riesa. Finding fast transformers: One-shot neural architecture search by component composition, 2020.
[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances in neural information processing systems, pages 5998-6669 6008, 2017.
[43] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 1804.07461, 2018.
[44] Alex Wang. Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for parral-p language understanding systems. Advances neural information processing systems, 32, 2019.
[45] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. International Conference on Learning Representations, 2020. URLlattps://ppenrviiie het/forummid-SSkga..o
[46] Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and Song Han. Hat: Hardware-aware transformers for efficient natural language processing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7675-7688, 2020.
[47] Hongyu Wang, Shuming Ma, Li Dong. Shaohan Huang, Dongdong Zhang, and Furu Wei. Deepnet: Scaling transformers to 1,000 layers. arXiv preprint arXiv:2203.00533 2022.
[48] Colin White, Mikhail Khodak, Renbo Tu, Shital Shah, Séhastien Bubeck, and Debadeepta Dey. A deeper look at zero-cost proxies for lightweight nas. In ICLR Blog Track, 2022. URL hctp:::/icclr--loo-tteeee wwwlooootrcc.iiiiiiiie
github.io/ httbbb/o///22////////eeeeee 2eeo cost---rreessot-toe..eee.oo proxies/.
[49] Martin Wistuba, Ambrish Rawat, and Tejaswini Pedapati. A survey on neural architecture search, 2019.
[50] Jin Xu, Xu Tan, Renqian Luo, Kaitao Song. Jian Li, Tao Qin, and Tic-Yan Liu. Nas-hert, Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, Aug 2021. doi: 10. 1145/ 3447548.3467266
URL htRLbhtp://ddxccom doi org/ 10. 01114//347544 3467262
[51] Jin Xu, Xu Tan, Kaitao Song, Rengian Luo, Yichong Leng, Tao Qin, Tie- Yan Liu, and Jian Li. Analyzing and mitigating interference in neural architecture search, 2021.
[52] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong, Pe-darts: Partial channel connections for memory-efficient architecture search. In International Conference on Learning Representations. 2019.
[53] Yichun Yin, Cheng Chen, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Autotinybert: Automatic hyper-paramete optimization for efficient pre-trained language models, arXiv preprint arXiv:2107. 13686, 2021.
[54] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv: 00962. 2019.
[55] Ofir Zafrir, Ariel Larey, Guy Boudoukh, Haihao Shen, and Moshe Wasserblat. Prune once for all: Sparse pre-trained language models. arXiv preprint arXiv:2111.051 2021.
[56] Arber Zela, Julien Niklas Siems, Lucas Zimmer, Jovita Lukasik, Margret Keuper, and Frank Hutter. Surro- gate nas benchmarks: Going beyond the limited search spaces of tabular nas benehmarks. In International Conference on Learning Representations, 2021.
[57] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint 1rXiv:1905.07880 2019.
[58] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre- pre-trained transformer language models. arXiv preprint arXhv:2005.0106. 2022.
13

---

[59] Xuan Zhang and Kevin Duh. Reproducible and efficient benchmarks for hyperparameter optimization of neural machine translation systems. htttps://iithb.. com Ette//etteee.ommrrrreeeeee
[60] Xuan Zhang and Kevin Duh, Reproducible and efficient benchmarks for hyperparameter optimization of neural machine translation systems. Transactions of the Association for Computational Linguistics, 8: 393-408, 2020.
[61] Yuekai Zhao, Li Dong, Yelong Shen, Zhihua Zhang, Furu Wei, and Weizhu Chen. Memory- efficeee differentiable transformer architecture search, 2021.
[62] Qinqin Zhou, Kekai Sheng, Xiawu Zheng, Ke Li, Xing Sun, Yonghong Tian, Jie Chen, and Rongrong Ji. Training-fri free transformer architecture scarch. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10894-10903, 10903, 2022.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sections 4.3 and Ap-
pendix E
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Appendix K
(d) Have you read the ethics review guidelines and ensured that your paper conforms them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material as URL)? [Yes] The source code for LTS is available at
trree/neuris___e htaiiipssssssrrccaaiaaaaa aashaa/chh///p
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix B
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Appendix B
(b) Did you mention the license of the assets? [N/A] All used code are open-source and publicly available
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "tXOSt2M8Sp", "reviewer": "Reviewer_rzmB", "summary": "The paper studies neural architecture search over autoregressive transformer models, and proposes a training-free approach to search the Pareto frontier of multiple objectives: latency, memory and perplexity. The proposed approach is based on an important finding that the perplexity is highly correlated with the number of parameters in the decoder, which offers a cheap and convenient way of model evaluation.\n", "questions": "Please address the comments above.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper proposes a training-free NAS approach to discover the Pareto front of auto-regressive transformer architectures, which can significantly reduce the searching cost in designing efficient models.\n- The authors provide numerous empirical results to justify the training-free proxy, together with its relation with the latency and memory of target devices.\n- Well written and easy to follow.\n\nWeakness:\nThere are still several concerns with the finding that the perplexity is highly correlated with the number of decoder parameters. \n- According to Figure 4, the correlation decreases as top-10% architectures are chosen instead of top-100%, which indicates that the training-free proxy is less accurate for parameter-heavy decoders. \n- The range of sampled architectures should also affect the correlation. For instance, once the sampled architectures are of similar sizes, it could be more challenging to differentiate their perplexity and thus the correlation can be lower.\n\n\nDetailed Comments:\n1. Some questions regarding Figure 4: 1) there is a drop of correlation after a short period of training, which goes up with more training iterations; 2) the title \"Top-x%\" should be further explained;\n2. Though the proposed approach yields the Pareto frontier of perplexity, latency and memory, is there any systematic way to choose a single architecture given the target perplexity?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "dej9EATlypY", "reviewer": "Reviewer_Zj8C", "summary": "This paper tackles the problem of efficient deployment of autoregressive transformer models in resource constrained environment. Finding architectures that can solve the dual problem of latency and accuracy in such environments can be difficult. This problem gets exacerbated when one tries to deploy across multiple hardware platforms. NAS, via its SuperNet methodology,  has been proposed to manage this problem. However, SuperNet training is expensive and can lead to large memory requirement. Additionally, training supernets is non-trivial as subnets can interfere with each other and rank correlation between sub architectures is not preserved. Non SuperNet approach to NAS generally have an expensive architecture evaluation phase where accuracy on a training data is measured, which makes it difficult to use in practice.\n\nThis paper makes the following contribution in this problem domain -\n1. The paper finds the surprising correlation that for autoregressive transformer model, the number of decoder parameters is a good proxy for validation perplexity. This allows for a very simple architecture evaluation phase.\n2. Building on this insight, they develop a lightweight transformer search algorithm that combines evolutionary search with this simple accuracy proxy and real runtime measurement on hardware to develop a framework that provides a better accuracy-runtime pareto frontier than the traditional method of layer scaling. ", "questions": "Questions - \n\nCan the authors comment on Weakness (1), (2) and (3)?", "limitations": "Have the authors adequately addressed the limitations and potential negative societal impact of their work? Yes", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths - \n1. Comparison with a large number of simple accuracy proxy methods from literature\n2. Ablation studies like the one to understand the impact of topology of the model on accuracy and how it impacts the proposed proxy method for accuracy\n3. The proposed solution can easily adapt to any hardware platform, while not losing its efficiency\n \n\nWeaknesses -\n1. The entire paper hinges on the idea that parameter count is a good proxy for validation perplexity. However, the validation of this claim is not done cleanly. The authors rely on training results on wikitext-103,  an extremely small dataset. Additionally, the use the same training hyperaparameter for different models to verify their claim for parameter count serving as a proxy for validation perplexity. Autoregressive models are trained on far larger datasets[2][3]. Infact, training on small datasets can have a significant impact on validation perplexity [2]. Additionally, dataset is expected to scale with size of the model.  Finally, [1] also shows that using the same training hyperparameters for models at different scale can lead to poor accuracy. These facts indicate that this claim is based on weak experimentation methodology. As a result, it becomes unclear if a more appropriate experiment method will lead to the same conclusion.\n\n2. Additionally, using validation metric as a measure of quality of an autoregressive model is also questionable. Autoregressive models have shown immense benefit in generative, zero shot and few shot/prompt tuned tasks. Additionally, these are the tasks for which these models are being currently deployed in the industry (see cohere.ai, OpenAI, AI21 APIs for these models). Given that these tasks represent the way trained autoregressive models are consumed in the industry, measuring accuracy based on some of these tasks would make sense. Its unclear whether validation perplexity is a good proxy for these tasks or not. This further makes it difficult to understand the impact of this work and the usefulness of the result.\n\n3. Their claims on impact of topology and relevance of the proxy metric for iso-parameter, different width/depth factor models is also suffering from similar methodology problem. As you make models deeper and thinner, initialization becomes extremely important to get good perplexity score. As a result, if the init of parameters is not explored, accuracy will suffer, making the claims harder to justify.\n\n[1] https://www.microsoft.com/en-us/research/blog/%C2%B5transfer-a-technique-for-hyperparameter-tuning-of-enormous-neural-networks/\n\n[2] https://arxiv.org/abs/2203.15556\n\n[3] https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "HHI_BFNCl0i", "reviewer": "Reviewer_zV4x", "summary": "This paper proposes to use the number of decoder parameters as a training-free metric for autoregressive transformer architecture search for language models. The paper then shows that the metric correlates strongly with the final perplexity of the model over the entire model sizes and topologies. Because of the training-free nature of the proxy proposed, the authors show that using otherwise standard search algorithms such as evolutionary search, it is possible to optimise for architectures that trade off perplexity, memory and latency on devices with modest computational power directly, and it is possible to recover much better Pareto frontier using the method proposed compared to the naive scaling of baseline architectures. \n", "questions": "Please see Strengths And Weaknesses", "limitations": "The authors have adequately discussed the limitations of their work both in Conclusion and in appendix. The authors should also be given credit for interleaving several broader impact discussions in the paper (rather than as a standalone section), such as the estimation of both GPU hours and $CO_2$ emission statistics in Table 1.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall I believe this is a sound paper with potential for real impact, and the writing and presentation are clear and good throughout. I have outlined my detailed comments on strengths and weaknesses below.  Note that I have combined this section with my questions (next section) as many of them are related, and given the time constraint for the rebuttal, I do not expect the authors to respond thoroughly to all of my questions and concerns especially where additional experiments are necessary; in these cases some qualitative response would suffice.\n\n\n## Strengths\n\nThe algorithm proposed is simple and effective in the search spaces investigated by the authors: the number of decoder parameters is very cheap to obtain, and is arguably cheaper than many of the previous zero- or low-cost proxies proposed in the NAS literature (also considered by the authors in Fig 5). The ability to directly search for better architectures on target devices with computational constraints as a result of the lowered computational cost can lead to very tangible benefits, so this paper has potential to be very impactful.\n\n\n## Weaknesses & questions\n\n1. A concern I have is that the fact that performance (perplexity in the NLP context) is correlated with some measure of #params is highly expected in general and is also demonstrated to be the case in NAS in particular (see the results of the size search space of NATS-Bench, an extension of NAS-Bench-201 [1]). In many cases, some measure of #params has been used as a search objective alongside performance [2] (not in this case, as the authors used mem usage and latency); this shows that the correlation between that and performance is assumed. While the paper does show some nuance by demonstrating the superiority of #decode params over the total #params (Fig 6), I am still a bit unconvinced by the fact that most of the analysis (Sec 4.1 - 4.3) use the absolute performance as the outcome variable, even though ultimately we are interested in the *trade-off* between performance and #params by finding a family the architectures that maximise performance at different part of the Pareto frontier instead of just maximising performance in Sec 4.4. It is worth noting that the existing zero-cost proxies the authors compared like synflow and jacob_norm were largely originally evaluated in cell-based search spaces where we often only aim to find a single architecture with its accuracy optimised [3,4], and that’s why it is more justified to use the absolute performance as the dependent variable in that case.  \n\t\n      Given this goal of finding Pareto-optimal family of architectures, I feel that analyses should instead pivot on the predictive ability of the # decoder params after the effect of other confounding variables are removed: for example, how well does #decoder param predicts perplexity on architectures with similar mem usage and/or latency (using a metric such as the partial correlation)? I feel this is a more direct test of how well the proposed proxy can truly help discover more Pareto-optimal architectures. While the authors did discuss these briefly in Appendix C, I believe these analyses are rather important and should be expanded: for example, 1) the paper considers both latency and mem_usage as optimization objectives, but App C only considers latency; 2) similar to how NAS predictor can often predict well over the entire search space but break down in high-accuracy regions (which are also what we care about), the proxy proposed might also suffer degradations nearer to the Pareto frontier and it would be interesting to investigate whether that is the case and 3) the results in App C do show that the proxy quality is worse at larger models, and it could be interesting to explain/investigate why it might be the case and discuss potential remedies.\n\n    On a related note, I feel it would also be good for the authors to include some explanations or intuitions on why do they think why # decoder params is a great proxy in this case. I feel that this could potentially be inspirational for future works that would like to develop on the idea but, for example, on a different search space or domain of application. It is also helpful in this particular case as this is the main innovation (the search techniques and other part of the algorithms are otherwise standard).\n\n2. Experiments: Sec 4.4 is where the main experiments are demonstrated to show promise of the method. My main concern is that while the results are promising, the baselines seem a bit weak that the authors simply scale the base, homogenous Transformer and GPT models. I am wondering whether it is possible also to compare against a single or a family of NAS architectures from, for example, one or more of the prior works the authors cited in Related Works?\n\n3. Compatibility/sensitivity with varying training hyperparameters: One thing also noted by the authors is that the best Transformers is often obtained by also tuning the training hyperparameters in addition to the architectural hyperparamaters (footnote 2). One interesting thing to investigate is whether the the suggested proxy still predict performance well when there are changes in hyperparameters. \n\n4. Related work: there is also a recent work that proposes training-free transformer search proxy [5]. While I recognise that since this paper is only published at CVPR 2022 it is concurrent to the current submission (and my rating is independent of this paper), the authors are encouraged to include some qualitative discussions & comparisons w.r.t this work.\n\n5. Train code: the authors have not provided code for the paper and mention they will open source the model upon acceptance. Would you please also clarify are you going to open-source the code? I think it would greatly enhance the reproducibility & benefit the broader community more if both code and models will be released.\n\n\n## References\n\n[1] Dong, X., Liu, L., Musial, K., & Gabrys, B. (2021). Nats-bench: Benchmarking nas algorithms for architecture topology and size. IEEE transactions on pattern analysis and machine intelligence.\n\n[2] Chen, M., Peng, H., Fu, J., & Ling, H. (2021). Autoformer: Searching transformers for visual recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 12270-12280).\n\n[3] Abdelfattah, M. S., Mehrotra, A., Dudziak, Ł., & Lane, N. D. (2021). Zero-cost proxies for lightweight NAS. ICLR.\n\n[4] White, C., Zela, A., Ru, R., Liu, Y., & Hutter, F. (2021). How powerful are performance predictors in neural architecture search?. Advances in Neural Information Processing Systems, 34, 28454-28469.\n\n[5] Zhou, Q., Sheng, K., Zheng, X., Li, K., Sun, X., Tian, Y., ... & Ji, R. (2022). Training-free Transformer Architecture Search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10894-10903).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0bpNWbTG4CX", "reviewer": "Reviewer_KSpx", "summary": "This paper adapts the number of decoder parameters as the search proxy to accelerate the design loop for decoder-only language models. The authors verified the assumption using 2800 trained data points  and based on the assumption they show 1.6-2.5x faster runtime speed and 1.3-2.0x lower peak memory on various hardware. ", "questions": "In Figure 3, WikiText-103 is a relative small dataset compared to standard GPT-2 level model trainign (e.g., WikiText of billions of tokens). If the dataset size becomes larger, will the precision of #num_of_parameters proxy become better or worse?\n\nIn Figure 8, the authors show PPL on evaluation. However, in Algorithm 1 there is no description about how the model is trained. Is every model trained separately, or there is parameter sharing techiniques?", "limitations": "Overall this is a good paper with clear method and solid experiment results. If the authors can address the aformentioned weakness above, I would like to improve the score. ", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n- The paper tackles on an important problem. \n- The problem formulation is clear and writing is well presented.\n- Evaluation in Figure 3 shows the correlation between #num parameters and final model performance.\n- The pareto-frontier exploration demonstrates significant improvements compared with naive transformer scaling.\n\n\nWeakness \n\n- The biggest assumption highly relies on decoder-only architecture. L377: Our focus is entirely on autoregressive, decoder-only transformer. This is not a general solution to explore NAS on tasks like NMT.\n- Only compared with scaled transformers, which is not considered as strong baseline. Since this is NAS work, the latency and accuracy from models derived with AutoML techniques should be discussed.\n- Free proxy does not mean free evaluation — the generated architecture still need to perform full training to obtain the model weights. Then even the search cost is reduced, the total cost will not be saved much compared to once-for-all studies like [1, 2].\n\n1. Hat: Hardware-aware transformers for efficient natural language processing.\n2. Prune Once for All: Sparse Pre-Trained Language Models", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept