LLM Performance Predictors are good initializers for Architecture Search (2024)

Abstract: In this work, we utilize Large Language Models (LLMs) for a novel use case: constructing Performance Predictors (PP) that estimate the performance of specific deep neural network architectures on downstream tasks. We create PP prompts for LLMs, comprising (i) role descriptions, (ii) instructions for the LLM, (iii) hyperparameter definitions, and (iv) demonstrations presenting sample architectures with efficiency metrics and ‘training from scratch’ performance. In machine translation (MT) tasks, GPT-4 with our PP prompts (LLM-PP) achieves a SoTA mean absolute error and a slight degradation in rank correlation coefficient compared to baseline predictors. Additionally, we demonstrate that predictions from LLM-PP can be distilled to a compact regression model (LLM-Distill-PP), which surprisingly retains much of the performance of LLM-PP. This presents a cost-effective alternative for resource-intensive performance estimation. Specifically, for Neural Architecture Search (NAS), we introduce a HybridSearch algorithm (HS-NAS) employing LLMDistill-PP for the initial search stages and reverting to the baseline predictor later. HSNAS performs similarly to SoTA NAS, reducing search hours by approximately 50%, and in some cases, improving latency, GFLOPs, and model size. The code can be found at: https://github.com/UBC-NLP/llmas.

heading: 1 Introduction
text: Large language models (LLMs) have diverse applications, encompassing both open-ended tasks (e.g., brainstorming and chat) and closed-ended tasks (e.g., summarization and question answering). This study explores a unique application of LLMs: constructing a performance predictor (LLM-PP) for a deep neural network (DNN) architecture. The predictor takes the DNN architecture description,
*Work was done while Ganesh was at UBC.
typically hyperparameters (e.g., #layers, #attention heads), as input and predicts the performance (e.g., BLEU score) for a specific downstream task. The aim is to create a performance predictor with low prediction errors compared to training from scratch. The hypothesis is that LLMs possess a ‘general understanding’ of DNN architectures, derived from relevant training data like DNN research papers and GitHub repositories. The main objective of this work is to leverage this understanding to design accurate, efficient, and broadly applicable performance predictors, beneficial for tasks like neural architecture search (NAS).
How to design an accurate performance predictor (PP)? To answer this, we create PP prompts precisely specifying the task. These prompts include: (i) role: high-level description of the assigned LLM role, (ii) instructions: detailed task instructions (e.g., downstream task, architecture, performance/efficiency metric) for the LLM to follow, (iii) hyperparameters: definitions of architecturespecific hyperparameters, and (iv) demonstrations: supervised examples for the PP task with architecture descriptions and performance metrics (e.g., BLEU score). Using GPT-4 (OpenAI, 2023a) as our primary LLM and WMT datasets for machine translation (MT) tasks, we find that GPT-4 with our PP prompts (LLM-PP) predicts architecture performance with a mean absolute error achieving the state-of-the-art (SoTA) and a slightly lower rank correlation coefficient compared to previous SoTA weight-sharing supernet-based performance predictors (Wang et al., 2020; Jawahar et al., 2023b).
Using GPT-4 for LLM-PP entails utilizing the GPT-4 API to score each architecture, rendering LLM-PP prohibitively expensive for various use cases. One example is NAS, where PP evaluates approximately 3,000 candidate architectures for each constraint (e.g., latency ≤ 100ms) (Wang et al., 2020). As of August 2023, GPT-4 pricing is 0.03$
10540
per 1K tokens 1. Assuming PP prompts consume about one-third of 1K tokens, the estimated cost is approximately ∼30$ for a single constraint on the target hardware. With varying constraint values (e.g., 100ms, 200ms), constraint types (e.g., latency, FLOPs, memory), and target hardware (e.g., Nvidia A100, Raspberry Pi), the cumulative cost can quickly become exorbitant (e.g., 1,800$).
How to design cost-effective PP? To answer this, we distill LLM-PP performance predictions into a tiny MLP model (LLM-Distill-PP) using architecture descriptions (e.g., hyperparameter lists) as input features. Surprisingly, LLM-Distill-PP can significantly maintain the performance of LLM-PP. Assuming LLM-Distill-PP needs only 3,000 examples, the estimated cost is approximately ∼30$ for a single downstream task, amortized across various constraint values, types, and target hardware.
Can LLM-Distill-PP speed up architecture search while preserving the efficiency and the quality of SoTA NAS? To answer this, we apply using LLM-Distill-PP as the PP to design efficient MT architectures via SoTA NAS methods like HAT (Wang et al., 2020). We introduce the HybridSearch algorithm (HS-NAS), where LLM-DistillPP serves as the PP for the first 15 search iterations, and a weight-sharing supernet (SoTA performance predictor) takes over for the remaining 15 iterations. HS-NAS achieves roughly 50% faster search than SoTA NAS, maintaining or improving on the performance of architectures designed by SoTA NAS. In some cases, it also yields reduced latency (∼2%), FLOPs (∼1%), and model size (∼2%).
Our main contributions are as follows:
1. We propose LLM-PP, leveraging few-shot prompting of LLM for accurate performance predictors, achieving SoTA mean absolute error.
2. We introduce LLM-Distill-PP, with a better amortized cost than LLM-PP, suitable for PPheavy use cases.
3. HS-NAS, a search algorithm, reduces NAS search time by half compared to SoTA, identifying more efficient architectures by leveraging LLM-Distill-PP and SoTA performance estimators.
4. We provide prompts, training and evaluation data for LLM-Distill-PP models, and code
1https://openai.com/pricing
with detailed reproducibility instructions.

heading: 2 Related Work
text: Performance Predictors. In NLP, a common approach to construct performance predictors is training a weight-sharing supernet model, jointly training various architectures by sharing weights with the largest model in the search space (Wang et al., 2020; Xu et al., 2022a; Jawahar et al., 2023a,b). During each training step, an architecture is randomly selected from the search space, and its corresponding weights are extracted from the largest model’s weight matrices. These weights are then trained for the target task. Post-training, architecture performance is predicted by extracting the relevant weights and evaluating on the validation set. Key challenges in supernet training include weight co-adaptation (Bender et al., 2018; Zhao et al., 2021), capacity bottleneck (Jawahar et al., 2023b), and gradient conflict (Gong et al., 2021). NAS for NLP. NAS is a general framework for designing efficient NLP architectures meeting userdefined constraints across various dimensions: (i) architecture family (encoder-only (Yin et al., 2021; Xu et al., 2022a, 2021, 2022b), decoder-only (Javaheripi et al., 2022), encoder-decoder (Wang et al., 2020; Jawahar et al., 2023a,b) without limiting to Transformers), (ii) constraint types (latency, FLOPs, model size), and (iii) tasks (task-agnostic pretraining (Xu et al., 2022a; Javaheripi et al., 2022; Jawahar et al., 2023b), task-specific training (Wang et al., 2020; Jawahar et al., 2023a)). The evolutionary search-based algorithm employs a performance predictor to identify high-quality architectures, utilizing real or predicted efficiency metrics to discard those not meeting specified constraints. LLMs for NAS. GENIUS (Zheng et al., 2023), a recent search algorithm for image classification, uses LLMs to generate convolution-based architectures. However, it trains these candidates from scratch, incurring high practical costs. Contrasting with our approach, (i) GENIUS uses LLMs to generate architectures, while we use LLMs to predict their performance, (ii) the search cost for our work is upper bounded by SoTA NAS for MT (∼ 5 NVIDIA V100 hours), much more efficient than GENIUS (∼ 960 NVIDIA V100 hours), and (iii) we focus on Transformer-based encoder-decoder architectures for machine translation. For more on the synergy between LLMs and AutoML, see Tornede et al. (2023).
Additional background on related topics such as LLMs and distillation can be found in A.1.

heading: 3 Performance Prediction Problem
text: Informally, the performance prediction problem entails providing a DNN architecture description (usually hyperparameters like #layers, #attention heads) to the predictor, which then outputs the performance (e.g., BLEU score) for a specified downstream task. An ideal predictor should minimize prediction errors compared to the performance achieved through training from scratch. Formally, let T represent a downstream task, AT its search space of architectures, and YT ⊂ R the real space of performance scores. Define DT as the data distribution over AT × YT . The performance predictor is denoted by fT : AT → YT . The labeled test set LtestT = {(ai, pi)}mi=1 ∼ (D)mT comprises architecture, performance pairs drawn i.i.d. from DT . pi is the performance obtained by training the architecture ai from scratch to convergence on task T (known as ‘training from scratch’ (TFS) performance).
The performance predictor’s quality is assessed using two metrics: Mean Absolute Error (MAE) calculates the mean absolute difference between predictions and their corresponding TFS performances, formalized as
∑ (ai,pi)∼(D)T |fT (ai)−pi| |(D)T | .
Kendall rank correlation coefficient is another metric that computes the ranking correlation between a set of predictions and their corresponding TFS performances, formalized as KendallTau([fT (a1), . . . , fT (am)], [p1, . . . , pm]). Examples for these metrics are discussed in Section A.2. Recently, Jawahar et al. (2023b) emphasized the importance of both MAE and Kendall-Tau metrics in evaluating performance predictor quality. For instance, a predictor with a 38% better MAE and a 12% worse Kendall-Tau, compared to a base predictor, led NAS to find an architecture with a 4% BLEU improvement. Conversely, a predictor with a 5% worse MAE and a 6% higher Kendall-Tau resulted in a NAS architecture with a 0.1% BLEU improvement. Hence, better MAE and better KendallTau are positively correlated with higher-quality architecture.

heading: 4 Baseline Performance Predictors
text: In NAS for NLP literature, the SoTA method for constructing performance predictors (fT ) involves training a weight-sharing supernet model on task
T . Simply put, a weight-sharing supernet model is the largest model in the search space, capable of parameterizing all architectures via weight sharing. The parameters for a specific architecture are obtained by extracting the relevant rows and columns from the supernet model’s weight matrix. Typically, the supernet is trained by iteratively sampling an architecture from the search space and training the extracted weights for that architecture. Formalizing the supernet’s training objective: Denote the training data distribution as Xtrain. Represent the training sample and label as x and y, where x, y ∼ Xtrain. arand is a uniformly sampled architecture from the search space AT . alarge and asmall denote the largest and smallest architectures inAT . The subnet with architecture a is denoted by sa, parameterized by the supernet model weights W . The training objective of the supernet using sandwich sampling (Yu et al., 2020) is given by
min W
Ex,y∼Xtrain [Earand∼A[L(sarand(x;W ), y)]
+ L(salarge(x;W ), y) + L(sasmall(x;W ), y)]. Hardware-aware Transformers (Wang et al., 2020) employs single-path one-shot (SPOS) optimization (Guo, Zichao and Zhang, Xiangyu and Mu, Haoyuan and Heng, Wen and Liu, Zechun and Wei, Yichen and Sun, Jian, 2020), focusing on optimizing only arand at each training step. Mixture-ofSupernets (Jawahar et al., 2023b) (MoS) utilizes mixture-of-experts (MoE) (William Fedus and Barret Zoph and Noam Shazeer, 2022) to enhance the supernet’s capacity, with the router specializing weights for each architecture. MoS comes in two variants: layer-wise MoS and neuron-wise MoS, differing in the degree of freedom for weight generation. Both variants of MoS employ sandwich sampling for supernet training.

heading: 5 LLM Performance Predictor (LLM-PP)
text: LLM demonstrates a “general understanding" of DNN architectures, likely acquired through training on relevant data sources like research papers and GitHub repositories. Testing these architecture understanding capabilities involves prompting LLM to generate hyperparameter definitions and design principles for architecture search (Zheng et al., 2023). These LLM capabilities contribute to effective performance prediction by aiding the mapping of DNN architectures to their performances.
To this end, we propose the LLM-based Performance Predictor (LLM-PP), which involves
prompting an LLM to generate performance predictions for DNN architectures. The prompts, referred to as PP prompts, must be meticulously designed to precisely convey the performance prediction task to the LLM. Illustrated in Figure 1, PP prompts break down the task into four main components: role, instructions, hyperparameters, and demonstrations, followed by the test architecture.
The role specifies the LLM’s role, describing the downstream task (e.g., machine translation) and the performance metric (e.g., BLEU). The instructions provide five detailed instructions covering the downstream task, DNN architecture, and model efficiency metrics. The first two focus on the task specifics, specifying the task type (e.g., machine translation), dataset (e.g., WMT’14 En-De), performance metric (e.g., BLEU), and inputs/outputs (e.g., source/target language) for ntask examples from the dataset. The third instruction details the DNN architecture, including backbone (e.g., Transformer), type (e.g., encoder-decoder), and a reference to the original DNN paper. The fourth instruction outlines efficiency metrics details (e.g.,
GFLOPs), included in the demonstrations. The final instruction directs the LLM to consider complex relationships between architecture configuration, performance, and efficiency metric. The third component, hyperparameters, defines architecturespecific hyperparameters. Demonstrations is the final component containing narch supervised examples, each representing an architecture from the search space with hyperparameter values, efficiency score, and TFS performance score. The design process of the LLM-PP prompt is discussed in A.4.

heading: 5.1 Evaluation Setup
text: Downstream tasks. We utilize established research (Wang et al., 2020; Jawahar et al., 2023a,b) and opt for popular machine translation (MT) benchmarks: WMT’14 En-De, WMT’14 En-Fr, and WMT’19 En-De. Detailed statistics of these benchmarks are available in A.6.1. Our chosen performance metric is BLEU (Papineni et al., 2002). DNN architecture. We adopt the Transformerbased Encoder-Decoder architecture (Vaswani et al., 2017). The implementation, training settings, and search space (A) mirror Wang et al. (2020), detailed in A.6.2. Our evaluation dataset (TFS-Eval) is sourced from Jawahar et al. (2023b), featuring 30 architectures with their TFS performance scores for each WMT dataset. FLOPs, latency, and model size computations for architectures are done using the implementation from Wang et al. (2020). Performance predictors. Baseline performance predictors include: (i) HAT (Wang et al., 2020), (ii) Supernet (Sandwich) (Jawahar et al., 2023b) (HAT, with sandwich sampling instead of SPOS), (iii) Layer-wise MoS (Jawahar et al., 2023b), and (iv) Neuron-wise MoS (Jawahar et al., 2023b). We build three LLM-PP variants, utilizing Mistral (Jiang et al., 2023) (Mistral-7B-Instruct-v0.1), ChatGPT (OpenAI, 2023b) (GPT-3.5-turbo, June version), and GPT-4 (OpenAI, 2023a) (June version). For PP prompts, we randomly sample: (i) 5 examples (ntask = 5) from the downstream task for the second instruction and (ii) 10 examples (ntask = 10) from TFS-eval for the demonstrations component. The remaining 20 examples from TFS-eval will be used for reporting the predictor quality. For all predictors, we repeat the experiments with three different seeds and report the average MAE and Kendall-Tau between the predictor performance and the TFS performance.

heading: 5.2 Results
text: LLM-PP predictions closely align with TFS performance scores compared to the baselines. Figure 2 illustrates the TFS versus performance predictor validation BLEU for different WMT benchmarks. The diagonal line (red line) represents the perfect predictor, where the predicted performance exactly matches the TFS score. The predictions from the supernet-based predictors (i.e., all nonLLM-based ones) are consistently underestimates of the TFS performance for all architectures across three benchmarks. In contrast, LLM-PP predictions are largely closer to the diagonal line, showcasing the high accuracy of LLM-PP.
LLM-PP achieves SoTA MAE, slightly trailing baselines in Kendall-Tau. Table 1 displays the
MAE and Kendall-Tau of baseline and LLM-PP predictors. Neuron-wise MoS stands out as the best baseline on average across datasets, boasting the lowest MAE and highest Kendall-Tau score. LLMPP Mistral outperforms supernet-based baselines in MAE for WMT’14 En-De and WMT’14 EnFr tasks. LLM-PP ChatGPT and LLM-PP GPT-4 surpass Neuron-wise MoS in MAE, with LLM-PP GPT-4 achieving the SoTA MAE score. However, LLM-PP slightly lags behind baselines in KendallTau. In A.5, we examine the histogram of distances between items in discordant pairs in the gold ranking for Neuron-wise MoS and LLM GPT-4. Discordant pairs of LLM-PP mostly cluster around the low gold ranking distances region, similar to Neuron-wise MoS, which shouldn’t significantly impact PP use cases (as observed in Section 7.1).
The resulting CDF of gold ranking distances for discordant pairs for LLM-PP GPT-4 and Neuronwise MoS are very similar. These results indicate that PP prompts can effectively design accurate performance predictors. Within LLM-PP, GPT-4 outperforms ChatGPT on both metrics across datasets. LLM-PP benefits from all the components of PP prompts. The last major row in Table 1 displays the performance when ablating different components of PP prompts. LLM-PP’s overall superior performance is attributed to having all PP prompt components together. Surprisingly, LLMPP outperforms baselines in MAE even without any instructions (Demonstration only), showcasing the LLM’s remarkable ability to grasp the performance prediction task based solely on demonstrations. While the MAE performance of different ablation variants is largely similar, there are differences in Kendall-Tau performance across variants. The second instruction (introducing downstream task-specific examples) and the fourth instruction (describing the efficiency metric) play crucial roles in achieving high Kendall-Tau for LLM-PP. LLM-PP requires at least 10 demonstrations to achieve SoTA MAE. The performance predictor’s quality notably improves with an increase in the number of demonstrations, as illustrated in Table 2. It becomes evident that LLM-PP requires at least 10 demonstrations to surpass the performance of SoTA supernet-based performance predictors in terms of MAE.
LLM-PP exceeds non-supernet baselines (White et al., 2022), with LLM-PP GPT-4 achieving a high Kendall Tau, as discussed in A.8.1. LLM-PP attains SoTA MAE and SoTA Kendall-Tau scores for low-resource/indigenous languages (Ebrahimi et al., 2023) (see A.8.2) and uncommon evaluation metric (COMET (Rei et al., 2022), see A.8.3). LLM-PP provides fairly robust performance predictions (see A.8.4). While LLM-PP excels in performance prediction quality, its cost scales linearly with the number of predictions. This cost can become prohibitive, especially for PP-heavy applications like NAS, where the number of predictions can reach several thousand.

heading: 6 Distillation of LLM-PP
text: To illustrate the cost, let’s consider the example of NAS run by HAT (Wang et al., 2020) for a latency constraint on a given hardware, involving the evaluation of approximately 3,000 candidate archi-
tectures. As of August 2023, the pricing for GPT-4 is 0.03$ per 1K tokens. Assuming PP prompts consume about one-third of 1K tokens, the estimated cost per constraint on a given hardware would be around 30$ (0.03∗30003 ). The total cost depends on the number of constraint types (e.g., latency, memory, FLOPs), values (e.g., 100ms, 200ms), and hardware options (e.g., Nvidia A100, Raspberry Pi). For instance, with three constraint types, five values for each constraint, and four target hardwares, the estimated cost could soar to approximately 1, 800$ (0.03∗3000∗3∗5∗43 ) per downstream task.
To address this cost challenge, we propose LLMDistill-PP, a cost-effective alternative trained on distilled outputs of LLM-PP. LLM-Distill-PP, a MLP based regressor, is trained using a distillation dataset for the PP task. This dataset is created by sampling architectures from the search space and recording the downstream task performance predicted by LLM-PP. LLM-Distill-PP is trained using architecture-specific hyperparameters as features and the distilled output as labels. Once trained, LLM-Distill-PP can predict the performance of unseen architectures for the given downstream task. If the number of distillation examples is small (e.g., 3, 000), the estimated cost to query LLM-PP will be approximately 30$ (0.03∗30003 ). This one-time cost of LLM-Distill-PP is amortized across different constraint types, values, and hardwares (e.g., 60 search runs), leading to a substantial 98.3% reduction in cost (from 1, 800$ to 30$). LLM-DistillPP achieves a superior efficiency-accuracy tradeoff, offering comparable accuracy to LLM-PP but with significantly faster prediction times (0.01s vs. 11.9s), as detailed in A.10.1. Setup. LLM-Distill-PP’s architecture encoding details can be found in A.6.3. The hyperparameters of its regression model, borrowed from HAT’s latency predictor, include 3 hidden layers, 400 as the hidden dimension, 128 as the batch size, 1e-5 as the learning rate, and 5000 as the number of training steps. Distillation from LLM-PP uses only 3000 architecture examples for each downstream task. Results. LLM-Distill-PP’s results are summarized in the third major row of Table 1. Despite its simple model design, LLM-Distill-PP performs similarly or better than LLM-PP for both ChatGPT and GPT4. In the case of ChatGPT, LLM-Distill-PP exhibits an average improvement of roughly 17% in both MAE and Kendall-Tau over LLM-PP. For GPT4, LLM-Distill-PP has a 7% lower average MAE
compared to LLM-PP while maintaining similar Kendall-Tau. Notably, LLM-Distill-PP achieves the SoTA MAE for the WMT’14 En-De task, outperforming LLM-PP by 20%. Two main factors contribute to this result. First, the smaller size of LLM-Distill-PP (a linear regression model with only 486K parameters) reduces the likelihood of overfitting compared to LLM-PP (an LLM with several billion parameters), resulting in better performance. Second, LLM-Distill-PP is a specialist model with parameters trained specifically for the performance prediction task using a few thousand examples. In contrast, LLM-PP is a generalist model that performs in-context learning with PP prompts and 10 demonstrations.

heading: 7 LLM-Distill-PP for Architecture Search
text: Given LLM-Distill-PP’s ability to achieve highperformance prediction quality in a cost-effective manner, we explore its application in a real-world task: NAS. In NAS, performance predictors typically rank candidate architectures to identify highperforming ones. As discussed in Section 2, existing NAS research in NLP primarily uses weight-sharing supernets as performance predictors. Therefore, we address the research question: Can LLM-Distill-PP accelerate architecture search while maintaining the efficiency and quality of SoTA NAS? To answer this question, we introduce the Hybrid-Search algorithm for NAS (HS-NAS). The core idea of HS-NAS is to employ LLM-Distill-PP for a subset of search iterations, utilizing the supernet for the remaining iterations. This approach is applied to the evolutionary search algorithm proposed in HAT.
LLM-Distill-PP will be used as performance predictor for all the search iterations in between llm-start-iteration and llm-end-iteration. In rest of the iterations, supernet will be used as performance predictor. When llm-start-iteration=1 and llm-end-iteration=num-iterations, HS-NAS
Algorithm 1 Hybrid-Search algorithm for Neural Architecture Search (HS-NAS). Changes to HAT’s search algorithm are in red color. The expanded algorithm can be found in A.9. Input: LLM-Distill-PP model: llm-distill-pp, Weight-sharing supernet: supernet, Latency predictor: latency-predictor, #Search iterations: num-iterations, Population size: population-size, Latency constraint: latency-constraint, LLM-Distill-PP Start Iteration: llm-start-iteration, LLM-Distill-PP End Iteration: llm-end-iteration, ... Output: best-architecture
1: popu ← population-size rand. samples from search space // create init. population 2: for iter ← 1 to num-iterations do 3: // gen. parents by picking top cand. arch. 4: if llm-start-iteration < iter <
llm-end-iteration then 5: parents ← top ‘num-parents’ arch.
from popu by llm-distill-pp 6: else 7: parents ← top ‘num-parents’ arch.
from popu by supernet 8: mut-popu = HAT’s mutation logic 9: cross-popu = HAT’s crossover logic
10: popu = parents∪mut-pop∪cross-pop 11: return top arch. from popu
uses LLM-Distill-PP as performance predictor for all the search iterations. HS-NAS comes with four arguments: (llm-distill-pp, supernet, llm-start-iteration, llm-end-iteration). For all our search experiments, we use LLMDistill-PP GPT-4 as llm-distill-pp due to its superior performance over the ChatGPT counterpart (see the third major row in Table 1). We use the latency-predictor and supernet from HAT. Other details of the setup (e.g., efficiency metric for search (search hours), and architecture (latency, GFLOPs, model size)) can be seen in A.7.

heading: 7.1 Results
text: Varying benchmarks. HS-NAS shows comparable performance to the SoTA across benchmarks, achieving approximately a 50% reduction in search hours. In some cases, it even enhances latency, GFLOPs, and model size, as illustrated in Table 3. This pattern highlights the effectiveness of using LLMs as good initializers for architecture search. Varying latency constraints. The trend observed in HS-NAS remains consistent across different latency constraints. Table 4 presents a comparison of the HS-NAS configuration (GPT-4, HAT, 1, 15) against the SoTA NAS for different latency constraints: 100ms, 150ms, and 200ms. Alongside a 50% reduction in search hours, HS-NAS attains comparable or improved GFLOPs and maintains the same model size compared to SoTA NAS. Varying start and end iteration pairs. Among different start and end iteration pairs, HS-NAS utilizing LLM-Distill-PP (GPT-4) for the initial 50% of iterations and HAT supernet for the remainder
performs comparably or outperforms HAT across all metrics. Table 5 presents the results of HS-NAS for various start and end iteration pairs. Utilizing LLM-Distill-PP for the entire search yields lower performance, indicating that a marginal degradation in Kendall-Tau hinders LLM-Distill-PP’s effectiveness in handling the complete search. These trends underscore the utility of a predictor with SoTA MAE scores for the initial search, while a predictor with SoTA Kendall-Tau is valuable for the later stages of the search.
Varying initialization seeds, FLOPs constraints, underlying supernet. HS-NAS exhibits resilience to initialization effects stemming from different seeds, yielding largely consistent results across metrics. Further details are provided in A.10.2. HS-NAS performs comparably to HAT under varying FLOPs constraints, showcasing a minimum 16% reduction in search hours, a 1.2% improvement in latency, consistent GFLOPs, and identical model sizes. These trends persist consistently
across benchmarks, as outlined in A.10.3. The superiority of HS-NAS remains robust across different underlying supernets, as elucidated in A.10.4. Trivially constructed efficient adaptations of SoTA. Search hours can be trivially reduced in several ways: halving the total number of search iterations and/or using distilled SoTA predictor instead of using supernet predictor directly. While these adaptations lead to a big drop in BLEU performance (1.8% for HAT (num-iter.=15)) or a big increase in latency and GFLOPs (9.7% and 32% respectively for Distilled HAT (num-iter.=15)), HS-NAS dominates these adaptions in search hour reductions, while maintaining SoTA performance and not degrading on any footprint metric, as detailed in A.10.5.
Putting all the observed trends of HS-NAS together, we find that the generality of HS-NAS extends to constraint types (latency, FLOPs), constraint values (different latencies, different FLOPs), different tasks (MT benchmarks), and underlying supernet (HAT, Neuron-wise MoS), while being robust to initialization effects.

heading: 8 Conclusion
text: This work shows that LLMs can be employed to create accurate and cost-effective performance predictors, providing insights into enhancing NAS. This contribution adds to the expanding field of LLMs in NAS, suggesting future research directions in adapting LLMs for both candidate architecture generation and joint performance prediction.

heading: 9 Limitations
text: • Expanding task domains. Our evaluation setup, centered on machine translation benchmarks, aligns with existing NAS for NLP literature (Wang et al., 2020; Jawahar et al., 2023a,b), primarily focusing on machine translation tasks. Investigating the applicability of the LLM-PP framework to diverse NLP
tasks (e.g., summarization, language modeling) and non-NLP domains (e.g., speech recognition, computer vision) stands as a crucial avenue for future exploration.
• Exploring diverse architectures. This work focused on classic Transformer architectures as outlined by Vaswani et al., aligning with NAS for NLP literature. While our primary investigation remained focused on these architectures, examining other architecture types (e.g., convolution embedding based (Salesky et al., 2023)) stands as a pertinent future direction.

heading: Acknowledgments
text: MAM acknowledges support from Canada Research Chairs (CRC), the Natural Sciences and Engineering Research Council of Canada (NSERC; RGPIN-2018-04267), Canadian Foundation for Innovation (CFI; 37771), and Digital Research Alliance of Canada.2 Lakshmanan’s research was supported in part by a grant from NSERC (Canada). We used ChatGPT for rephrasing and grammar checking of the paper.

heading: A Appendix

heading: A.1 Related Work - Extended
text: LLMs. LLMs can be classified into two categories based on their training methods: foundation and instruction-tuned LLMs. Foundation LLMs, which includes GPT-3 (Brown et al., 2020), GLaM (Du et al., 2022), LLaMA-1 (Touvron et al., 2023a), undergo language model training on unannotated corpus from the web. These LLMs typically encode a lot of useful knowledge in their parameters and can be used for a downstream task by either finetuning or zero/few-shot prompting. Instructiontuned LLMs are usually foundation LLMs that undergo instruction-tuning, where LLMs are explicitly fine-tuned to follow user defined instructions well. Such LLMs include InstructGPT (Ouyang et al., 2022), ChatGPT (OpenAI, 2023b), GPT4 (OpenAI, 2023a), LLaMA-2 (Touvron et al., 2023b), and PaLM-2 (Anil et al., 2023). In practice, instruction-tuned LLMs can follow a wide range of user’s instructions, even those that are outside the instruction tuning data distribution (Ouyang et al., 2022). However, depending on the task, instructiontuned LLMs are prone to generating content that are factually incorrect, hallucinated, ignores user’s instruction, toxic, and so on (Ouyang et al., 2022). These challenges make the current SoTA LLMs unreliable for critical applications such as medical diagnosis (Singhal et al., 2022).
Distilling LLMs. Distilling the generations from LLMs to smaller student models has become commonplace in NLP these days (Taori et al., 2023; Chiang et al., 2023; Wu et al., 2024; Mukherjee et al., 2023). The key motivations for such efforts include: (i) cost reduction: most LLMs are either behind a paywall or require high-end GPUs (e.g., NVIDIA A100) with high GPU memory (e.g., 80GB) to use, (ii) latency reduction: most LLMs are too slow even on high-end hardware (e.g., OPT-175B takes 4s for decoding 16 sequences of length 1024 on 8 NVIDIA A100 80GB GPUs (Xiao et al., 2023)), and (iii) customization: most LLMs are general purpose and are difficult to finetune. The commonly used distillation technique is sequence level knowledge distillation (Kim and Rush, 2016), where the student models are finetuned on responses from teacher LLMs via a standard language modeling objective.

heading: A.2 Examples for Metrics

heading: A.2.1 Mean Absolute Error
text: If predictions and TFS performances match perfectly, MAE will be zero, e.g., predictions are [23.4, 25.9, 28.1] and TFS performances are [23.4, 25.9, 28.1]. If predictions and TFS performances are mostly similar, MAE will be low, e.g., predictions are [23.4, 25.9, 28.1] and TFS performances are [23.3, 25.8, 28.2], MAE is 0.1. If predictions and TFS performances are extremely different, MAE will be high, e.g., predictions are [21.2, 24.0, 22.1] and TFS performances are [23.3, 25.8, 28.2], MAE is 3.33.

heading: A.2.2 Kendall-Tau
text: If predictions and TFS performances match perfectly, Kendall-Tau will be 100, e.g., predictions are [23.4, 25.9, 28.1] and TFS performances are [23.4, 25.9, 28.1]. If predictions and TFS performances are different but their architecture rankings are similar, Kendall-Tau will be 100, e.g., predictions are [23.4, 25.9, 28.1] and TFS performances are [22.2, 23.4, 25.1]. If predictions and TFS performances are different and their architecture rankings are dissimilar, Kendall-Tau will be negative, e.g., predictions are [23.4, 25.9, 28.1] and TFS performances are [23.4, 25.1, 22.2], Kendall-Tau is -0.33.

heading: A.3 Prompt Template - Expanded version
text: The expanded version of the prompt template can be seen in Figure 3.

heading: A.4 Prompt Template - Design Process
text: The design process began by examining crucial elements of the machine translation task, commonly used model architectures, and relevant efficiency metrics. Initially, we presented only demonstrations, borrowing hyperparameter wording from HAT’s configuration file. Subsequently, we added the role and definition of each hyperparameters, using wording from HAT’s helper description. Moving forward, our aim was to craft instructions enabling the LLM to grasp essential tasks, architecture, and metric details. Most instructions are prefixed with ‘You should’ to encourage strict adherence. Five instructions were incorporated. The first specifies the dataset, translation direction, and quality metric. The second provides examples randomly sampled from the training set, presented with generic prefixes (‘Input:’ for source sentence, ‘Output:’ for target sentence). The third outlines
the architecture, citing the ‘Attention Is All You Need’ (Vaswani et al., 2017) paper, assuming the LLM is familiar with this popular work. Standard settings and optimization algorithms are noted for training the architectures. The fourth identifies the efficiency metric in the demonstrations. The final instruction aims to summarize the relationships the LLM should learn to solve the task effectively.

heading: A.5 Kendall-Tau - Fine-grained analysis
text: We perform a fine-grained analysis of Kendall-Tau performance for Neuron-wise MoS and LLM-PP GPT-4. In figure 4, we plot the histogram of distance between the items in the discordant pairs in the gold ranking for Neuron-wise MoS and LLM GPT-4 across three MT benchmarks. The discordant pairs of LLM-PP lie mostly around low gold ranking distances region (like Neuron-wise MoS), which should not ideally have a big negative impact for the NAS task. In figure 5, we plot the corresponding cummulative distribution function (CDF). The CDF of gold ranking distances for discordant pairs for LLM-PP GPT-4 and Neuron-wise MoS are very similar.

heading: A.6 Machine Translation Details

heading: A.6.1 Machine Translation - Dataset Statistics
text: The statistics of the MT benchmarks is shown in Table 6.

heading: A.6.2 Machine Translation - Training Details and Search Space
text: Settings for training machine translation model include: 40K training steps, a cosine learning rate scheduler, Adam optimizer, and a warmup of learning rate from 10−7 to 10−3 with cosine annealing. The validation loss is used for model selection. The beam size is four with length penalty of 0.6. The search space (A) is borrowed from HAT (Wang et al., 2020), which is also shown in Table 7.

heading: A.6.3 Architecture Encoding
text: Each machine translation architecture is encoded using a list of following 10 values:
1. Encoder embedding dimension corresponds to embedding dimension of the encoder.
2. Encoder #layers corresponds to number of encoder layers.
3. Average encoder FFN. intermediate dimension corresponds to average of FFN intermediate dimension across encoder layers.
4. Average encoder self attention heads corresponds to average of number of self attention heads across encoder layers.
5. Decoder embedding dimension corresponds to embedding dimension of the decoder.
6. Decoder #Layers corresponds to number of
decoder layers.
7. Average Decoder FFN. Intermediate Dimension corresponds to average of FFN intermediate dimension across decoder layers.
8. Average decoder self attention heads corresponds to average of number of self attention heads across decoder layers.
9. Average decoder cross attention heads corresponds to average of number of cross attention heads across decoder layers.
10. Average arbitrary encoder decoder attention corresponds to average number of encoder layers attended by cross-attention heads in each decoder layer (-1 means only attend to the last layer, 1 means attend to the last two layers, 2 means attend to the last three layers).

heading: A.7 Search and Evaluation Setup - Details
text: The hyperparameters of HS-NAS’s search algorithm are taken from HAT: num-iterations=30, population-size=125, num-parents=25, num-mutations=50, num-crossover=50, and mutate-prob=0.3. We experiment with three latency-constraints: 100ms, 150ms, and 200ms. Once the search returns the best architecture, the final weights for this architecture is obtained by training the architecture from scratch to convergence using HAT’s training settings
(see A.6.2). The target hardware for search is NVIDIA V100 GPU with 32GB GPU RAM. The efficiency metric for search is search hours, which accounts for the time taken to complete all the search iterations. We focus on the following architecture-specific efficiency metrics: (i) latency - time taken in milliseconds to encode a sentence in source language and generate the translation in target language, (ii) GFLOPs - gigaFLOPs taken for the feedforward propagation, and (iii) model size - number of architecture-specific parameters in millions. Scripts to compute these metrics are taken from HAT’s codebase 3 and we refer readers to the HAT paper for more details about how these metrics are computed.

heading: A.8 LLM-PP - Extended Results
text: A.8.1 LLM-PP vs. non-supernet baselines. LLM-PP beats non-supernet baselines as well. We add comparison to five non-supernet baselines: #Params, #FLOPs, grad-norm, snip, and snyflow (see White et al. for details). From Table 8, it is clear that LLM-PP GPT-4 achieves a high Kendall Tau, outperforming all the non-supernet baselines. These results along with Table 1 showcases the superior performance of LLM-PP across a wide range of baselines.
A.8.2 LLM-PP on recent datasets and low-resource/indigenous languages.
LLM-PP works well for recent datasets and lowresource/indigenous languages. Compared to SoTA
3https://github.com/mit-han-lab/ hardware-aware-transformers
performance predictors, LLM-PP GPT-4 works well for recent datasets (e.g., 2023 benchmark), low-resource/indigenous languages (e.g., Bribri, Chatino). From the recent shared task: “AmericasNLP 2023 Shared Task on Machine Translation into Indigenous Languages” (Ebrahimi et al., 2023), we take three machine translation benchmarks: Bribri to Spanish, Chatino to Spanish, and Spanish to Bribri. Compared to WMT 2014, WMT 2019 benchmarks, these three benchmarks are very recent (2023 year) and one of the languages in each translation direction is an low-resource/indigenous language (Bribri, Chatino). As shown in Table 9, we compare LLM-PP GPT-4 against SoTA performance (BLEU) predictors on these benchmarks in terms of quality (MAE, Kendall-Tau). It is clear that LLM-PP achieves the SoTA MAE score across these benchmarks, which is consistent with the trends in WMT 2014, WMT 2019 benchmarks (as shown in Table 1). Impressively, on two of these benchmarks, LLM-PP also achieves the SoTA Kendall-Tau score. Put together, these results clearly showcase that LLM-PP generalizes well to recent datasets and low-resource languages.

heading: A.8.3 LLM-PP for COMET metric.
text: LLM-PP generalizes well to uncommon evaluation metrics. We build performance predictors that predict the Crosslingual Optimized Metric for Evaluation of Translation (COMET) (Rei et al., 2022) (Unbabel/wmt22-comet-da), which is relatively newer than the BLEU metric. Consider the Table 10 (performance averaged across two seeds), on the Bribri to Spanish task and the Chatino to Spanish task, LLM-PP achieves the SoTA MAE and SoTA Kendall Tau performance compared to SoTA performance predictors. These results show that LLM-PP generalizes well to uncommon evaluation metrics like COMET. Note that we exclude Spanish to BriBri task, since COMET does not support Bribri.
A.8.4 LLM-PP for robust predictions. LLM-PP provides fairly robust performance predictions. We compute the predictions for 8500 randomly sampled architectures using LLM-PP GPT-4 three times and compute the standard deviation of the three predictions for each architecture. The mean of the standard deviation for 8500 architectures is very low: 0.21, 0.27, 0.27 BLEU for WMT’14 En-De, WMT’14 En-Fr, and WMT’19 En-De respectively. Thus, LLM-PP provides fairly robust performance predictions. For all our search experiments, we use a single estimate from LLMPP.

heading: A.9 HS-NAS - Expanded Algorithm
text: The expanded algorithm for HS-NAS can be found in Algorithm 2.

heading: A.10 LLM-Distill-PP - Extended Results
text: A.10.1 Performance predictor quality vs. prediction time. Table 11 shows the efficiency (time taken to predict performance for 10 architectures) and accuracy (MAE, Kendall) for supernet-based PP (HAT, Layer-wise MoS, Neuron-wise MoS), LLM-PP (GPT-4), and LLM-Distill-PP (GPT-4). LLMDistill-PP provides the best efficiency-accuracy tradeoff with on par accuracy as LLM-PP but significantly faster prediction time (0.01s vs. 11.9s).
A.10.2 Varying initialization seeds. HS-NAS seems robust to initialization effects caused by different seeds, achieving largely similar numbers on all metrics. This result is shown in Table 12, where latency numbers change slightly while numbers for other metrics are almost the same.
A.10.3 Varying FLOPs constraints. HS-NAS performs similarly to HAT for different FLOPs constraints, with at least 16% reduction in search hours, 1.2% improvement in latency, same GFLOPs and same model size. Table 13 contains these superior results of HS-NAS across 2.5 and 3.0
Algorithm 2 Hybrid-Search algorithm for Neural Architecture Search (HS-NAS). Changes to HAT (Wang et al., 2020)’s search algorithm are in red color.

heading: Input:
text: LLM-Distill-PP model: llm-distill-pp, Weight-sharing supernet: supernet, Latency predictor: latency-predictor, #Search iterations: num-iterations, Population size: population-size, #Parents: num-parents, #Mutations: num-mutations, #Crossovers: num-crossover, Mutate probability: mutate-prob, Latency constraint: latency-constraint, LLM-Distill-PP Start Iteration: llm-start-iteration, LLM-Distill-PP End Iteration: llm-end-iteration Output: best-architecture
1: popu← population-size rand. samples from search space // create init. population 2: for iter ← 1 to num-iterations do 3: // gen. parents by picking top cand. arch. 4: if llm-start-iteration < iter < llm-end-iteration then 5: parents← top ‘num-parents’ arch. from popu by llm-distill-pp 6: else 7: parents← top ‘num-parents’ arch. from popu by supernet 8: // gen. cand. via mutation 9: mutate-popu = {}
10: for mi← 1 to num-mutations do 11: gene← mutate a random eg from popu with mutate-prob 12: if gene satisfies latency-constraint via latency-predictor then 13: mutate-popu = mutate-popu ∪ gene 14: // gen. cand. via cross-over 15: crossover-popu = {} 16: for ci← 1 to num-crossover do 17: gene← crossover two random eg from popu 18: if gene satisfies latency-constraint via latency-predictor then 19: crossover-popu = crossover-popu ∪ gene 20: // upd. population 21: popu = parents ∪ mutate-popu ∪ crossover-popu 22: return top arch. from popu
GFLOPs constraints. These trends largely hold true across benchmarks as well, as shown in Table 14.
A.10.4 Varying underlying supernet. The dominance of HS-NAS seems consistent across the underlying supernet. In the results so far, HAT is the supernet used by HS-NAS. In Table 15, we replace HAT with Neuron-wise MoS and show that HS-NAS performs similarly to Neuron-wise MoS, with at least 50% reduction in search hours, better or similar model size and GFLOPs.
A.10.5 Trivially constructed efficient adaptations of SoTA
Search hours can be trivially reduced in several ways: halving the total number of search iterations and/or using distilled SoTA predictor instead of using supernet predictor directly. As shown in Table 16, the former approach suffers from a big drop in BLEU performance (1.8% for HAT (num-iter.=15)), while the latter approach suffers from a big increase in latency and GFLOPs (9.7% and 32% respectively for Distilled HAT (num-iter.=15)). On the other hand, HS-NAS dominates these adaptions in search hour reductions, while maintaining the performance of SoTA and not degrading on any footprint metric.

heading: Future Work
text: , speech recognition, computer vision) stands as a crucial avenue for future exploration. • Exploring diverse architectures. This work focused on classic Transformer architectures as outlined by Vaswani et al. , aligning with NAS for NLP literature. While our primary investigation remained focused on these architectures, examining other architecture types (e. g. , convolution embedding based (Salesky et al. , 2023)) stands as a pertinent future direction