W-PCA Based Gradient-Free Proxy for Efficient Search of Lightweight Language Models
Authors: Shang Wang
Venue: ICLR 2025 Poster (2025)
Keywords: zero-shot NAS
gradient-free
lightweight language models
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: The demand for efficient natural language processing (NLP) systems has led to the development of lightweight language models. Previous work in this area has primarily focused on manual design or training-based neural architecture search (NAS) methods. Recently, zero-shot NAS methods have been proposed for evaluating language models without the need for training. However, prevailing approaches to zero-shot NAS often face challenges such as biased evaluation metrics and computational inefficiencies.
In this paper, we introduce weight-weighted PCA (W-PCA), a novel zero-shot NAS method specifically tailored for lightweight language models. Our approach utilizes two evaluation proxies: the parameter count and the number of principal components with cumulative contribution exceeding $\eta$ in the feed-forward neural (FFN) layer.  Additionally, by eliminating the need for gradient computations, we optimize the evaluation time, thus enhancing the efficiency of designing and evaluating lightweight language models.
We conduct a comparative analysis on the GLUE and SQuAD datasets to evaluate our approach. The results demonstrate that our method significantly reduces training time compared to one-shot NAS methods and achieves higher scores in the testing phase compared to previous state-of-the-art training-based methods. Furthermore, we perform ranking evaluations on a dataset sampled from the FlexiBERT search space. Our approach exhibits superior ranking correlation and further reduces solving time compared to other zero-shot NAS methods that require gradient computation.

Published as a conference paper at ICLR 2025
W-PCA BASED GRADIENT-FREE PROXY FOR EFFI- CIENT SEARCH OF LIGHTWEIGHT LANGUAGE MODELS
Shang Wang ShanghaiTech University
wangshang20240 www.a//02222eaaaaaaaaaaaacee www.tttehhhhcc...cm
ABSTRACT
The demand for efficient natural language processing (NLP) systems has led to the development of lightweight language models. Previous work in this area has primarily focused on manual design or training-based neural architecture search (NAS) methods. Recently, zero-shot NAS methods have been proposed for evaluating language models without the need for training. However, prevailing approaches to zero-shot NAS often face challenges such as biased evaluation metrics and computational inefficiencies. In this paper, introduce weight- weighted PCA (W-PCA), a novel zero-shot NAS method specifically tailored for lightweight language models. Our approach utilizes two evaluation proxies: the parameter count and the number of principal components with cumulative contribution exceeding 7/ in the ferw-forward neural (FFN) layer. Additionally, by eliminating the need for gradient computations, we optimize the evaluation time, thus enhancing the efficiency of designing and evaluating lightweight language models. We conduct comparative analysis on the GLUE and SQuAD datasets evaluate our approach. The results demonstrate that our method significantly reduces training time compared to one-shot NAS methods and achieves higher scores in the testing phase compared to previous state-of-the-arr training-base -based methods. Furthermore, we perform ranking evaluations on a dataset sampled from the FlexiBERT search space. Our approach exhibits superior ranking correlation and further reduces solving time compared to other zero-shot NAS methods that require gradient computation.
INTRODUCTION
Large language models (LLMs) have shown exceptional per-
Search Time of NAS Mernods
formance across various domains (OpenAl 2023). How- ever, their size and computational demands pose challenges in cosstrre-cnnstrained environments like mobile devices and edge computing. Therefore, there a growing need to explore lightweight language models that can operate efficiently these platforms. One approach to address this challenge is
1330
837
through knowledge distillation (KD) (Liu al. 2023 Li et al.
2023c Li & Jin 2022: Li et al. 2023b Li 2022 Li et al.
2024a). where larger language model
teacher to
train a smaller, more lightweight language model (Ture et al.
2019 Sanh et al. 2020; Jiao al.) 2020; Sun al. [2020]
Figure 1: Comparison of the run- ning time between W-PCA and other training-based NAS methods for lightweight language models. Our method achieves a substantial reduc- tion in search time for the optimal network structure by two to three orders of magnitude, as we not need to train the supernet.
Wang et al. 2020 However, the student models trained for
these tasks were manually designed. To effectively search for
student models, the use of neural architecture search (NAS)
has become essential.
NAS is technique that automates the process of designing neural networks, enabling the exploration of a wide range of architectures identify the most optimal ones for given task. Vanilla NAS approaches primarily used reinforcement learning (Zoph & Le 2016) genetic algorithms (Real et al. 2019)
train neural networks from scratch,

---

Published as a conference paper at ICLR 2025
Synaptic Diversity 0:0.021 pi 0.174
Synaptic Saliency
Activation Distance
Head Inpooriance
0.130 gr 0.185
T:
p 0:123
p: 0.170
00223
1.-
0.004
00128
0.000
00115
6.7
0.000
80228
66-
10235
0.001
Saftmax Confidence
#params
t:0.037 p: 0,046
pr. 0.677
247
0.429 0.546
240
0.526
0.06
250-
14-
3.05
3.04
250
10-
68
3.00
230
GLUE
GLUE Score
GUVE Score
GLUE
Figure 2: Plots depicting the evaluation of zero-shot proxy metries 500 randomly sampled architectures from the FlexiBERT search space. As in literature (Serianni & Kalital 2023), we use the GLUE score of each neural network as the ground truth and evaluate the performance of each zero-shot proxy metric based on its ranking correlation with the ground truth. The specific calculations of PCA is described in Section 3.2 and the respective zero-shot proxies used for the
comparison are summarized in Section 2.3 Our metric W-PCA is calculated as the product of the number of parameters (#params) and the principal component analysis (PCA).
but these methods were computationally expensive. Subsequently, one-shot NAS methods, such as gradient-based (Liu et al., 2018) and single path one-shot (SPOS) methods (Guo et al. (020)). were proposed. These methods are more efficient as they leverage pre-trained models or parameter sharing, requiring the establishment of a supernet in advance from which to sample the optimal subnetworks. Importantly, many lightweight model search tasks in natural language understanding (NLU) are accomplished using one-shot NAS (Xu al.. 2021; Dong al., 2021 Gao et al.) 2022). While one-shot NAS reduces training costs compared to training from scratch, it still requires various training strategies to effectively train the supernet. However, to further enhance search efficiency, is necessary to introduce zero-shot NAS (Mellor et al., 2021a). Zero-shot, also known training- free NAS, is a promising approach that climinates the need for training neural networks and directly evaluates their performance using proxy metrics. This significantly reduces the training time and computational resources required for NAS.
Existing zero-shot NAS methods (Abdelfattah et al. 2020 Wei et al. 2024 Dong et al. 2023a b) have primarily focused on ranking correlations on NAS benchmark datasets (Klyuchnikov et al. 2022), with limited consideration for specific deep learning tasks. This limitation hinders their applicability and effectiveness in practical scenarios. Additionally, these methods often solely consider single feature of the models, leading to biased evaluations and potentially overlooking important characteristics.
In our research, we aim address these limitations and improve the applicability of zero-shot NAS. In SectionS) 5 we conducted ranking correlation experiments. As illustrated in Figure 2 our attempts to incorporate previous zero-shot proxies into language model evaluation yielded unsatisfactory results. However, we observed a strong correlation in ranking between principal component analysis (PCA) and the number of parameters (#params), with their product demonstrating even better performance.
Motivated by these findings, we propose novel approach called Weighte PCA (W-PCA), which takes into account both the parameter count and the number of principal components with cumulative contribution exceeding given 1) threshold in the model. By integrating these two factors, our aim is to achieve a more accurate evaluation of language models in the context of zero-shot NAS. Furthermore, have designed search space specifically for NLU tasks and applied our designed zero-shot proxies, as well as the previous zero-shot proxies used in Transformer, to this search space. To the best of our knowledge, this is the first work that applies zero-shot NAS NLU tasks.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
2.1 LIGHTWEIGHT BERT MODELS
Ture et al. observed that distillation and pre-training + fine-tuning have mutually reinforcing effects. DistilBERT (Sanh et al. 2020) utilizes triple loss function for training the lightweight model. TinyBERT (Jiao et al. 2020) applies distillation in both the pre-training and task-specific learning phases. MobileBERT (Sun et al. 2020) proposes bottleneck structure to reduce the parameter count. MiniLM (Wang et al.) 2020) introduces a compression method called deep self-attentive distillation. In this study, we incorporate both the standard BERT-base (Devlin et al. 2019) and MobileBERT models, along with their weight-sharing variations, where each layer integrated into the supernet.
2.2
ONE-SHOT NAS FOR EFFICIENT MODELS
Numerous methods have been proposed for performing neural architecture search (NAS) (Hu et al. Dong et al. 2022 Sun et al., 2024: Li et al., 2024d c) to develop efficient models. NAS- BERT (Xu et al. 2021) trains large supernet on carefully designed search space that includes diverse architectures, generating multiple compressed models with adaptable sizes and latency. Efficien(BERT (Dong al., 2021) proposes a three-stage coarse-to-fine search scheme to optimize the combination of the multilayer perceptron (MLP) in the feed-forward network (FFN), ultimately reducing the parameter count of the FFN. AutoBERT-Zerr (Gao et al. 2022) devises a search space that includes unary and binary math operators for constructing attention structures and backbones for general pre-trained language models (PLMs) from scratch. the best of our knowledge, the most recent NAS method that incorporates lightweight BERT models in the experiment.
2021:
2.3 ZERO-SHIOT NAS
Zero-shot NAS has been applied to transformer- bassed architectures several ways. We provide a summary of these applications below.
Synaptic Saliency (Tanaka et al. (020) aims to prevent layer collapse during network pruming, as this collapse significantly reduce the accuracy of the network. The formulation for this approach expressed as follows:
Oc 0e
S())
where L. represents the loss function, 0 denotes the network's parameters, and @ is the Hadamard product. AAbdefattah et al. generalize synaptic saliency as a zero-shot metric for NAS by summing over all parameters in the network: S = == S(O.)
Synaptic Diversity builds upon previous research on rank collapse in transformers. In this phe- nomenon, the output of a multihead attention block tends to converge to rank for given set of inputs, which significantly impairs the performance of the transformer. Zhou et al. propose a method that utilizes the nuclear norm of attention head's weight matrix W m approximation of its rank. This approach leads to the computation of the synaptic diversity score as follows:
aL
Sp Σ
Iuc ④
I [mllnne
OW
Activation Distance proxy metric introduced by Mellor al. to assess the ReLU activations of network. By computing the Hamming distance between the activations within the initialized network for each input in minibatch, this metric determines the similarity of the activation maps. The authors observe that when the activation maps for given set of inputs exhibit higher similarity, the network faces greater difficulty in disentangling the input representations during the training process.
Jacobian Covariance evaluates the Jacobian J =
of the network's 's loss function with respect to the minibatch inputs. Further details of this metric can be found in the original paper (Mellor et al.) 2021b).
(
(OL ixi
aL
7xN
Jacobian Cosine (Celotti et al. 2020 is proposed as an improvement to the Jacobian Covariance metric, aiming to enhance computation speed and effectiveness. This improvement involves utilizing
3

---

Published as a conference paper at ICLR 2025
cosine similarity instead of covariance matrix to measure similarity. The metric is computed as follows:
N 1N
S = N2 - N [=1
-
Here, Jn represents the normalized Jacobian, and 7 is the identity matrix. The metric computed using minibatch of N inputs. In their large noise and more noised scores, the authors introduce various noise levels to the input minibatch, hypothesizing that architectures exhibiting high accuracy will demonstrate robustness against noise.
Attention Confidence, Importance, and Softmax Confidence "Confident" attention heads exhibit high attention towards single token, indicating their potential importance to the transformer's task. Researchers have proposed different approaches to calculating confidence, including examining the softmax layer of the attention head and analyzing the sensitivity of the attention head to weight masking by computing the product between the attention head's 's output and the gradient of its weights. Serianni & Kalita summarize the findings from (Voita al. 2019: Behnke & Heafield 2020 Michel al. 2019) regarding the following metrics:
Confidence: A, (X) tे Emen mmax(Atte (x,)))
Softmax Confidence: (X) ΣN mxx)(,((x)))
Importance: A₁ (X) | XX)
ac()
(XI,(X
where X = { represents minibatch of N inputs, L denotes the loss function of the model, and Att,, on denote an attention head and its softmax, respectively. To obtain an overall metric for the entire network, [Serianni netrirk, & Kalita extend these scores by averaging them across all H attention
heads: A(X)
Ei 1 (X)
OUR GRADIENT-FREE WEIGHT-WEIGHTED PCA PROXY
3.1
MOTIVATION FOR USING PCA
1,1
(a)
Figure 3: (a) and (b) show the PCA score curves for BERT (Devlin et al. 2019) and MobileBERT (Sun ct al. 2020), respectively, at different epochs during training (7=0.999). (c) presents the progression of GLUE scores for BERT and MobileBERT
training epochs.
Inspired by the work in Pan et al. (003),, which leverages PCA to optimize the training of Vision Transformers, analyzed the trends in PCA variation during the training of BERT and MobileBERT. From Figure 3 we can draw the following conclusions:
1. Tracking performance via PCA. As shown in Figure 3 the overall GLUE score and the PCA values for each layer progressively increase with the number of training epochs. This indicates that PCA values effectively reflect the performance of the neural network. Specifically, the steady rise in PCA values suggests that the network's 's internal representations become more structured and discriminative.
Diminishing returns after peak performance. Our observations also revealed that after reaching peak values, the PCA curves flatten out, indicating that further training yields diminishing returns.
4

---

Published as a conference paper at ICLR 2025
This aligns with the conclusions in Table 13 where discussed that prolonged training results in minimal performance gains.
3. Early PCA values predictors. Notably, layers with higher PCA values epoch 0 (i.e., before training begins) tend to maintain higher PCA values throughout training. This led us to hypothesize that the initial PCA values could serve effective indicators for comparing neural network architectures during the NAS phase.
To validate these hypotheses, conducted experiments shown in Figure 2 which confirmed that early PCA values correlate well with the final performance ranking of the networks. This insight motivated to further pursue accuracy comparison experiments, yielding promising results.
3.2 VANILLA PCA PROXY
For given 7, the distribution of PCA principal component values reflects the proportion of useful information the matrix. We attempt to use this distribution as metric for evaluating neural network performance. Our metric is computed as follows:
Sy(X) (TA_dim(X,,,
(I)
Here, X E BBNxx represents minibatch of inputs, where B represents the batch size, N is the token length, and D represents the embedding dimension. 1j represents the cumulative contribution rate of principal components. We calculate the PCA values for the hidden states after the initial linear transformation in the FFN layer. Specifically. if we express the FFN layer as:
FFN(X) = o(XW, + b1) W2 + b2
Then, we compute the PCA value of the XW1 + b part, denoted as H E Bx.Nx in the following text, where D' represents the hidden dimension. To compute the PCA values, we first reshape H into a two-dimensional matrix H' E (BN)xxD Then, we subtract the mean of each feature (column) from the data to center it:
Heentered H' - H' (0)
(3)
We then calculate the covariance matrix of the centered data as:
C (BN) - H. Heentered centereed
(4)
Next, perform eigenvalue decomposition the covariance matrix C E D'XD' to obtain eigenvalues (A) and eigenvectors (V):
C - VAVT
(5)
Here, is diagonal matrix eigenvalues and the matrix of eigenvectors. After sorting each eigenvalue X; in descending order, we determine the minimum number of eigenvectors k required to explain at least 7) variance:
k min
n}
(6)
Σ
A
This value of k represents the required PCA_dim( (X) By analyzing PCA_ _ii (the dimensions with PCA values exceeding a threshold 11), we can identify the dimensions that contain higher amount of valuable information.
The metric for an m-layer neural network model obtained by summing S)) over all layers, resulting in:
S(X)
Σ (X)
/=1
5

---

Published as a conference paper at ICLR 2025
Here, the metric (X) represents the PCA-based value for specific layer
Finally, to compute the overall metric S(X) for an m-layer neural network model, we sum the effectively assess
layer-specific metrics (X) over all layers. By utilizing this methodology, the performance of candidate architectures based on their PCA values and identify the dimensions that contribute significantly to the valuable information in the hidden states.
3.3 WEIGHT-WEIGHTRY PCA PROXY
It is extremely challenging to discover or design proxy that outperforms weight parameters (#Params) in terms of stability and performance (Li al. 2023a). After identifying PCA an excellent proxy. multiplying it with #Params is a worthwhile attempt, as the additional computation time for proxy negligible compared to training neural networks. Thus, we propose new metric called W-PCA, which quantifies the amount of valuable information captured by each dimension relative to the number parameters the architecture.
The W-PCA metric is computed the product of the number of weight parameters (w) and the PCA value for each dimension. Mathematically, it can be expressed as:
W-PCA(X) = w S(X)
(8)
Advantages of our method include:
The W-PCA metric captures the relationship between the number of parameters and the valuable information in each dimension. This relevance is crucial in evaluating the efficiency and effectiveness of candidate architectures. By considering the PCA values, we can identify dimensions that contribute the most the architecture's performance, allowing for informed Docision-making during architecture search.
Strong Correlation:
Gradient- Unlike many traditional optimization methods that rely on gradients, our method- ology is gradient-free. This eliminates the need for extensive backpropagation and derivative calcula- tions, making the evaluation process more efficient and less computationally expensive.
3. One forward propagation only: Our methodology requires only forward propagation during the evaluation of candidate architectures. This simplifies the implementation and reduces the com- putational overhead, as it avoids the need for complex and resource-intensigg operations such as backpropagation.
By leveraging the advantages of strong relevance, gradient-freenest and the use of only forward propagation, our methodology based the W-PCA metric provides an efficient and effective approach for training-free architecture search. It enables researchers and practitioners to evaluate candidate architectures based on their valuable information content relative to the number of parameters, facilitating the exploration of architecture design space and aiding in the development of more efficient and effective models.
SEARCH SPACE FOR NLU TASKS
To enhance the evaluation of W-PCA's performance on NLU tasks, we have meticulously crafted search space. Drawing inspiration from SPOS (Guo et al.) 2020). our search targets model comprised multiple layers, with each layer capable of being a lightweight BERT model. The hidden dimension of the FFN layer within each BERT block is determined through random selection. This deliberate randomness aids in exploring a diverse range of architectures during the search process.
RANKING EVALUATION
5.1 DATASETS
To assess the accuracy of the proposed proxy indicators for neural network evaluation, we employed benchmark consisting of well-trained BERT structure suggested by Serianni & Kalita as the

---

Published as a conference paper at ICLR 2025
Popubsiton of recitectoro
Canditac block
Oulput
Ouroot
Add Numm
WEPCA 1.57
108c
Take
Bos archttetuue
tiptr dimencon
Add Norm
Adll Neze
Student
MIKA
canditee hidla dineriion
Iapus
Lavar
Input
Inper
dimenian
Teacher Backhon:
Da BERT
(b) MeR
Figure 4: Overview of the W-PCA framework for NLU tasks. The search space consists of m layers, each with 2 candidate blocks and candidate dimensions, resulting in a total of (2 n) combinations. A genetic algorithm (detailed parameterization provided in Section 6.2.1] is employed identify the optimal structure with the highest W-PCA value. This structure subsequently refined through additional training using knowledge distillation (KD). In the figure, FFN and MHA represent the ferd-forward network and multi-head attention, respectively.
testing dataset. Specifically, this benchmark selected 500 structures from the FlexiBERT (Tuli et al. 2023) search space (as presented Table 7 and utilized ELECTRA (Clark et al. 2020). rather than the MLM method, for training to efficiently pretrain compact BERT model. he training dataset comprised 8,013,769 documents sourced from the OpenWebTex (Gokaslan et al. 2019) corpus, amounting total of 38GB. For detailed training information, please refer to Appendix B After training, the scores obtained by fine-tuning on the GLUE dataset will serve as the reference for evaluating the correlation among different zero-shot proxies.
5.2 RESULTS AND ANALYSIS
Table 1: Comparison of different zero-shot proxies the FlexiBERT benchmark. "Time" represents the computa- tion time for the metric calculated 1,000 times. Both and p are computed to measure the ranking correlation between each zero-shot proxy and the ground truth of the neural networks, represented by their BLEU scores
The evaluation results include the Kendall rank correlation coefficient (Kendall 7) and the Spearman rank cor- relation coefficient (Spearman p). Ta- ble I demonstrates that Vanilla PCA has already surpassed the previous shot proxy in terms of ranking correla- tion, and W-PCA performs even better than Vanilla PCA. Furthermore, W-PCA achieves higher computational efficiency
Proxy
Time
-fee
P
Synaptic Diversity Zhou et al.
0.021
0.174
Synaptic Saliency Ahdelfatiah
0.185
Activation Distance (Melorr all 2021a
68
0.123
than proxies due to the absence of gra-
Jacobian Cosine Celothi 2020
103
0.116
0.149
dient computation. In Appendix C we
Head Importunce Scrianii Kalta
2023
112
0.048
0.170
compare the ranking stability of W-PCA
Head Confidence
Semann & Kaina
2023
81
0.306
0.364
with other zero-shot metrics using differ-
Vanilla PCA
61
0.466
0.677
ent initialization weights and batch in-
W-PCA
74s
0.526
0.698
puts.
ACCURACY COMPARISION
6.1 DATASETS
To enable accurate comparison to other lightweight BERT models, evaluate the performance of W-PCA using the GLUE (Wang et al. 2018) and SQuAD (Rajpurkar et al.) 2016) datasets and their corresponding task-specific evaluations.

---

Published as a conference paper at ICLR 2025
Table 2: Performance comparison of the test set on the GLUE benchmark. The performance of all
zero-shot proxies is evaluated on the search space depicted in Figure 4 Latency measurements of the models are conducted using the NVIDIA A100 GPU.
Model
Type
PParams
Latency
QNLI
MRPC
SST:2
CoLA
STS-B
MNLL-m/mm
RTE
QOP
AVG
BERT-base
2019
2744ms
90.5
88.9
93.5
52.1
85.8
84.6/83.4
66.4
BERT-buse (065))
manual
108,9M
274ms
91.4
88.7
93.0
19.0
87.5
81,9/3399
76.6
71.3
80.7
BERT-iny et 2019
momual
Mms
83.2
87.6
19.5
77.1
75/4/119
62.6
BERT-small
Tureel ml 2DTV
momual
28. 8M
79hms
86.4
83.4
89.7
27.8
77.0
77.6/77.0
61.8
68.1
72.1
DisilBERT-c
Sanh
A120
67.0M
15
88.9
86.9
92.5
49.0
81.3
82.6/81.
58.4
70.1
TiayBERT-4
Jiao at 2130
mannal
14.5M
45ms
87.7
88.5
91.2
27.2
83.0
81.8/80.7
64.9
69.6
75.0
MobileBERT-ing
2020
manual
15.1M
62ms
89.5
87.9
91.7
16.7
80.1
81,5/816
65.1
68.9
77.0
20/1
one-shot
15.7M
62mm
89.3
89.9
92.4
38.1
85,1
69.4
71.2
77.9
FMhcienRERT -4
Kxng cF
one-shot
16.0M
6dms
90.6
88.9
92.3
42,5
83.6
83.08225
67.8
71.2
78.0
Synapric Saliency
Aldelfatiah
2020
zero-shoe
58ms
89.4
88.1
91.0
33.6
83.1
82.688111
70.6
76.6
Activation Distance
Meller 3 k14
zero shot
15.6
60ms
88.9
87.6
91.2
30.7
82.9
$1.1/0.44
70.4
70.1
75.9
Synaptic Diversity
Zhou 33
57ms
25.8
B1.7
81,3/80.2
70.6
Head Confidence
Senanni
Kaiin
M22
peoo-shor
15.5M
63ms
89.5
88.3
92.4
31.7
85.7
82.8/819
74.0
70.9
77.5
Softmax Confidence
Kanna M023
15.6M
61ms
88.4
87.5
90.8
32.5
83.5
81,2/80.5
70.3
69.9
76.1
W-PCA Tiay
zero shot
9.6M
38ms
88.7
87.6
91.9
27.1
B1.8
81.1/79.8
71.1
70.3
75.9
W-PCAA-Small
zuto-hbot
15,6M
54mx
90.3
88.7
91,5
38.4
86.4
82,882.2
73.8
70,8
76.3
6.2
IMPLEMENTATION DETAILS
6.2.1 SEARCH SPACE
wherein we set the values of m and n to 12 and 6, respectively. Each block possesses a hidden size of 528, with the inner hidden size of the MobileBERT series blocks being one-fourth of the total hidden size. The hidden dimensions of the FFN increase by a factor of 132 for each multiple ranging from to n. To calculate the value of PCA A_dim we set 1) to 0.99, with the selection process detailed in Appendix [ To identify the combination of blocks that yields the highest PCA value, we utilize a genetic algorithm, the detailed implementation of which can be found in Appendix E This algorithm uses a population size of 50 and generation count of 40. The crossover probability is set to I, the mutation probability to 0.1, and the upper limit for the model parameters is set to 15.7M, resulting in the W-PCA-Smal model. By further reducing the upper limit for the model parameters to 10M and halving the number of layers (m), we obtain the W-PCA-Timy model.
The search space is illustrated in Figure 4
6.2.2 TRAINING
Once we obtain the desired architecture, we pretrain the model using the complete English Wikipedia (Devlin et al. (211)) and BooksCorpus (Zhu et al. 2015).. We then proceed to fine-tune the model on each individual downstream task. During pretraining, the network is trained with a batch size set to 256. For the fine-tuning phase of the downstream tasks, the network is trained with batch size set to 32. The CoLA task is trained for 50 epochs, while the other tasks are trained for 10 epochs. The learning rate is set at 0.0001 during pretraining. In the fine-tuning phase, the learning rate is set at 0.00005 for GLUE tasks and 0.0001 for SQuAD tasks. The training process utilizes the Adam optimizer with 31 and B2 values set at 0.9 and 0. 0.999, respectively. The weight decay is to 0.01. The learning rate decays linearly with a warm-up ratio set to 0. 1. The KD loss function used in our approach is described in Appendix F
6.3 RESULTS ON GLUE
6.3.1 MODEL ACCURACY AND LATENCY
T presents the results of the GLUE scores and model latency for the KD-based methods. Among them, except for the BERT-base teacher model we used ourselves, the results all the manual and one-shot methods in the table are from relevant papers. Since zero-shot NAS methods have not been used in NLU tasks before, we applied the recent top-performing zero-shot proxy approaches on Transformer language models to the search space shown in Figure 4
As shown in Table under the search space depicted in Figure our W-PCA metric achieved higher average scores the GLUE test set compared to all baseline manual and one-shot methods. At the same time, it outperformed the previous state-of-the. (SOTA) method EfficientBERT (Dong et al. 2021) in terms of parameter count, latency, and average score in the field of lightweight models.
8

---

Published as a conference paper at ICLR 2025
Table 3: Comparison of results on the GLUE dev set with other NAS methods. The "Time" column represents the GPU days consumed by the NAS method search. It is not feasible to make a subcomparison with AutoBERT-Zero-sing -mall as it does not provide individual scores for each task in the GLUE dev set.
Model
#Params
Time
QNLI
MRPC
SST-2
CoLA
STS-B
MNLI-m
RTE
QQP
AVG
JAS-BERT-I0
Xu et ml
2021
10.0M
96
86.3
79.1
88.6
34.0
84.8
76.4
66.6
88.5
75.5
BERT-30
RU44
88.4
84.6
80.3
EtficienBERTT
INY
Dong 51
2021
9.4M
58 d
89.3
90.1
90.1
39,1
79.9
81.7
63.2
86.7
77.5
EfficienBERT
Dong ot 2121
15.7M
58 d
90.4
91.5
91.3
50.2
82.5
83.1
66.8
87.3
80.4
AutoBERT small
-1,000
Synaptic Diversity (Zhou
2022
15.6M
0.7 d
88.9
87.6
91.4
32.0
84.1
81.0
73.4
88.2
78.3
Head Confidence
(Seriani & Kalna 2023
15.6M
0.5
90.1
89.7
92.4
37.5
84,1
82.5
75.9
89.1
80.2
Softmax Confiderioe
Serianii Kalita 2023
15.6M
0.5
89.4
88.3
92.0
32.6
84.7
81.6
73.9
88.9
78.9
W-PCA-TTmy W-PCA-Small
9.6M
89.2
89.2
33,2
71.1
78.4
15.6M
0.5
90.8
90.5
92.8
44.0
85.3
82.9
76.1
88.8
81.4
Additionally, W-PCA achieved the highest score on the STS-B task. It is worth noting that, in the same search space, the optimal structure found by W-PCA surpasses all previous zero-shot methods (Abdelfattah et al., 2020; Mellor et al., 20212 Zhou et al.) 2022 Serianni & Kalital (023) applied to Transformer language models, highlighting its exceptional ability in exploring optimal network structures in zero-shot NAS methods.
6.3.2 SEARCH EFFICIENCY
As shown in Table under our search space, the search efficiency of all zero-shot proxies (including our W-PCA method) has been improved by two to three orders of magnitude compared to previous bhining-based NAS, and achieved competitive performance. The three zero-shot proxies, Synaptic Diversity (Zhou et al. 2022), Head Confidence (Serianni & Kalita 2023), and Softmax Confidence (Serianni & Kalita 2023), can compete with the optimal structures found by previous training- NAS in our search space. Our W-PCA method surpasses all previous training-based methods in the field of lightweight language models in terms of average score and achieves the best average score. Moreover, in three out of eight tasks, W-PCA achieves the highest performance. Our method discovers the latest SOTA effects in the field of lightweight models with almost negligible search cost, reducing greenhouse gas CO2 emissions by two three orders of magnitude and significantly improving the utilization of global energy resources.
It is also worth noting that in the internal comparison of zero shot proxies, Head Confidence (Serianni & Kalita 2023), Softmax Confidence (Serianni & Kalita, 2023), and our W-PCA method require shorter search time than the Synaptic Diversity (Zhou et al. 2022) method, which needs to compute W-PCA-TI -iny model has lower parameter
gradients, by an additional 0.2 GPU days. Additionally, limit set during the search, resulting in slightly lower computation time for the forward propagation of each neural network individual, thus reducing the search time by 0.1 GPU days compared to the W-PCA-Small model.
6.4 RESULTS ON SQUAD
We compared the W-PCA proposed in this arti- cle with manually designed lightweight models, namely TinyBERT (Jiao et al. 2020), MiniLM (Wang et al. 2020 and the one-shot NAS method EfficienBERT (Dong et al.] 2021). on the SQuAD dataset. The results presented in Table 4 Despite having fewer parameters than TinyBERT-4, Minil.M6,, and Efficient+++ the W-PCA-Small model outperforms these meth- ods in terms of both EM and Fl scores on both the SQuAD vl.1 and SQuAD v2.0 datasets. This observation demonstrates the robust adapt- ability of the investigated models across diverse datasets.
Table 4: Results on SQuAD dev sets. ": our imple- mentation.
SQuAD v1.1
SQuAD v2.0
Model
#Params
EM/F1
EM/F1
BERT-base
108.9M
80.8/88.5
-f-
BERT-base"
108.9M
80.7/88.2
75.7/78.7
TiRyBERT-4
14.5M
72.7/82.1
68.2/71.8
MiniLM-6
22.9M
J/-
-/72.7
EfficicmBERT++
16.0M
78.3/86.5
73.0/76.1
W-PCA-Tiny
9.6M
74.6/83.5
69.0/72.1
W-PCA-Small
15.6M
78.4/86.7
73.3/76.8
'A reduction of 0.34k lbs and 3.4k Ibs of CO2 emissions for every 100-fold and 1,000-fold, respectively Strubell al. 2019).

---

Published as a conference paper at ICLR 2025
Table Comparison results of W-PCA and its product counterparts proxies on the GLUE dev set.
Proxy
#Params
QNLI
MRPC
SST-2
CoLA
STS-B
MNLI-m
RTE
QQP
AVG
#Params
15.7M
89.3
88.8
90.7
43.8
83.6
82.6
76.1
87.5
80.3
V-PCA
15.6M
89.9
91.4
92.7
39.4
84.9
82.9
76.0
88.9
80.8
W-PCA
15.6M
90.8
90.5
92.8
44.0
85.3
82.9
76.1
88.8
81.4
6.5
ABLATIONS
In order to investigate the effects of each component of W-PCA on the experimental results, we performed ablation experiments. Specifically, utilized the components of W-PCA, as described in Equation (8) where the first component is the number of parameters (#Params) and the second component
V-PCA value (defined in Equation (7p). as fitness values for the genetic algorithm to explore the optimal network structure in Section 6.2. We then compared the performance of the discovered network structures with W-PCA.
The results, presented in Table demonstrate that by multiplying the number of parameters with the V-PCA value and using W-PCA as the zero-shot evaluation metric, the performance of the searched networks significantly improves compared using either #Params V-PCA alone the evaluation metric.
Encouragingly, the incorporation of extra feature does not necessitate significant rise in compu- tational time, thereby rendering the multiplication approach highly efficient. For further ablations, please refer Appendix
7 CONCLUSION
In this paper, we propose W-PCA and significantly improving the utilization of global energy resources., novel zero-shot NAS method specifically designed for lightweight language models. In the ranking correlation experiments conducted on the search space of FlexiBERT, W-PCA achieves Kendall т score that surpasses the previous method by 0.220 and Spearman p score that surpasses the previous method by 0.334. In the accuracy experiments conducted on GLUE and SQuAD, W-PCA not only achieves the highest score, but also significantly improves search efficiency. On the GLUE test set. W-PCA improves search efficiency by best-performing one-shot NAS method, with an average score improvement of 0.3. On the GLUE dev set, W-PCA improves search efficiency by 2,000 times and achieves average score improvement of 0.9 compared to the previous best-performing one-shot NAS method. In future work, we will extend 2024a Li al. 2024b e
hundredfold compared the previous
our approach more compression tasks (Dong Our work contributes to the advancement of NAS methods for lightweight language models, enabling the design and optimization of efficient and effective systems for natural language processing.
Dong
2024b).
Limitations This work focuses on the ranking correlation tasks commonly addressed in prior zero- shot NAS methods and on NLU tasks relevant to lightweight models. However, recent language model research increasingly centers on generative models with over 1B parameters. In Appendix we further discuss potential extensions related to these large-scale generative models.
ACKNOWLEDGMENTS
Special thanks to Professor Yajun Ha from ShanghaiTTcc Tech University. During the resubmission of this paper ICLR, he helped us summarize the contributions of the paper and suggested more suitable title.
REFERENCES
Mohamed S Abdelfauta,, Abhinav Mehrotra, Lukasz Dudziak, and Nicholas Donald Lane. Zero-cost proxies for lightweight In International Conference on Learning Representations, 2020.
OOur implementation of W-PCA is available: http:://gthuhhcoo//22234AAAA
10

---

Published as a conference paper at ICLR 2025
Maximiliana Behnke and Kenneth Heafield. Losing heads in the lottery: Pruning transformer attention in neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2664-26744 2020.
Luca Celotti, Ismael Balafrej, and Emmanuel Calvet. Improving zero-shot neural architecture search with parameters scoring. 2020.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See htlps://vicuna. Imsys. org (accessed 14 April 2023), 2023.
Kevin Clark, Minh-Thang Luong, Quoc Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003. 2020.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open
instruction- tuned 1lm, 2023. URL bttps: : www wwt.daaaaabiccc.o
www.cooo//c........ wwt.c///wwo..oo//o//oooooooo comme www. httpaa/w......aaaaeeaaoooottoot-
mt/dolll-firsse
11m
Jacob Devlin, Ming-Wa Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, 2019.
Chenhe Dong, Guangrun Wang, Hang Xu, Jiefeng Peng, Xiaozhe Ren, and Xiaodan Liang. Effi- cientbert: Progressively searching multilayer perceptron via warm-up knowledge distillation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 1424-1437, 2021.
Peijie Dong, Xin Niu, Lujun Li, Linzhen Xie, Wenbin Zou, Tian Ye, Zimian Wei, and Hengyue Pan.
Prior-guided one-shot neural architecture search. arXiv preprint 2rXiv:2206.13333 2022.
Peijie Dong, Lujun Li, and Zimian Wei. Diswot: Student architecture search for distillation without training. In CVPR, 2023a.
Peijie Dong, Lujun Li, Zimian Wei, Xin Niu, Zhiliang Tian, and Hengyue Pan. Emq: Evolving training-free proxies for automated mixed precision quantization. arXiv preprint 105554 2023b.
Peijie Dong, Lujun Li, Zhenheng Tang, Xiang Liu, Xinglin Pan, Qiang Wang, and Xiaowen Chu. Pruner-zero: Evolving symbolic pruning metric from scratch for large language models. In ICML, 2024a.
Peijie Dong, Lujun Li, Yuedong Zhong. Dayou Du, Ruibo Fan, Yuhan Chen, Zhenheng Tang. Qiang Wang, Wei Xue, Yike Guo, et al. Stbllm: Breaking the 1-bit barrier with structured binary 1lms. arXiv preprint arXiv:2408.0/800 2024b.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama herd of models. arXiv preprint 24Xiv:2407.2 2024.
Jiahui Gao, Hang Xu, Han Shi, Xiaozhe Ren, LH Philip, Xiaodan Liang, Xin Jiang, and Zhenguo Li. Autobert-zero: Evolving bert backbone from scratch. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 10663-10671, 2022.
Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanic Tellex. Openwebtext corpus, 2019.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2024.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng. Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In Computer Vision-ECCT 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XVI 16, pp. Springer, 2020.
11

---

Published as a conference paper at ICLR 2025
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:2212.09689 2022.
Yiming Hu, Xingang Wang, Lujun Li, and Qingyi Gu. Improving one-shot nas with shrinking-and- expanding supernet. Pattern Recognition, 2021.
Zi-Hang Jiang, Weihao Yu, Daquan Zhou, Yunpeng Chen, Jiashi Feng, and Shuicheng Yan. Convbert: Improving bert with span-based dynamic convolution. Advances in Neural Information Processing Systems, 33:12837-12843 2020.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4163 4174, 2020.
Nikita Klyuchnikov, Ilya Trofimov, Ekaterina Artemova, Mikhail Salnikov, Maxim Fedorov, Alexan- der Filippov, and Evgeny Burnaev. Nas-bench-nlp: neural architecture search benchmark for natural language processing. IEEE Access, 1145736-457444 2022.
Guihong Li, Yuedong Yang, Kartikeya Bhardwaj, and Radu Marculescu. Zico: Zero-shot nas via inverse coefficient of variation on gradients. In The Eleventh International Conference on Learning Representations, 2023a.
Lujun Li. Self-regulated feature learning via teacher-fiie free feature distillation. ECCV 2022, 2022.
Lujun Li and Zhe Jin. Shadow knowledge distillation: Bridging offline and online knowledge transfer. Advances in Neural Information Processing Systems, 2022.
Lujun Li, Peijie Dong, Anggeng Li, Zimian Wei, and Yang Ya. Kd-zero: Evolving knowledge distiller for any teacher-student pairs. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b.
Lujun Li, Peijie Dong, Zimian Wei, and Ya Yang, Automated knowledge distillation via monte carlo tree search. In ICCV, 2023c.
Lujun Li, Yufan Bao, Peijie Dong, Chuanguang Yang, Anggeng Li, Wenhan Luo, Qifeng Liu, Wei Xue, and Yike Guo. Detkds: Knowledge distillation search for object detectors. In ICML, 2024a.
Lujun Li, Peijie, Zhenheng Tang, Xiang Liu, Qiang Wang, Wenhan Luo, Wei Xue, Qifeng Liu, Xiaowen Chu, and Yike Guo. Discovering sparsity allocation for layer-wise pruning of large language models. In NeulPS, 2024b.
Lujun Li, Haosen Sun, Shiwen Li, Peijie Dong. Wenhan Luo, Wei Xue, Qifeng Liu, and Yike Guo. Auto-gas: Automated proxy discovery for training-free generative architecture search. ECCV, 2024c.
Lujun Li, Zimian Wei, Peijie Dong, Wenhan Luo, Wei Xue, Qifeng Liu, and Yike Guo. Attnzero: efficient attention discovery for vision transformers. In ECCV, 2024d.
Wei Li, Lujun Li, Mark Lee, and Shengjie Sun. Als: Adaptive layer sparsity for large language models via activation correlation assessment. In NeulPS, 2024c.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In International Conference on Learning Representations, 2018.
Xiaolong Liu, Lujun Li, Chao Li, and Anbang Yao. Norm: Knowledge distillation via n-to-one representation matching. arXiv preprint arXiv:2305.1 13803, 2023.
Joe Mellor, Jack Turner, Amos Storkey, and Elliot J Crowley, Neural architecture search without training. In International Conference on Machine Learning, pp. 7588-7598. PMLR, 2021a.
Joseph Mellor, Jack Turner, Amos Storkey, and Elliot J. Crowley. Neural architecture search without training, 2021b.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one?, 2019.
12

---

Published as a conference paper at ICLR 2025
R OpenAl. Gpt-4 technical report. arXiv, pp. 2303-08774,
Xuran Pan, Xuan Jin, Yuan He, Shiji Song, Gao Huang, et al. Budgeted training for vision transformer. In The Eleventh International Conference on Learning Representations, 2023.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv: arXiv:/606.085 166.05220, 2016.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pp. 4780-4789, 2019.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:/910.01108, 2020.
Aaron Serianni and Jugal Kalita. Training-fres free neural architecture search for RNNs and trans- formers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2522-2540, Toronto, Canada, July 2023, Asso- ciation for Computational Linguistics. doi: 10. 1...8663/112222.acl-- acl-long.144 142. URL https acl-logg.. 142
1/aclantholooe logy-
Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hes- tness, and Nolan Dey. SlimPajama: A 627B token cleaned and dedu- plicated version of RedPajama.
https://www.com www.:eeerrrbbreeeeeeeeeeeeeeeeee
s1
a- toke and hew.d-/euuplccttteeeeerrroiooooi ofee jama
2023. URL httts:://huggiggggetteee cttps/ce..ood/ateeeeeeee http:///eeeeeeetaeeeeee////ee//a
667B
Emma Strubell, Ananya Ganesh, and Andrew Mccallum. Energy and policy considerations for deep
learning in nip. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisttiss, pp. 3645-3650, 2019.
Haosen Sun, Lujun Li, Peijie Dong, Zimian Wei, and Shitong Shao. Auto-das: Automated proxy discovery for training- free distillation- aware architecture search. ECCV, 2024.
Yanan Sun, Bing Xue, Mengjie Zhang, and Gary G Yen. Evolving deep convolutional neural networks for image classification. IEEE Transactions on Evolutionary Computation, 24(2):394 407, 2019.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: compact task-agnostic bert for resource-limited devices. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2158-2170, 2020.
Hidenori Tanaka, Daniel Kunin, Daniel L. K. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow, 2020.
Shikhar Tuli, Bhishma Dedhia, Shreshth Tuli, and Niraj K Jha. Flexibert: Are current transformer architectures too homogeneous and rigid? Journal of Artificial Intelligence Research, 77:39-70, 2023.
Iulia Ture, Ming-Wei Chang. Kenton Lee, and Kristina Toutanova. Well-read students learn better: 2019.
On the importance of pre-training compact models. arXiv preprint
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 5797-5808, 2019.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. Proceedings of the 2018 EMNLP Workshop BlackboxNI.P: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November doi: 18.18653/v 11/5/3/1//III/ 1/W18- 5446. URL November 2018. Association for Computational Linguistics.
http:////aaaathohooooooooo org /W18 5446
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33:5776-5788, 2020.
13

---

Published as a conference paper at ICLR 2025
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannanch Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint ar arXiv:2212.10566 2022a.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Benchmarking generalization via in-context instructions on 1,600+ language tasks. arXiv e-prints, pp. arXiv-2204, 2022b.
Zimian Wei, Lujun Li, Peijie Dong, Anggeng Li, Menglong Lu, Hengyue Pan, and Dongsheng Li. Auto-prox: Training-free vision transformer architecture search via automatic proxy discovery. AAAI, 2024.
Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, and Song Han. Lite transformer with long-short range attention. In International Conference on Learning Representations, 2019.
Jin Xu, Xu Tan, Renqian Luo, Kaitao Song, Jian Li, Tao Qin, and Tin Liu. Nas-bert: task-agnostic and adaptive-size bert compression with neural architecture search. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 1933-19942 2021.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre- pre-trained transformer language models. arXiv preprint arXiv:2205.0/00 2022.
Qinqin Zhou, Kekai Sheng, Xiawu Zheng, Ke Li, Xing Sun, Yonghong Tian, Jie Chen, and Rongrong Ji. Training-free transformer architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10894-10903, 2022.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pp. 19-27, 2015.
Barret Zoph and Quoc Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations, 2016
14

---

Published as a conference paper at ICLR 2025
Table 6: Performance comparison on the GLUE test set for model composed of identical blocks.
Block Type
iParams
QNLI
MRPC
SST-2
CoLA
STS-B
MNLI-m/om
RTE
QQP
AVG
BERT (Devlin 2019
24.8M
90.9
89.0
92.9
46.7
87.2
84.4/83.5
71.0
79.7
MobileHERT (Sun al. 2020
10.4M
87.2
87.7
91.4
31.8
84.9
81.5/81.4
69.6
70.8
76.3
Transssaffrrrr (Wu al. 2019
34.6M
90.2
89.2
92.5
36.4
86,4
83,1/82.3
74.0
70.8
78.3
ConvBERT (iiag ct 2020
37.4M
89.7
89.4
91.9
36.0
85.8
82.5/82.0
72.1
70.7
77.8
A
WHY BERT AND MOBILEBERT?
As shown in Table 6 we have attempted to train a model by composing 12 blocks of the same type, with dimension of 528 for FFN. The results indicate that BERT and MobileBERT blocks outperform other blocks in terms of the unit parameter performance. Therefore, we have included these two fundamental blocks in search space. In fact, if construct supernet described in Section H.3, the optimal network structure selected by the genetic algorithm will also only choose BERT and MobileBERT blocks, without considering any other types of blocks.
B TRAINING DETAILS OF THE FLEXIBERT SEARCH SPACE
Table 7: The FlexiBERT search space comprises total of 10,621,440 architectures.
Architecture Element
Hyperparameters Values
Embedding dimension
(128, 256)
Number Encoder Layers
(2, 4)
Type of attention operator
[self-attention, linear transform, span- based dynamic convolution]
Number of operation heads
(2, 4)
Hidden dimension
[512, 1024)
Number
stacks
(1,
if self-attention
(scaled dot-product, multiplicative)
Attention
if linear transform
(discrete Fourier, discrete cosine)
operation
dynamic convolution
convolution kernel size: (5, 9]
All transformer architectures within the search space trained on TPUv2s with 8 cores and 64 GB of memory using Google Colaboratory. The entire process of pretraining and finetuning the benchmark took approximately 25 TPU days. For the evaluation of training-free metrics, 2.8 GHz Intel Cascade Lake processors with either 16 or 32 cores and 32 GB of memory were employed.
In terms of hyperparameter settings, except for setting the training steps 100,000 during the pre-training phase, everything else is the same as training ELECTRA-Smin Specifically, during the pre-training phase, the generator size multiplier is warmup step is 10,000, the learning rate is 5e-4, and the batch_ size is 128. During the fine-tuning phase, the learning rate 3e-4, the layerwise Ir decay 0.8, the warmup fraction is 0.1, the attention dropout is 0.1, and the batch_size is 32. For the RTE and STS tasks, 10 epochs are trained, while for other tasks, epochs are trained. Both during pre-training and fine-tuning phases, the learning rate decay is linear, the vocabulary size is 30522, the dropout is 0.1, the weight decay value is 0.01, the value for the Adam optimizer is Ic-6, 31 value is 0.9, and 3g value is 0.999.
1/4, the mask percentage is set to 15%, the
C ABLATIONS OF RANKING EVALUATION
To examine the stability of zero-shot metries, we conducted a series studies to investigate the effects of random architecture initialization (Figure and varying batch inputs (Figure ( on the evaluation zero-shot metrics within the FlexiBERT search space. The range fluctuations observed in various zero-shot metrics in neural network model demonstrates that, while the number of parameters remains unaffected, other zero-shot metrics exhibit varying degrees of fluctuation depending the initialization of parameters and batch inputs. The figure indicates that PCA alone, serving as a proxy metric, shows good stability, and W-PCA exhibits the same level of stability as PCA due multiplication by constant parameter that does alter the magnitude of the original
15

---

Published as a conference paper at ICLR 2025
Synaptic Diversity
Synaptic Saliency
165
Activation Distance
Head Importance
00175
0.000
0.20
00125
10015
00000
Confidence
600
140
éparams
144
350
420
3.5
440
30-
350
25
200
20-
250
200
Figure 5: Evaluation of zero-shot metrics with various initialization weights FlexiBERT search space. Ten architectures are randomly sampled from the search space, representing decile ranges of the GLUE score (c.g., 0-10%, 10-20%, 90-100% To ensure robustness, ten different random seeds are employed for weight initialization.
Synaptic Diversity
Synaptic Sa lieecy
Activation Distance
Head Importance
0.5
0.011
6.20
00150
3.0
0.0015
7.5
7.0
003.0
5.5
6.0
5.5
00005
5.0
Head Seftmax Confidence
PCA
#params
W-PCA
0.25
3.0
0.231
16
Figure 6: Evaluation of zero-shot metrics with various minibatch inputs in the FlexiBERT search space. Ten architectures are randomly sampled from the search space, representing decile ranges of the GLUE score (e.g., 0-10%, 10-20%. .... 90-100%). The same ten minibatches, each with size of 128, are randomly chosen from the OpenWebTee dataset for each architecture and metric.
proxy metric's fluctuation. When compared other zero-shot metrics, PCA and W-PCA exhibit slightly better stability than Activation Distance (Mellor et al. (021a) and Head Softmax Confidence (Serianni & Kalita 2023), and significantly better stability than Synaptic Diversity (Zhou 2022). Synaptic Saliency (bbelfattah et al.) 2020). and Head Importance (Serianni & Kalita| 2023). These findings demonstrate that our proposed method not only achieves superior ranking evaluation but also demonstrates improved stability.
16

---

Published as a conference paper at ICLR 2025
D ROBUSTNESS WITH DIFFERENT 7j's
In our NLU task experiments, when train- ing networks composed 12 identical BERT or MobileBERT blocks, we observed an ex- tremely uneven distribution of principal com- ponents. Specifically, the largest eigenvalues, when sorted, are predominantly concentrated at the beginning, while the remaining eigen- values are close to zero. The dimensions re- quired for each layer to achieve different ry val- ues when training these models are shown in Table X From the table, it is evident that the BERT blocks exhibit more uneven distribution compared to the MobileBERT blocks, as only 54.2% of the dimensions are needed to reach a 0.999 principal component contribution rate. However, setting 7/ to 0.999 would result a loss of distinction for MobileBERT blocks, as they would only have integer values between 125 and 132. Since our search space includes net- works composed of both BERT and MobileBERT blocks, we compromised by setting 7) to 0.99.
Table 8: Principal component dimensions at differ- ent 17 values for neural network model composed of 12 identical blocks during training.
Model
7= 0.9
0.99
0.999
BERT
91 (17.2%)
1.59 (30.1%)
286 (54.2%)
528
MobileBERT
58 (43.9%)
101 (76.5%)
125 (94.7%)
132
For the ranking correlation experiments, each block has more combination possibilities. We adjusted different values and obtained the following ranking correlations in Table 9 As shown the table, both V-PCA and W-PCA maintain high rank correlation regardless of the 1 value, demonstrating the robustness of the y) variable in the rank correlation experiments. Ad- ditionally, W-PCA consistently outperforms V- PCA in both Kendall's and Spearman's 's correlations across different values, further indicating that incorporating the number of parameters as a factor in the proxy enhances the stability of the rank correlation results.
Table Results obtained from varying 7 values in rank correlation experiments.
Proxy 7 0.9
"
0.99
7 0.999
Vanilla PCA
0.466
0.677
0.449
0.667
0.433
0.653
W-PCA
0.526
0.698
0.513
0.689
0.499
0.688
E IMPLEMENTATION DETAILS OF GENETIC ALGORITHM
The genetic algorithm is a widely used classic algorithm for solving combinatorial optimiza- tion problems. In Figure 7 illustrate the flowchart of the genetic algorithm. Early NAS algorithms (Real et al.) 2019; Sun et al., 2019) often encoded each individual neural net- work and then trained them from scratch. To efficiently identify the best combination of lightweight BERT blocks, we encoded and solved the combination using genetic algorithm.
paloudate
several generations loop
Diagram of genetic algorithm. In order to effectively solve practical prob- lems using a genetic algorithm, it is crucial to define a suitable method for encoding the solutions. Additionally, in each genera- tion, the fitness of every individual should be calculated, and only the most excellent individuals should be selected for the appli- cation of crossover and mutation operators. These operators are used to generate the individuals that will make up the next gen- cration.
Figure 7:
E. ENCODE
When there are m layers in the combination, we use an integer array of length m to encode the selection method for each layer. Each integer in the array rep- resents the type of block chosen for that layer. Taking the 12-layer model of W-PCA-Small as example, if the number at a given position is less than or equal to 5, it indicates the selection of a BERT-type block; other- wise, a MobileBERT-tye block is chosen. Each integer represents the dimension of the hidden dimension. Let x be the integer at the current layer position, then the hidden dimension is calculated as 132 (x%6 1).
17

---

Published as a conference paper at ICLR 2025
E.2 CROSSOVER
Algorithm 1: Crossover operation in genetic algorithm
Input: Parent 1 encoding pl, Parent 2 encoding p2
Output: Offspring encoding
Function Crossover p
Create an empty offspring encoding child
for < to length(p1) do
if random number < 0.5 then
Add the gene from the corresponding position the parent encoding to the
offspring encoding
child[i] - p
else
Add the gene from the corresponding position in the parent 2 encoding to the offisrinn encoding
child [i] < p2[i]
end
11
end
12
return child
We employ the single-point crossover method as outlined in Algorithm | to produce offspring encodings. In each generation, parents randomly chosen from the top 10 individuals, and offssring are generated through crossover until the desired number of offspring is obtained.
E.3 MUTATION
We set the mutation probability 0.1. When the current position mutation is triggered, the current position will randomly mutate into another integer within the selection range.
F KD Loss FUNCTION
The distillation loss function of EfficientBERT (Dong et al. 202I) forms the basis of our approach. For the student model, define Ci the loss for the multi-head attention (MHA) output and Li as the loss for the feed-forward network (FFN) output in the m-th layer. The embedding loss, represented by Lembd, is also included. These losses calculated using the mean squared error (MSE) as follows:
Cittt MSE A W. AT) o:
Ciidd = MSE W A. H
(9)
Lemba = MSE W E
Here, A and H represent the outputs of the MHA and FFN layers, respectively, in the i-th layer of the student model. Similarly, A and H represent the outputs of the MHA and FFN layers, respectively, in the j-th layer of the teacher model corresponding the i-th layer of the student model.
For our fixed teacher model, BERT-base, which comprises layers, a one-to-one sequential corre- spondence exists between the layers of the student and teacher models when both models have 12 layers. However, in the case of student model with only 6 layers, the correspondence remains one- to-one, but with 2-layer interval. This implies that the first layer of the student model corresponds to the second layer of the teacher model, and so forth, until the sixth layer the student model aligns with the twelfth layer of the teacher model.
The trainable matrices W.. WA. and W. are used to adjust the dimensionality of the student and teacher models. Additionally, define Lpred the prediction loss, which is calculated using soft cross-entropy (CE):
Lpred CF(28 of
(10)
18

---

Published as a conference paper at ICLR 2025
McDkuBERT
SERT-000
BERT-bas
MccRuBERT
MobNMSER
AADBUBERT
(4) AParriT
d 660
860
d 528
6= 596
d 586
d= 296
d= 998
6 828
6 928
d= 898
4 396
BERT bous
BERT-base
BERT--2000
MoolkBBRT
BERT-O000
MOOIMBERT
MONE
MoblwBERT
MADSABERT
Mode
VANGA
528
d 930
529
d= 628
c= 528
d= 250
18 VN- (T
BERT-base
McBRIBERT
(d) W P0A
Figure 8: Visualizations of the searched architectures, where represents the hidden dimensions.
Here, z represents the predicted logit vector.
The total loss a combination of the above terms:
Σ Cors + Chiads) + Lembd + Thrred
(11)
The coefficient y is used to control the contribution of the predicted loss. set at 0 during the pretraining phase and during the fine-tuning phase.
G VISUALIZATION OF ARCHITECTURES
Figure illustrates the schematic diagram of the network structure. It is observed that all models preferentially choose MobileBERT as the candidate block, suggesting that MobileBERT is better suited for lightweight language models in comparison to BERT-base. Furthermore, with the exception of the searched model that solely relies on parameter count as the search evaluation metric, the candidate blocks of MobileBERT are predominantly located the higher layers, indicating that this architecture is more adept at analyzing high-level semantic information.
H MORE ABLATIONS ON ACCURACY COMPARISION
H.1 DIFFERENT INITIALIZATIONS
Table 10: Statistical significance tests based on average scores from runs with different random seed initializations on the GLUE dev set.
Proxy GLUE
#Params
V-PCA
W-PCA
80.3+0.0
80.9+0.09
81.5+0.14
To validate the stability of each component of W-PCA, we conducted multiple runs using #Params, V-PCA, and W-PCA as proxies. The results are presented Table [0] As shown the table, under different weight initializations, the structures identified using #Params across runs since #Params does not change. However, when V-PCA and W-PCA are used as proxies, the genetic algorithm identifies different structures, resulting in varying GLUE scores after training. On average, W-PCA demonstrates a clear advantage over V-PCA. and both W-PCA and V-PCA outperform #Params. This further validates the effectiveness of the proposed proxy.
proxy remain identical
19

---

Published as a conference paper at ICLR 2025
Table 11: Performance comparison of larger-scale models on the GLUE test set.
Model
@Paramms
QNLI
MRPC
SST-2
CoLA
STS-B
MNLI-m/mm
RTE
QQP
AVG
TinyBERT-6 Jiao
2020
67.0M
89.8
89.0
92.0
38.8
83.1
83.8/83.2
65.8
71.4
77.4
RfficienBERR DDon et al. 2021
70.1M
90.4
89.0
92.6
46.2
83.7
84.1/83.2
67.7
74.4
78.7
W-PCA-Large
66.9M
90.9
88.7
93.0
40.0
87.5
84.6/83.3
75.6
71.5
79.5
H.2 COMPARISON OF LARGER--IZZER MODELS
In the main body of the paper. we primarily focus on lightweight language models with parameter sizes of approximately 15M and 10M. To investigate the applicability of W-PCA on larger language models in search processes, we carried out experiments by utilizing larger model that expanded the size of the search space as described in Section 6.2.1 Specifically, we doubled the hidden_ hidden_size size and the hidden__immenion dimension of n candidate dimensions. Moreover, we increased the parameter limit the genetic algorithm to 67M, resulting in the creation of our W-PCA-Large model. Our model, presented in Table 11, outperforms TinyBERT-6 (Jiao et al., 2020 and EfficientBERT (Dong et al., (202T) models of similar scale in terms of average GLUE score, despite having slightly lower parameter count. This indicates that W-PCA also exhibits strong adaptability in larger search spaces.
languug
H.3 COMPARISON WITH ONE-SHOT NAS
Table 12: Comparison of search and training stages for one-shot and zero-shot methods.
Method
Search Stage
Training Stage
Pretrain the
Finetune the supernet for neural network structure
Finding the optimal
one-shot
Re-pretrain the optimal network structure
Fine-tune for each downstream task
supernet
downstreum tasks
using genetic algorithm
shoo
Finding optimal neural network structure using, genetic algoniti
In Section 4 refer to the composition method of the supernet in the one-shot NAS method SPOS
(Guo et al. 2020), and construct the combination method of the lightweight BERT model in our zero-shot NAS search space. In order compare its performance with the one-shot NAS, we now build real supernet, as shown in Figure 4 with a total of m layers and 2 x n candidate blocks in each layer. Before searching for the optimal structure using a genetic algorithm, we performed one round of pre-training and fine-tuning based on the SPOS method. During each batch, a random path selected from the (2 x n)m combinations for forward propagation and backward parameter updates. After completing the pre-training and fine-tuning process, we proceeded with the workflow described in the zero-shot NAS experiment. This involved searching for the optimal network architecture on the supernet using genetic algorithm, and then re-pretraining and fine-tuning this optimal architecture for downstream tasks.
Implementation Details. We first pretrain the supernet on English Wikipedia (Devlin et al. 2019) and BooksCorpus (Zhu et al., 2015). then utilize 90% of the training set from each GLUE task for fine-tuning. We reserve the remaining 10% of the MNLI task evaluate the accuracy of architectures in the search. During the pre-training and fine-tuning process, the number of epochs set to 10, and the batch_ match_sizeiin is set to 256 for both. The learning rate for pre-training is set Ic-4, and the learning rate for fine-tuning is set to 4c-4. The optimizer, weight decay, and learning rate adjustment strategy are the same in the training section. The loss function used is still the MSE loss function described in Appendix F
Results Analysis. As displayed in Table 13 despite extensive computational resources devoted to the one-shot NAS search, the performance enhancement for various- models of W-PCA is not significant. To further investigate the reasons behind this, we outline the steps of one-shot NAS and zero-shot NAS in Table [2] Both approaches involve finding the optimal network structure through search stage, followed by training this structure the subsequent training stage. In the search stage of one-shot NAS, supernet training is necessary, whereas zero-shot NAS only requires temporary construction of a neural network at each sampling step to compute the zero-shot proxy. Consequently, the memory overhead for zero-shot NAS is minimal. Contrasting with zero-shot NAS, one-shot NAS incurs additional time overhead due to pre-training the supernet on the corpus and performing global fine-tuning downstream tasks. Additionally, one-shot NAS possesses extra pre-trained weights
20

---

Published as a conference paper at ICLR 2025
Table 13: Comparison of zero-shot and one-shot methods on the GLUE test set in the same search space. "Time" also refers to the GPU time consumption in the NAS stage.
Model
Type
#Params
Time
QNLI
MRPC
SST-2
CoLA
STS-B
MNLI-m/mm
RTE
QQP
AVG
W-PCA-Tiny
zero-shot
9.6M
0.4
88.7
87.6
91.9
27.4
84.8
81,1/79.8
71.1
70.3
75.9
one-shot
9.7M
24
89.2
87.5
92.3
28.9
83.7
81,4/80.5
71.4
70,5
76.2
W-PCA-Small
zero-shot
15.6M
0.5 d
90.3
88.7
91.5
38.4
86.4
82.8/82.2
73.8
70.8
78.3
one-shot
15.6M
28
90.3
88.9
92.5
36.1
86.7
83.7/82.5
74.4
70.6
78.4
Table 14: Comparison of zero-shot NAS methods on the GLUE dev set within the EfficientBERT search space.
Proxy
#Params
Time
QNLI
MRPC
SST-2
CoLA
STS-B
MNLI-m
RTE
QQP
AVG
Synaptic Diversity Zhou et al. 2022 Head Confidence
15.6M
1.0d
84.1
85.2
86.1
21.1
76.5
77.2
66.5
78.2
71.9
(Senammi Kalia 2023
15.7M
0.8
83,1
84,4
86.8
21.8
80.9
76.3
66.1
84.6
73.0
Softmax Confidence (eeannn Kahia
77.6
W-PCA
15.6M
86.0
85.3
90.1
24.4
76.5
65.8
86.2
74.3
apart from the different searched network structures. After the network structure searched by zero-shot NAS, no pre-training weights are used, and the weights are randomly initialized, resulting in relatively minor effects on accuracy. Therefore, zero-shot NAS remains the most cost-effective search solution.
H.4 EXPERIMENTS ON EFFICIENTBERT SEARCH SPACE
Previous zero-shot NAS studies have primarily assessed performance on NAS Benchmark datasets, using rank correlation to evaluate the effectiveness of these zero-shot proxies. To facilitate com- parison with other zero-shot proxies applied to Transformer models in text tasks, we designed the search space described in Section and compared method within this context. To further ensure robustness, we conducted additional comparisons using the EfficientBERT search space, with the results shown in Table [4] The data demonstrate that our proposed zero-shot proxy maintains significant advantage in accuracy across various search spaces.
CAUSAL LANGUAGE MODELING TASK
To validate our performance more sufficiently, we further transfer our method to causal language modeling (CLM) task.
I.1 SETUP
Searching. We use some 1B-3B parameter OPT Zhang et al. 2022) and LLaMA3 (Dubey et al. 2024) models baseline models, keeping the number of layers unchanged while searching for the optimal combinations of MHA blocks and FFN dimensions in each layer of the models. For fair comparison, we set the maximum parameter limit for the searched models be the same as that of the baseline models.
Pretraining, We pretrain the models for causal language modeling (CLM) using a randomly selected 1 % of texts from the SlimPajama dataset (Soboleva et al. 2023). For this task, we train the models generated by each zero-shot proxy on 8 NVIDIA VI0O GPUs, with total batch size of 64 for epochs. A cosine annealing schedule is applied to the learning rate, starting at 5 x 10-1, Optimization performed using the Adam optimizer with a weight decay of 0. 1. accommodate the differences in configurations, such as the tokenizer and training loss, between OPT and LLaMA3, we train two separate models for each proxy.
Instruction finetuning. To obtain models suitable for chat applications, we finetune each pretrained model using the instruction dataset tntabricks-dolly--. (Conover et al. 2023) with the KD method MiniLLM (Gu et al., 2024).. Specifically, to supervise other models. The training data and strategies employed are consistent with those of M.n.L..M.
OPT-13B and 3.aMA3.2-11 IIB the teacher model
21

---

Published as a conference paper at ICLR 2025
Table 15: Evaluation results referring the average GPT-4 feedback scores (GPT4) and Rouge-L scores (R-L) obtained from 5 runs.
DollyEval GPT4
Selfinst
VicunaEval GPT4
S-NI
UnNT
Teacher
Proxy
#larams
R-L.
GPT4
R-L.
R-L
R-L
R-L
OPT-1.3B (baseline)
60.7
26.7
47.0
14.8
50.6
17.9
28.6
33.4
Synaptic Diversity Zhou
2022
1.3B
60.4
26.9
47.3
15.2
51.8
17.6
29.1
34.2
OPT-13B
Head Confidence
Serianni & Kalina 2023
1.38
61.5
27.3
48.0
15.3
51.5
18.1
29.5
34.3
Softmax Confidence (erianm & Kalita 2023
1,3B
61.2
27.1
47.5
14,6
50.8
18.2
28.9
33,9
W-PCA
1.3B
62.0
27.6
47.9
16.1
52.3
19.0
29.2
35.0
OPT-2.7B (baseline)
63.2
27.4
52.7
17.2
55.9
19.1
30.7
35.1
Synaptic Diversity Zhou al. 2022
2.7B
63.8
27.5
52.9
17.8
56.2
18.9
31.5
35.8
OPT-13B
Head Confidence (Semanm & Kalita
2023
2.7B
64.0
27.8
53.2
17.5
56.5
19.4
31.6
35,9
Softmax Confidence (Serianmi & Kalita
2023
2.7B
63.2
27.8
53.0
17.3
56.1
19.2
31.0
35.4
W-PCA
2.7B
64.5
28.3
53.9
18.1
57.2
20.1
32.1
36.2
LLLMM3.2 3B (baseline)
74.0
29.8
69.1
23.6
64.6
25.2
36.2
43.9
Synaptic Diversity (Zhou et al.
2022
3B
73.8
30.5
70.1
23.8
65.0
25.4
36.7
44.8
LLaMA3.2-1IB
Head Confidence (Semaam & Kalit 2023
3B
74.4
30.9
71.4
24.2
65.2
26.2
37.1
45.6
Softmax Confidence (Serianm & Kalita 2023
38
74.1
30.1
70.4
23.6
64.9
25.9
37.2
44.3
W-PCA
3B
75.1
31.4
71.8
24.7
65.9
26.8
37.6
45.2
Each model required approximately 20 GPU days for pretraining and around 28 GPU hours for finetuning in our experiments.
Quantitative evaluation. perform a numerical evaluation on five instruction-follooing datasets:
EvllyEval: This a 500-sa test set that we extracted from the databricks-dolll-15k dataset.
SelfInst (Wang et al.
2022a): : A user-oriented instraction-follovin set comprising 252
samples.
VicunaEval (Chiang et al. in the Vicuna project.
2023
The evaluation includes 80 challenging questions used
S-NI: The test set of Superr-aturallnnstuuttooo 9,000 samples across 119 tasks.
(Wagg et al.
2022b which consists of
UnNI: The set of Unnaturallnstrrutions (Honovich et al. 2022 which contains 60,000 samples.
1.2 QUANTITATIVE RESULTS ON LANGUAGE BENCHMARKS
We evaluated the performance of the W-PCA model by comparing it with other models trained using MiniLLM. As shown in Table [5] across various model sizes, W-PCA consistently identifies architectures that outperform the baseline all evaluation metrics. Compared other zero-shot NAS methods, W-PCA leads in 6 out of metrics for the 1.T-B size, all metrics for the OPT-2.7B size, and out of metrics for the LLaMA3.2 33.2-BB size. These results demonstrate the effectiveness of our method in transferring performance to CLM tasks.
22


Reviews:
[{"review_id": "iECdR4bbuO", "reviewer": "Reviewer_Z4ps", "summary": "This paper introduces W-PCA, a novel zero-shot NAS method specifically tailored for light weight language models.", "questions": "1.Could you provide results of some other methods based on your BERT models?\n\n2.I am curious about the applicability of W-PCA on current LLMs(e.g. LLama).\n\n3.How do you calculate GPU days in table 3 and estimate that AutoBERT-Zero-small requires around 1,000 GPU days on GLUE dev set?\n\n4.You claim in the conclusion that W-PCA achieves a Kendall score and a Spearman Score that surpass previous methods by 0.207 and 0.325 respectively. However in table 1, the differences are 0.220 and 0.334. Could you clarify how do you calculate the score differences?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.The method is simple but achieves both better latency and accuracy on NLU tasks.", "weaknesses": "1.The format of this paper is confusing. The gap between figures and text(e.g. Figure 2) seems to be abnormal. Table 4 is also not placed correctly.\n\n2.Poor writing and low-quality figures. For example, I have no idea on the pipeline and implementation of W-PCA until the experiement section. the lines representing dimensions are not straight lines in figure 4. The comparison in the conclusion part(e.g. Kendall score and Spearman score) seems to be inconsistent with experiment results. \n\n3.Although authors claim that W-PCA achieves SOTA, their results are based on their own version of BERT model. They compare their methods with others, whose results are directly copied from their original papers. The comparison is unfair and may mitigate the reliability of their results, especially on the GLUE benchmark.\n\n4.The experiments only focus on BERT models and they only have are around 100M. This limits the applicability of W-PCA to current LLMs, which typically have several billion parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xyQ5ovzeuG", "reviewer": "Reviewer_PoWa", "summary": "The authors present a gradient-free method of evaluation of performance of candidate neural networks as part of neural architecture search. Their approach relies on estimating model's performance on target dataset through the PCA-score of FFN layers weighted by the number of parameters. The benefits of this approach include: strong correlation with target metric, gradient-free approach and the need for only a single forward pass on candidate network. Authors conduct experiment to evalaute the proposed method by sampling and evaluating 500 structures from FlexiBERT benchmark. The structures are evaluated as part of student model in distillation setup. The datasets are GLUE and SQuAD.", "questions": "1) L075-L081 - a lot of empty space here\n2) Figure 3 - legend is very small, not easy to read\n3) Do you always compute PCA scores at epoch 0, i.e right after models parameters are randomly initialized?\n4) L461 - Actual values of reduced CO2 footprint would be more convincing.\n5) Table 1 and subsection 5.2 - I suggest clarifying between what values do you compute correlation.\n6) If answer to 3 is yes: does W-PCA score depends on the random seed that is used during initialization? If so, have you considered computing W-PCA as an average of scores from different initializations?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1) Authors propose a simple yet effective-enough method for evaluating candidate networks during NAS. It helps achieve networks with lowest latency and high performance on target datasets.\n2) The method itself is highly efficient and allows for reasonably fast NAS iterations without compromising on quality as shown in Table 1.", "weaknesses": "1) Parts of paper appear to be LLM-written (for instance: L281-L304). Itself it is not a bad thing, yet those parts of paper lack substance and sometimes repeat the same information.\n2) This paper would benefit from more experiments on decoder models (in addition to existing experiments on encoder models). This would make the paper even more relevant. Indeed, in the introduction, authors cite OpenAI's techincal report as evidence that LLMs have shown substantial performance across domains. It would be naturally to include said LLMs (maybe not OpenAI's LLMs, but generally) into the scope of this paper.\n3) A reasonably-sized limitations section would also benefit the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Udjhrk5ytZ", "reviewer": "Reviewer_8SHz", "summary": "The paper introduces a new zero-shot approach for NAS of lightweight language models, using a score based on PCA dimensionality and the model's parameter count. Experimental results demonstrate that this method outperforms prior NAS approaches and manually designed lightweight architectures.", "questions": "I suggest moving the Zero-Shot NAS section to the experimental settings to provide a clearer explanation of the baselines.\n\nAdditionally, Figure 2 at the beginning needs much more explanation, as it's difficult to understand the main idea without reading the entire paper. The figure illustrating the method could also be improved, as it currently doesn’t contribute to comprehension effectively.\n\n\n\n\n**Overall**, I find the paper to be a good quality and ready to reconsider the score, once the authors address all the raised concerns.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The presentation is clear and well-written. The method outperforms previous baselines, is computationally efficient, and is grounded in solid foundational insights and preliminary observations.", "weaknesses": "**Major Weaknesses:**\n1. The models used are somewhat outdated, as BERT is rarely a focus in current research, which has largely shifted toward GPT-style models.\n2. Reproducibility is challenging without the provision of code, making it difficult for others to replicate the study’s findings.\n3. The procedure is a bit unclear. Are the architectures tuned during genetic search\n4. Are the other baselines fine-tuned? Do they employ Knowledge Distillation? If not, the comparison may be unfair.\n\n\n**Minor Weaknesses:**\n\n1. The term \"PCA Value\" is misleading, as PCA is primarily a dimensionality reduction method. It should be introduced with a brief explanation in the introduction and abstract, before discussing the preliminary experiments.\n2. Table 5 is unclear—does it suggest that using only the parameter count as a proxy achieves better results than all previous methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zxZsAhA89M", "reviewer": "Reviewer_uDq6", "summary": "This paper introduce an approach that leverages a combination of principal component analysis (PCA) values and parameter counts as a zero-shot proxy metric to evaluate model candidates. By bypassing training and relying solely on initial weight information, W-PCA aims to efficiently estimate the potential of each candidate architecture, with significant gains in computational efficiency.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper provides an empirical evidence in Figure 2, showing that initial PCA scores of BERT-based candidate models correlate with their final performance, which is an interesting and effective shortcut for evaluating model quality.\n2. The paper compares W-PCA with other zero-shot and one-shot NAS methods across benchmarks such as GLUE and SQuAD, demonstrating competitive performance. Moreover, the latency is drastically reduced.", "weaknesses": "1. While the paper presents a correlation between PCA values of initial weights and final performance, this correlation may be specific to the BERT architecture and the chosen search space. It remains unclear if this method would generalize well to other model architectures, such as GPT models or non-BERT Transformer variants.\n2. The paper show curves of PCA scores over different training epoch in Figure 3 as a motivation of using PCA in the NAS, but it is better to also show a curve of accuracy besides epoch to better claim \"Tracking performance via PCA\". \n3. The intuition behind combining PCA scores with parameter count to form the W-PCA metric is not well explained. While Figure 2 suggests that W-PCA achieves some gains, the performance patterns are not significantly more distinct than those observed with vanilla PCA.\n4. In the ablation study, W-PCA shows only marginal improvements over baseline methods. Providing statistical significance tests on these improvements would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)