W-PCA Based Gradient-Free Proxy for Efficient Search of Lightweight Language Models

Authors: Shang Wang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: zero-shot NAS, gradient-free, lightweight language models

Abstract:
The demand for efficient natural language processing (NLP) systems has led to the development of lightweight language models. Previous work in this area has primarily focused on manual design or training-based neural architecture search (NAS) methods. Recently, zero-shot NAS methods have been proposed for evaluating language models without the need for training. However, prevailing approaches to zero-shot NAS often face challenges such as biased evaluation metrics and computational inefficiencies.
In this paper, we introduce weight-weighted PCA (W-PCA), a novel zero-shot NAS method specifically tailored for lightweight language models. Our approach utilizes two evaluation proxies: the parameter count and the number of principal components with cumulative contribution exceeding $\eta$ in the feed-forward neural (FFN) layer.  Additionally, by eliminating the need for gradient computations, we optimize the evaluation time, thus enhancing the efficiency of designing and evaluating lightweight language models.
We conduct a comparative analysis on the GLUE and SQuAD datasets to evaluate our approach. The results demonstrate that our method significantly reduces training time compared to one-shot NAS methods and achieves higher scores in the testing phase compared to previous state-of-the-art training-based methods. Furthermore, we perform ranking evaluations on a dataset sampled from the FlexiBERT search space. Our approach exhibits superior ranking correlation and further reduces solving time compared to other zero-shot NAS methods that require gradient computation.

Decision: accept (poster)
Reviewer scores: 5, 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1730186649273

thread_type:
official_review

<thread_object>
creator: reviewer_udq6

<content>
summary:
This paper introduce an approach that leverages a combination of principal component analysis (PCA) values and parameter counts as a zero-shot proxy metric to evaluate model candidates. By bypassing training and relying solely on initial weight information, W-PCA aims to efficiently estimate the potential of each candidate architecture, with significant gains in computational efficiency.

strengths:
1. The paper provides an empirical evidence in Figure 2, showing that initial PCA scores of BERT-based candidate models correlate with their final performance, which is an interesting and effective shortcut for evaluating model quality.
2. The paper compares W-PCA with other zero-shot and one-shot NAS methods across benchmarks such as GLUE and SQuAD, demonstrating competitive performance. Moreover, the latency is drastically reduced.

weaknesses:
1. While the paper presents a correlation between PCA values of initial weights and final performance, this correlation may be specific to the BERT architecture and the chosen search space. It remains unclear if this method would generalize well to other model architectures, such as GPT models or non-BERT Transformer variants.
2. The paper show curves of PCA scores over different training epoch in Figure 3 as a motivation of using PCA in the NAS, but it is better to also show a curve of accuracy besides epoch to better claim "Tracking performance via PCA". 
3. The intuition behind combining PCA scores with parameter count to form the W-PCA metric is not well explained. While Figure 2 suggests that W-PCA achieves some gains, the performance patterns are not significantly more distinct than those observed with vanilla PCA.
4. In the ablation study, W-PCA shows only marginal improvements over baseline methods. Providing statistical significance tests on these improvements would be beneficial.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730708849798

thread_type:
official_review

<thread_object>
creator: reviewer_8shz

<content>
summary:
The paper introduces a new zero-shot approach for NAS of lightweight language models, using a score based on PCA dimensionality and the model's parameter count. Experimental results demonstrate that this method outperforms prior NAS approaches and manually designed lightweight architectures.

strengths:
The presentation is clear and well-written. The method outperforms previous baselines, is computationally efficient, and is grounded in solid foundational insights and preliminary observations.

weaknesses:
**Major Weaknesses:**
1. The models used are somewhat outdated, as BERT is rarely a focus in current research, which has largely shifted toward GPT-style models.
2. Reproducibility is challenging without the provision of code, making it difficult for others to replicate the study’s findings.
3. The procedure is a bit unclear. Are the architectures tuned during genetic search
4. Are the other baselines fine-tuned? Do they employ Knowledge Distillation? If not, the comparison may be unfair.


**Minor Weaknesses:**

1. The term "PCA Value" is misleading, as PCA is primarily a dimensionality reduction method. It should be introduced with a brief explanation in the introduction and abstract, before discussing the preliminary experiments.
2. Table 5 is unclear—does it suggest that using only the parameter count as a proxy achieves better results than all previous methods?

questions:
I suggest moving the Zero-Shot NAS section to the experimental settings to provide a clearer explanation of the baselines.

Additionally, Figure 2 at the beginning needs much more explanation, as it's difficult to understand the main idea without reading the entire paper. The figure illustrating the method could also be improved, as it currently doesn’t contribute to comprehension effectively.




**Overall**, I find the paper to be a good quality and ready to reconsider the score, once the authors address all the raised concerns.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721303116

thread_type:
official_review

<thread_object>
creator: reviewer_powa

<content>
summary:
The authors present a gradient-free method of evaluation of performance of candidate neural networks as part of neural architecture search. Their approach relies on estimating model's performance on target dataset through the PCA-score of FFN layers weighted by the number of parameters. The benefits of this approach include: strong correlation with target metric, gradient-free approach and the need for only a single forward pass on candidate network. Authors conduct experiment to evalaute the proposed method by sampling and evaluating 500 structures from FlexiBERT benchmark. The structures are evaluated as part of student model in distillation setup. The datasets are GLUE and SQuAD.

strengths:
1) Authors propose a simple yet effective-enough method for evaluating candidate networks during NAS. It helps achieve networks with lowest latency and high performance on target datasets.
2) The method itself is highly efficient and allows for reasonably fast NAS iterations without compromising on quality as shown in Table 1.

weaknesses:
1) Parts of paper appear to be LLM-written (for instance: L281-L304). Itself it is not a bad thing, yet those parts of paper lack substance and sometimes repeat the same information.
2) This paper would benefit from more experiments on decoder models (in addition to existing experiments on encoder models). This would make the paper even more relevant. Indeed, in the introduction, authors cite OpenAI's techincal report as evidence that LLMs have shown substantial performance across domains. It would be naturally to include said LLMs (maybe not OpenAI's LLMs, but generally) into the scope of this paper.
3) A reasonably-sized limitations section would also benefit the paper.

questions:
1) L075-L081 - a lot of empty space here
2) Figure 3 - legend is very small, not easy to read
3) Do you always compute PCA scores at epoch 0, i.e right after models parameters are randomly initialized?
4) L461 - Actual values of reduced CO2 footprint would be more convincing.
5) Table 1 and subsection 5.2 - I suggest clarifying between what values do you compute correlation.
6) If answer to 3 is yes: does W-PCA score depends on the random seed that is used during initialization? If so, have you considered computing W-PCA as an average of scores from different initializations?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731103566252

thread_type:
official_review

<thread_object>
creator: reviewer_z4ps

<content>
summary:
This paper introduces W-PCA, a novel zero-shot NAS method specifically tailored for light weight language models.

strengths:
1.The method is simple but achieves both better latency and accuracy on NLU tasks.

weaknesses:
1.The format of this paper is confusing. The gap between figures and text(e.g. Figure 2) seems to be abnormal. Table 4 is also not placed correctly.

2.Poor writing and low-quality figures. For example, I have no idea on the pipeline and implementation of W-PCA until the experiement section. the lines representing dimensions are not straight lines in figure 4. The comparison in the conclusion part(e.g. Kendall score and Spearman score) seems to be inconsistent with experiment results. 

3.Although authors claim that W-PCA achieves SOTA, their results are based on their own version of BERT model. They compare their methods with others, whose results are directly copied from their original papers. The comparison is unfair and may mitigate the reliability of their results, especially on the GLUE benchmark.

4.The experiments only focus on BERT models and they only have are around 100M. This limits the applicability of W-PCA to current LLMs, which typically have several billion parameters.

questions:
1.Could you provide results of some other methods based on your BERT models?

2.I am curious about the applicability of W-PCA on current LLMs(e.g. LLama).

3.How do you calculate GPU days in table 3 and estimate that AutoBERT-Zero-small requires around 1,000 GPU days on GLUE dev set?

4.You claim in the conclusion that W-PCA achieves a Kendall score and a Spearman Score that surpass previous methods by 0.207 and 0.325 respectively. However in table 1, the differences are 0.220 and 0.334. Could you clarify how do you calculate the score differences?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732207031235

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear reviewers,

​		Thank you for your valuable feedback! We have submitted the rebuttal.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors propose a gradient-free method for evaluating the performance of candidate neural networks within neural architecture search. This method estimates a model's performance on a target dataset by using the PCA-score of FFN layers, weighted by the number of parameters. Key advantages of this approach include a strong correlation with target metrics, a gradient-free design, and the ability to evaluate a candidate network with only a single forward pass. The authors validate their method through experiments involving 500 sampled structures from the FlexiBERT benchmark, tested as student models in a distillation setup on the GLUE and SQuAD datasets.

Strengths:

- W-PCA offers a straightforward yet powerful method that achieves competitive accuracy while reducing latency.
- The zero-shot nature of W-PCA drastically reduces computational requirements, making it practical for large-scale NAS tasks.

Weaknesses:
- The writing can be improved and the computational cost for search seems to be quite intensive. 
- While W-PCA shows gains over vanilla PCA, these improvements are relatively modest in the ablation study.