LPZero: Language Model Zero-cost Proxy Search from Zero
Authors: name: Peijie Dong; name: Lujun Li; name: Xiang Liu; name: Zhenheng Tang; name: Xuebo Liu; name: Qiang Wang; name: Xiaowen Chu
Published: 2024-10-07 07:41:48

Abstract: In spite of the outstanding performance, Neural Architecture Search (NAS) is
criticized for massive computation. Recently, Zero-shot NAS has emerged as a
promising approach by exploiting Zero-cost (ZC) proxies, which markedly reduce
computational demands. Despite this, existing ZC proxies heavily rely on expert
knowledge and incur significant trial-and-error costs. Particularly in NLP
tasks, most existing ZC proxies fail to surpass the performance of the naive
baseline. To address these challenges, we introduce a novel framework,
LPZero, which is the first to automatically design ZC proxies for
various tasks, achieving higher ranking consistency than human-designed
proxies. Specifically, we model the ZC proxy as a symbolic equation and
incorporate a unified proxy search space that encompasses existing ZC proxies,
which are composed of a predefined set of mathematical symbols. To
heuristically search for the best ZC proxy, LPZero incorporates genetic
programming to find the optimal symbolic composition. We propose a
Rule-based Pruning Strategy (RPS), which preemptively eliminates
unpromising proxies, thereby mitigating the risk of proxy degradation.
Extensive experiments on FlexiBERT, GPT-2, and LLaMA-7B demonstrate LPZero's
superior ranking ability and performance on downstream tasks compared to
current approaches.