Low-Rank Adapters Meet Neural Architecture Search for LLM Compression
Authors: name: J. Pablo Muñoz, name: Jinjie Yuan, name: Nilesh Jain

The rapid expansion of Large Language Models (LLMs) has posed significant
challenges regarding the computational resources required for fine-tuning and
deployment. Recent advancements in low-rank adapters have demonstrated their
efficacy in parameter-efficient fine-tuning (PEFT) of these models. This
retrospective paper comprehensively discusses innovative approaches that
synergize low-rank representations with Neural Architecture Search (NAS)
techniques, particularly weight-sharing super-networks. Robust solutions for
compressing and fine-tuning large pre-trained models are developed by
integrating these methodologies. Our analysis highlights the potential of these
combined strategies to democratize the use of LLMs, making them more accessible
for deployment in resource-constrained environments. The resulting models
exhibit reduced memory footprints and faster inference times, paving the way
for more practical and scalable applications of LLMs. Models and code are
available at
https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning.