Ferret: Federated Full-Parameter Tuning at Scale for Large Language
  Models
Authors: name: Yao Shu, name: Wenyang Hu, name: See-Kiong Ng, name: Bryan Kian Hsiang Low, name: Fei Richard Yu

Large Language Models (LLMs) have become indispensable in numerous real-world
applications. Unfortunately, fine-tuning these models at scale, especially in
federated settings where data privacy and communication efficiency are
critical, presents significant challenges. Existing methods often resort to
parameter-efficient fine-tuning (PEFT) to mitigate communication overhead, but
this typically comes at the cost of model accuracy. To address these
limitations, we propose federated full-parameter tuning at scale for LLMs
(Ferret), the first first-order method with shared randomness to enable
scalable full-parameter tuning of LLMs across decentralized data sources while
maintaining competitive model accuracy. Ferret accomplishes this through three
aspects: (1) it employs widely applied first-order methods for efficient local
updates; (2) it projects these updates into a low-dimensional space to
considerably reduce communication overhead; and (3) it reconstructs local
updates from this low-dimensional space with shared randomness to facilitate
effective full-parameter global aggregation, ensuring fast convergence and
competitive final performance. Our rigorous theoretical analyses and insights
along with extensive experiments, show that Ferret significantly enhances the
scalability of existing federated full-parameter tuning approaches by achieving
high computational efficiency, reduced communication overhead, and fast
convergence, all while maintaining competitive model accuracy. Our
implementation is available at https://github.com/allen4747/Ferret.