Distributed Quasi-Newton Method for Fair and Fast Federated Learning

Authors: Shayan Mohajer Hamidi, Linfeng Ye
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Federated Learning, Fairness

Abstract:
Federated Learning (FL) is a promising technology that enables edge devices/clients to collaboratively and iteratively train a machine learning model under the coordination of a central server. The most common approach to FL is first-order methods, where clients send their local gradients to the server in each iteration. However, these methods often suffer from slow convergence rates. As a remedy, second-order methods, such as quasi-Newton, can be employed to accelerate convergence. Unfortunately, similarly to the first-order FL methods, the application of second-order methods in FL can lead to unfair models, achieving high average accuracy while performing poorly on certain clients' local datasets. To tackle this issue, in this paper we introduce  a novel second-order FL framework, dubbed distributed quasi-Newton federated learning (DQN-Fed). This approach seeks to ensure fairness while leveraging the fast convergence properties of quasi-Newton methods in the FL context. Specifically, DQN-Fed helps the server update the global model in such a way that (i) all local loss functions decrease to promote fairness, and (ii) the rate of change in local loss functions aligns with that of the quasi-Newton method. We prove the convergence of DQN-Fed and demonstrate its linear-quadratic convergence rate. Moreover, we validate the efficacy of DQN-Fed across a range of federated datasets, showing that it surpasses state-of-the-art fair FL methods in both fairness and convergence speed. The Code for paper is publicly available at https://github.com/ICMLDQNFed/ICMLDQN.

Decision: desk reject

Reviews and discussion:

Metareview: