Decentralized Learning for Overparameterized Problems: A Multi-Agent Kernel Approximation Approach

( N2/M ) bits ( resp . O ( N √ N/M ) real values ) to achieve minimax optimal generalization performance . Further , we show that the proposed algorithms can significantly reduce the communication complexity compared with state-of-the-art algorithms , for distributedly training models to fit UCI benchmarking datasets . Moreover , each agent needs to share about 200N/M bits to closely match the performance of the centralized algorithms , and these numbers are independent of parameter and feature dimension . 1 INTRODUCTION . Recently , decentralized optimization has become a mainstay of the optimization research . In decentralized optimization , multiple local agents hold small to moderately sized private datasets , and collaborate by iteratively solving their local problems while sharing some information with other agents . Most of the existing decentralized learning algorithms are deeply rooted in classical consensus-based approaches ( Tsitsiklis , 1984 ) , where the agents repetitively share the local parameters with each other to reach an optimal consensual solution . However , the recent trend of using learning models in the overparameterized regime with very high-dimensional parameters ( He et al. , 2016 ; Vaswani et al. , 2017 ; Fedus et al. , 2021 ) poses a significant challenge to such parameter sharing approaches , mainly because sharing model parameters iteratively becomes excessively expensive as the parameter dimension grows . If the size of local data is much smaller than that of the parameters , perhaps a more efficient way is to directly share the local data . However , this approach raises privacy concerns , and it is rarely used in practice . Therefore , a fundamental question of decentralized learning in the overparameterized regime is : ( Q ) For overparameterized learning problems , how to design decentralized algorithms that achieve the best optimization/generalization performance by exchanging minimum amount of information ? We partially answer ( Q ) in the context of distributed kernel learning ( Vert et al. , 2004 ) . We depart from the popular consensus-based algorithms and propose an optimization framework that does not require the local agents to share model parameters or raw data . We focus on kernel learning because : ( i ) kernel methods provide an elegant way to model non-linear learning problems with complex data dependencies as simple linear problems ( Vert et al. , 2004 ; Hofmann et al. , 2008 ) , and ( ii ) kernelbased methods can be used to capture the behavior of a fully-trained deep network with large width ( Jacot et al. , 2018 ; Arora et al. , 2019 ; 2020 ) . Table 1 : Comparison of the total communication required per node by different algorithms for non-overparameterized ( NOP ) and overparameterized ( OP ) regimes . Please see Appendix B for a detailed discussion of the algorithms . Here N is entire sample size , UB on M denotes the upper bound on the number of nodes , M , d is the data dimension , β ≥ 2 is a constant , and T denotes the total communication ( iterations ) rounds utilized by the distributed algorithms . Algorithm Kernel UB on M Communication ( Real Values ) NOP OP DKRR-CM ( Lin et al. , 2020 ) Any O ( N T+1 2 ( T+2 ) ) dTN dTN DKRR-RF-CM ( Liu et al. , 2021 ) RF O ( N T+1 2 ( T+2 ) ) O ( T √ N ) O ( TNβ ) Decentralized-RF ( Richards et al. , 2020 ) RF O ( N 1 3 ) O ( T √ N ) O ( TNβ ) DKLA/COKE ( Xu et al. , 2020 ) RF Any M T √ N O ( TNβ ) RF N √ N M O ( N1+β M ) Algorithm 2 ( this work ) GIP Any M O ( N2 M ) O ( N2 M ) Distributed implementation of kernel learning problems is challenging . Current state-of-the-art algorithms for kernel learning either rely on sharing raw data among agents and/or imposing restrictions on the number of agents ( Zhang et al. , 2015 ; Lin et al. , 2017 ; Koppel et al. , 2018 ; Lin et al. , 2020 ; Hu et al. , 2020 ; Pradhan et al. , 2021 ; Predd et al. , 2006 ) . Some recent approaches rely on specific random feature ( RF ) kernels to alleviate some of the above problems . These algorithms reformulate the ( approximate ) problem in the parameter domain and solve it by iteratively sharing the ( potentially high-dimensional ) parameters ( Bouboulis et al. , 2017 ; Richards et al. , 2020 ; Xu et al. , 2020 ; Liu et al. , 2021 ) 1 . These algorithms suffer from excessive communication overhead , especially in the overparameterized regime where the number of parameters is larger than the data size N . For example , implementing the neural tangent kernel ( NTK ) with RF kernel requires at least O ( Nβ ) , β ≥ 2 , random features ( parameter dimension ) using ReLU activation ( Arora et al. , 2019 ; Han et al. , 2021 ) 2 . For such problems , in this work , we propose a novel algorithmic framework for decentralized kernel learning . Below , we list the major contributions of our work . [ GIP Kernel for Distributed Approximation ] We define a new class of kernels suitable for distributed implementation , Generalized inner-product ( GIP ) kernel , that is fully characterized by the angle between a pair of feature vectors and their respective norms . Many kernels of practical importance including the NTK can be represented as GIP kernel . Further , we propose a multi-agent kernel approximation method for estimating the GIP and the popular RF kernels at individual agents . [ One-shot and Iterative Scheme ] Based on the proposed kernel approximation , we develop two optimization algorithms , where the first one only needs one-shot information exchange , but requires sharing data labels among the agents ; the second one needs iterative information exchange , but does not need to share the data labels . A key feature of these algorithms is that neither the raw data features nor the ( high-dimensional ) parameters are exchanged among agents . [ Performance of the Approximation Framework ] We analyze the optimization and the generalization performance of the proposed approximation algorithms for ` 2 loss . We show that GIP kernel requires communicating O ( N2/M ) bits and the RF kernel requires communicating O ( N √ N/M ) real values per agent to achieve minimax optimal generalization performance . Importantly , the required communication is independent of the function class and the optimization algorithm . We validate the performance of our approximation algorithms on UCI benchmarking datasets . In Table 1 , we compare the communication requirements of the proposed approach to popular distributed kernel learning algorithms . Specifically , DKRR-CM ( Lin et al. , 2020 ) relies on sharing data and is therefore not preferred in practical settings . For the RF kernel , the proposed algorithm outperforms other algorithms in both non-overparameterized and the overparameterized regimes when T > N/M . In the overparameterized regime the GIP kernel is more communication efficient compared to other algorithms . Finally , note that since our analysis is developed using the multiagent-kernel-approximation , it does not impose any upper bounds on the number of machines in the network . Notations : We use R , Rd , and Rn×m to denote the sets of real numbers , d-dimensional Euclidean space , and real matrices of size n×m , respectively . We use N to denote the set of natural numbers . 1For a detailed literature review please see Appendix B . 2To achieve approximation error = O ( 1/ √ N ) . N ( 0 , Σ ) is multivariate normal distribution with zero mean and covariance Σ . Uniform distribution with support [ a , b ] is denoted by U [ a , b ] . 〈a , b〉 ( resp . 〈a , b〉H ) denotes the inner-product in Euclidean space ( resp . Hilbert space H ) . The inner-product defines the usual norms in corresponding spaces . Norm ‖A‖ of matrix A denotes the operator norm induced by ` 2 vector norm . We denote by [ a ] i or [ a ] ( i ) the ith element of a vector a . [ A · a ] ( i ) j denotes the ( i · j ) th element of vector A · a . Moreover , A ( : ,i ) is the ith column of A and [ A ] mk is the element corresponding to mth row and kth column . Notation m ∈ [ M ] denotes m ∈ { 1 , .. , M } . Finally , 1 [ E ] is the indicator function of event E . 2 PROBLEM STATEMENT . Given a probability distribution π ( x , y ) over X × R , we want to minimize the population loss L ( f ) = Ex , y∼π ( x , y ) [ ` ( f ( x ) , y ) ] , ( 1 ) where x ∈ X ⊂ Rd and y ∈ R denote the features and the labels , respectively . Here , f : X → R is an estimate of the true label y . We consider a distributed system of M agents , with each agent m ∈ [ M ] having access to a locally available independently and identically distributed ( i.i.d ) dataset Nm = { x ( i ) m , y ( i ) m } ni=1 with3 ( x ( i ) m , y ( i ) m ) ∼ π ( x , y ) . The total number of samples is N = nM . The goal of kernel learning with kernel function , k ( · , · ) : X × X → R , is to find a function f ∈ H ( whereH is the reproducing kernel Hilbert space ( RKHS ) associated with k ( Vert et al. , 2004 ) ) that minimizes ( 1 ) . We aim to solve the following ( decentralized ) empirical risk minimization problem min f∈H { R̂ ( f ) = L̂ ( f ) + λ 2 ‖f‖2H = 1 M M∑ m=1 L̂m ( f ) + λ 2 ‖f‖2H } , ( 2 ) where λ > 0 is the regularization parameter and L̂m ( f ) = 1n ∑ i∈Nm ` ( f ( x ( i ) m ) , y ( i ) m ) is the local loss at each m ∈ [ M ] . Problem ( 2 ) can be reformulated using the Representer theorem ( Schölkopf et al. , 2002 ) with L̂m ( α ) = 1n ∑ i∈Nm ` ( [ Kα ] ( i ) m , y ( i ) m ) , ∀m ∈ [ M ] , as min α∈RN { R̂ ( α ) = L̂ ( α ) + λ 2 ‖α‖2K = 1 M M∑ m=1 L̂m ( α ) + λ 2 ‖α‖2K } , ( 3 ) where K ∈ RN×N is the kernel matrix with elements k ( x ( i ) m , x ( j ) m̄ ) , ∀m , m̄ ∈ [ M ] , ∀i ∈ Nm and ∀j ∈ Nm̄ . The supervised ( centralized ) learning problem ( 3 ) is a classical problem in statistical learning ( Caponnetto & De Vito , 2007 ) and has been popularized recently due to connections with overparameterized neural network training ( Jacot et al. , 2018 ; Arora et al. , 2019 ) . An alternate way to solve problem ( 2 ) ( and ( 3 ) ) is by parameterizing f in ( 2 ) by θ ∈ RD as fD ( x ; θ ) = 〈θ , φD ( x ) 〉 where φD : X → RD is a finite dimensional feature map . Here , φD ( · ) is designed to approximate k ( · , · ) with kD ( x , x′ ) = 〈φD ( x ) , φD ( x′ ) 〉 ( Rahimi & Recht , 2008 ) . Using this approximation , problem ( 2 ) ( and ( 3 ) ) can be written in the parameter domain with L̂m , D ( θ ) = 1n ∑ i∈Nm ` ( 〈θ , φD ( x ( i ) m ) 〉 , y ( i ) m ) , ∀m ∈ [ M ] , as min θ∈RD { R̂D ( θ ) = L̂D ( θ ) + λ 2 ‖θ‖2 = 1 M M∑ m=1 L̂m , D ( θ ) + λ 2 ‖θ‖2 } . ( 4 ) Note that ( 4 ) is a D-dimensional problem , whereas ( 3 ) is an N -dimensional problem . Since ( 4 ) is in the standard finite-sum form , it can be solved using the standard parameter sharing decentralized optimization algorithms ( e.g. , DGD ( Richards et al. , 2020 ) or ADMM ( Xu et al. , 2020 ) ) , which share D-dimensional vectors iteratively . However , when ( 4 ) is overparameterized with very large D ( e.g. , D = O ( Nβ ) with β ≥ 2 for the NTK ) , such parameter sharing approaches are no longer feasible because of the increased communication complexity . An intuitive solution to avoid sharing these high-dimensional parameters is to directly solve ( 3 ) . However , it is by no means clear if and how one can efficiently solve ( 3 ) in a decentralized manner . The key challenge is that , unlike the conventional decentralized learning problems , here each loss term ` ( [ Kα ] ( i ) m , y ( i ) m ) is not separable 3The techniques presented in this work can be easily extended to unbalanced datasets , i.e. , when each agent has a dataset of different size . over the agents . Instead , each agent m ’ s local problem is dependent on k ( x ( i ) m , x ( j ) m̄ ) with m 6= m̄ . Importantly , without directly transmitting the data itself ( as has been done in Predd et al . ( 2006 ) ; Koppel et al . ( 2018 ) ; Lin et al . ( 2020 ) ) , it is not clear how one can obtain the required ( m·i ) th element of Kα . Therefore , to develop algorithms that avoid sharing high-dimensional parameters by directly ( approximately ) solving ( 3 ) , it is important to identify kernels that are suitable for decentralized implementation and propose efficient algorithms for learning with such kernels .

Summary: This work considers multi-agent optimization over RKHS together with random feature approximations. The crux is the development of two different techniques for decentralized computation through a novel introduction of a generalized inner-product kernel. Convergence analysis and numerical validation are provided.