Decentralized Learning for Overparameterized Problems: A Multi-Agent Kernel Approximation Approach

( N2/M ) bits ( resp . O ( N √ N/M ) real values ) to achieve minimax optimal generalization performance . Further , we show that the proposed algorithms can significantly reduce the communication complexity compared with state-of-the-art algorithms , for distributedly training models to fit UCI benchmarking datasets . Moreover , each agent needs to share about 200N/M bits to closely match the performance of the centralized algorithms , and these numbers are independent of parameter and feature dimension . 1 INTRODUCTION . Recently , decentralized optimization has become a mainstay of the optimization research . In decentralized optimization , multiple local agents hold small to moderately sized private datasets , and collaborate by iteratively solving their local problems while sharing some information with other agents . Most of the existing decentralized learning algorithms are deeply rooted in classical consensus-based approaches ( Tsitsiklis , 1984 ) , where the agents repetitively share the local parameters with each other to reach an optimal consensual solution . However , the recent trend of using learning models in the overparameterized regime with very high-dimensional parameters ( He et al. , 2016 ; Vaswani et al. , 2017 ; Fedus et al. , 2021 ) poses a significant challenge to such parameter sharing approaches , mainly because sharing model parameters iteratively becomes excessively expensive as the parameter dimension grows . If the size of local data is much smaller than that of the parameters , perhaps a more efficient way is to directly share the local data . However , this approach raises privacy concerns , and it is rarely used in practice . Therefore , a fundamental question of decentralized learning in the overparameterized regime is : ( Q ) For overparameterized learning problems , how to design decentralized algorithms that achieve the best optimization/generalization performance by exchanging minimum amount of information ? We partially answer ( Q ) in the context of distributed kernel learning ( Vert et al. , 2004 ) . We depart from the popular consensus-based algorithms and propose an optimization framework that does not require the local agents to share model parameters or raw data . We focus on kernel learning because : ( i ) kernel methods provide an elegant way to model non-linear learning problems with complex data dependencies as simple linear problems ( Vert et al. , 2004 ; Hofmann et al. , 2008 ) , and ( ii ) kernelbased methods can be used to capture the behavior of a fully-trained deep network with large width ( Jacot et al. , 2018 ; Arora et al. , 2019 ; 2020 ) . Table 1 : Comparison of the total communication required per node by different algorithms for non-overparameterized ( NOP ) and overparameterized ( OP ) regimes . Please see Appendix B for a detailed discussion of the algorithms . Here N is entire sample size , UB on M denotes the upper bound on the number of nodes , M , d is the data dimension , β ≥ 2 is a constant , and T denotes the total communication ( iterations ) rounds utilized by the distributed algorithms . Algorithm Kernel UB on M Communication ( Real Values ) NOP OP DKRR-CM ( Lin et al. , 2020 ) Any O ( N T+1 2 ( T+2 ) ) dTN dTN DKRR-RF-CM ( Liu et al. , 2021 ) RF O ( N T+1 2 ( T+2 ) ) O ( T √ N ) O ( TNβ ) Decentralized-RF ( Richards et al. , 2020 ) RF O ( N 1 3 ) O ( T √ N ) O ( TNβ ) DKLA/COKE ( Xu et al. , 2020 ) RF Any M T √ N O ( TNβ ) RF N √ N M O ( N1+β M ) Algorithm 2 ( this work ) GIP Any M O ( N2 M ) O ( N2 M ) Distributed implementation of kernel learning problems is challenging . Current state-of-the-art algorithms for kernel learning either rely on sharing raw data among agents and/or imposing restrictions on the number of agents ( Zhang et al. , 2015 ; Lin et al. , 2017 ; Koppel et al. , 2018 ; Lin et al. , 2020 ; Hu et al. , 2020 ; Pradhan et al. , 2021 ; Predd et al. , 2006 ) . Some recent approaches rely on specific random feature ( RF ) kernels to alleviate some of the above problems . These algorithms reformulate the ( approximate ) problem in the parameter domain and solve it by iteratively sharing the ( potentially high-dimensional ) parameters ( Bouboulis et al. , 2017 ; Richards et al. , 2020 ; Xu et al. , 2020 ; Liu et al. , 2021 ) 1 . These algorithms suffer from excessive communication overhead , especially in the overparameterized regime where the number of parameters is larger than the data size N . For example , implementing the neural tangent kernel ( NTK ) with RF kernel requires at least O ( Nβ ) , β ≥ 2 , random features ( parameter dimension ) using ReLU activation ( Arora et al. , 2019 ; Han et al. , 2021 ) 2 . For such problems , in this work , we propose a novel algorithmic framework for decentralized kernel learning . Below , we list the major contributions of our work . [ GIP Kernel for Distributed Approximation ] We define a new class of kernels suitable for distributed implementation , Generalized inner-product ( GIP ) kernel , that is fully characterized by the angle between a pair of feature vectors and their respective norms . Many kernels of practical importance including the NTK can be represented as GIP kernel . Further , we propose a multi-agent kernel approximation method for estimating the GIP and the popular RF kernels at individual agents . [ One-shot and Iterative Scheme ] Based on the proposed kernel approximation , we develop two optimization algorithms , where the first one only needs one-shot information exchange , but requires sharing data labels among the agents ; the second one needs iterative information exchange , but does not need to share the data labels . A key feature of these algorithms is that neither the raw data features nor the ( high-dimensional ) parameters are exchanged among agents . [ Performance of the Approximation Framework ] We analyze the optimization and the generalization performance of the proposed approximation algorithms for ` 2 loss . We show that GIP kernel requires communicating O ( N2/M ) bits and the RF kernel requires communicating O ( N √ N/M ) real values per agent to achieve minimax optimal generalization performance . Importantly , the required communication is independent of the function class and the optimization algorithm . We validate the performance of our approximation algorithms on UCI benchmarking datasets . In Table 1 , we compare the communication requirements of the proposed approach to popular distributed kernel learning algorithms . Specifically , DKRR-CM ( Lin et al. , 2020 ) relies on sharing data and is therefore not preferred in practical settings . For the RF kernel , the proposed algorithm outperforms other algorithms in both non-overparameterized and the overparameterized regimes when T > N/M . In the overparameterized regime the GIP kernel is more communication efficient compared to other algorithms . Finally , note that since our analysis is developed using the multiagent-kernel-approximation , it does not impose any upper bounds on the number of machines in the network . Notations : We use R , Rd , and Rn×m to denote the sets of real numbers , d-dimensional Euclidean space , and real matrices of size n×m , respectively . We use N to denote the set of natural numbers . 1For a detailed literature review please see Appendix B . 2To achieve approximation error = O ( 1/ √ N ) . N ( 0 , Σ ) is multivariate normal distribution with zero mean and covariance Σ . Uniform distribution with support [ a , b ] is denoted by U [ a , b ] . 〈a , b〉 ( resp . 〈a , b〉H ) denotes the inner-product in Euclidean space ( resp . Hilbert space H ) . The inner-product defines the usual norms in corresponding spaces . Norm ‖A‖ of matrix A denotes the operator norm induced by ` 2 vector norm . We denote by [ a ] i or [ a ] ( i ) the ith element of a vector a . [ A · a ] ( i ) j denotes the ( i · j ) th element of vector A · a . Moreover , A ( : ,i ) is the ith column of A and [ A ] mk is the element corresponding to mth row and kth column . Notation m ∈ [ M ] denotes m ∈ { 1 , .. , M } . Finally , 1 [ E ] is the indicator function of event E . 2 PROBLEM STATEMENT . Given a probability distribution π ( x , y ) over X × R , we want to minimize the population loss L ( f ) = Ex , y∼π ( x , y ) [ ` ( f ( x ) , y ) ] , ( 1 ) where x ∈ X ⊂ Rd and y ∈ R denote the features and the labels , respectively . Here , f : X → R is an estimate of the true label y . We consider a distributed system of M agents , with each agent m ∈ [ M ] having access to a locally available independently and identically distributed ( i.i.d ) dataset Nm = { x ( i ) m , y ( i ) m } ni=1 with3 ( x ( i ) m , y ( i ) m ) ∼ π ( x , y ) . The total number of samples is N = nM . The goal of kernel learning with kernel function , k ( · , · ) : X × X → R , is to find a function f ∈ H ( whereH is the reproducing kernel Hilbert space ( RKHS ) associated with k ( Vert et al. , 2004 ) ) that minimizes ( 1 ) . We aim to solve the following ( decentralized ) empirical risk minimization problem min f∈H { R̂ ( f ) = L̂ ( f ) + λ 2 ‖f‖2H = 1 M M∑ m=1 L̂m ( f ) + λ 2 ‖f‖2H } , ( 2 ) where λ > 0 is the regularization parameter and L̂m ( f ) = 1n ∑ i∈Nm ` ( f ( x ( i ) m ) , y ( i ) m ) is the local loss at each m ∈ [ M ] . Problem ( 2 ) can be reformulated using the Representer theorem ( Schölkopf et al. , 2002 ) with L̂m ( α ) = 1n ∑ i∈Nm ` ( [ Kα ] ( i ) m , y ( i ) m ) , ∀m ∈ [ M ] , as min α∈RN { R̂ ( α ) = L̂ ( α ) + λ 2 ‖α‖2K = 1 M M∑ m=1 L̂m ( α ) + λ 2 ‖α‖2K } , ( 3 ) where K ∈ RN×N is the kernel matrix with elements k ( x ( i ) m , x ( j ) m̄ ) , ∀m , m̄ ∈ [ M ] , ∀i ∈ Nm and ∀j ∈ Nm̄ . The supervised ( centralized ) learning problem ( 3 ) is a classical problem in statistical learning ( Caponnetto & De Vito , 2007 ) and has been popularized recently due to connections with overparameterized neural network training ( Jacot et al. , 2018 ; Arora et al. , 2019 ) . An alternate way to solve problem ( 2 ) ( and ( 3 ) ) is by parameterizing f in ( 2 ) by θ ∈ RD as fD ( x ; θ ) = 〈θ , φD ( x ) 〉 where φD : X → RD is a finite dimensional feature map . Here , φD ( · ) is designed to approximate k ( · , · ) with kD ( x , x′ ) = 〈φD ( x ) , φD ( x′ ) 〉 ( Rahimi & Recht , 2008 ) . Using this approximation , problem ( 2 ) ( and ( 3 ) ) can be written in the parameter domain with L̂m , D ( θ ) = 1n ∑ i∈Nm ` ( 〈θ , φD ( x ( i ) m ) 〉 , y ( i ) m ) , ∀m ∈ [ M ] , as min θ∈RD { R̂D ( θ ) = L̂D ( θ ) + λ 2 ‖θ‖2 = 1 M M∑ m=1 L̂m , D ( θ ) + λ 2 ‖θ‖2 } . ( 4 ) Note that ( 4 ) is a D-dimensional problem , whereas ( 3 ) is an N -dimensional problem . Since ( 4 ) is in the standard finite-sum form , it can be solved using the standard parameter sharing decentralized optimization algorithms ( e.g. , DGD ( Richards et al. , 2020 ) or ADMM ( Xu et al. , 2020 ) ) , which share D-dimensional vectors iteratively . However , when ( 4 ) is overparameterized with very large D ( e.g. , D = O ( Nβ ) with β ≥ 2 for the NTK ) , such parameter sharing approaches are no longer feasible because of the increased communication complexity . An intuitive solution to avoid sharing these high-dimensional parameters is to directly solve ( 3 ) . However , it is by no means clear if and how one can efficiently solve ( 3 ) in a decentralized manner . The key challenge is that , unlike the conventional decentralized learning problems , here each loss term ` ( [ Kα ] ( i ) m , y ( i ) m ) is not separable 3The techniques presented in this work can be easily extended to unbalanced datasets , i.e. , when each agent has a dataset of different size . over the agents . Instead , each agent m ’ s local problem is dependent on k ( x ( i ) m , x ( j ) m̄ ) with m 6= m̄ . Importantly , without directly transmitting the data itself ( as has been done in Predd et al . ( 2006 ) ; Koppel et al . ( 2018 ) ; Lin et al . ( 2020 ) ) , it is not clear how one can obtain the required ( m·i ) th element of Kα . Therefore , to develop algorithms that avoid sharing high-dimensional parameters by directly ( approximately ) solving ( 3 ) , it is important to identify kernels that are suitable for decentralized implementation and propose efficient algorithms for learning with such kernels .

Summary: The decentralization problem has data distributed among many agents and each agent is wanted to maintain some privacy. In this paper, the authors study the decentralized empirical risk minimization problem with reproducing kernel hilbert space. Two large classes of kernels are considered: (1) generalized inner-product (GIP) kernel based on arccosine kernel (proposed in this work), (2) random feature (RF) kernel. In order to attain decentralization, authors approximate kernels based on the inner product of two finite vectors, and propose algorithms (one-shot and iterative) to optimize private models. In addition, authors study in theory the approximation error for kernels, optimization algorithm performance and generalization error. Finally, experiments are presented to validate the algorithms and theoretical results.