Private Multi-Task Learning: Formulation and Applications to Federated Learning

1 INTRODUCTION . Multi-task learning ( MTL ) aims to solve multiple learning tasks simultaneously while exploiting similarities/differences across tasks ( Caruana , 1997 ) . Multi-task learning is commonly used in applications that warrant strong privacy guarantees . For example , MTL has been used in healthcare , as a way to learn over diverse populations or between multiple institutions ( Baytas et al. , 2016 ; Suresh et al. , 2018 ; Harutyunyan et al. , 2019 ) ; in financial forecasting , to combine knowledge from multiple indicators or across organizations ( Ghosn & Bengio , 1997 ; Cheng et al. , 2020 ) ; and in IoT computing , as an approach for learning in federated networks of heterogeneous devices ( Smith et al. , 2017 ; Hanzely & Richtárik , 2020 ; Hanzely et al. , 2020 ; Ghosh et al. , 2020 ; Sattler et al. , 2020 ; Deng et al. , 2020 ; Mansour et al. , 2020 ) . While MTL can significantly improve accuracy when learning in these applications , there is a dearth of work studying the privacy implications of multi-task learning . In this work , we develop and theoretically analyze methods for MTL with formal privacy guarantees . Motivated by applications in federated learning , we aim to provide task-level privacy1 , where each task corresponds to a client/device/data silo , and the goal is to protect the sensitive information in each task ’ s data ( McMahan et al. , 2018 ) . We focus on incorporating differential privacy ( DP ) ( Dwork et al. , 2006 ) , which ( informally ) requires an algorithm ’ s output to be insensitive to the change of any single entity ’ s data . For MTL , using task-level DP directly would require the entire set of predictive models across all tasks to be insensitive to changes in the private data of any single task . This requirement is too stringent for most applications , as it implies that the predictive model for task k must have little dependence on the training data for task k , thus preventing the usefulness of the model ( see Figure 1 ) . To circumvent this limitation , we leverage a meaningful relaxation of DP known as joint differential privacy ( JDP ) ( Kearns et al. , 2014 ) , which requires that for each task k , the set of output predictive models for all other tasks except k is insensitive to k ’ s private data . As a consequence , the client ’ s private data in task k is protected even if all other clients/tasks collude and share their private data and output models ( as long as client k keeps their data private ) . In contrast to standard DP , JDP allows the predictive model for task k to depend on k ’ s private data , helping to preserve the task ’ s utility . Using JDP , we then develop new learning algorithms for MTL with rigorous privacy and utility guarantees . Specifically , we propose Private Mean-Regularized MTL , a simple framework for learning 1In federated learning applications , where each MTL task typically corresponds to a client ’ s local training task , this can be equivalently viewed as ‘ client-level ’ or ‘ user-level ’ privacy ( McMahan et al. , 2018 ) . multiple tasks while ensuring task-level privacy . We show that our method achieves ( , δ ) -JDP . Our scalable solver builds on FedAvg ( McMahan et al. , 2017 ) , a common method for communicationefficient federated optimization . We analyze the convergence of our solver on both nonconvex and convex objectives , demonstrating a tradeoff between privacy and utility , and evaluate this trade-off empirically on multiple federated learning benchmarks . We summarize our contributions below : • Our work is the first we are aware of to provide formal definitions of task-level differential privacy for multi-task learning objectives ( Section 3 ) . Our definitions rely on joint differential privacy and are applicable to commonly-used multi-task relationship learning objectives . • Using our privacy definitions , we propose Private Mean-Regularized MTL , a simple MTL framework that provides task-level privacy ( Section 4 ) . We prove that our method achieves ( , δ ) -JDP , and we analyze the convergence of our communication-efficient solver on convex and nonconvex objectives . Our convergence analysis extends to non-private settings with partial participation , which may be of independent interest for problems in cross-device federated learning . • Finally , we explore the performance of our approach on common federated learning benchmarks ( Section 5 ) . Our results show that it is possible to retain the accuracy benefits of MTL in these settings relative to global baselines while still providing meaningful privacy guarantees . Further , even in cases where the MTL objective achieves similar accuracy to the global objective , we find that privacy/utility benefits exist when employing the private MTL formulation . 2 BACKGROUND AND RELATED WORK . Multi-task learning . Multi-task learning considers jointly solving multiple related ML tasks . Our work focuses on the general and widely-used formulation of multi-task relationship learning ( Zhang & Yeung , 2010 ) , as discussed in Section 3 . This form of MTL is particularly useful in privacy-sensitive applications where datasets are shared among multiple heterogeneous entities ( Baytas et al. , 2016 ; Smith et al. , 2017 ; Ghosn & Bengio , 1997 ) . In these cases , it is natural to view each data source ( e.g. , financial institution , hospital , mobile phone ) as a separate ‘ task ’ that is learned in unison with the other tasks . This allows data to be shared , but the models to be personalized to each data silo . For example , in the setting of cross-device federated learning , MTL is commonly used to train a personalized model for each device in a distributed network ( Smith et al. , 2017 ; Liu et al. , 2017 ) . Federated learning . A motivation for our work is the application of federated learning ( FL ) , in which the goal is to collaboratively learn from data that has been generated by , and resides on , a number of private data silos , such as remote devices or servers ( McMahan et al. , 2017 ; Kairouz et al. , 2019 ; Li et al. , 2020a ) . To ensure client-level differential privacy in FL , a common technique is to learn one global model across the distributed data and then add noise to the aggregated model to sufficiently mask any specific client ’ s update ( Kairouz et al. , 2019 ; McMahan et al. , 2018 ; Geyer et al. , 2017 ) . However , a defining characteristic of federated learning is that the distributed data are likely to be heterogeneous , i.e. , each client may generate data via a distinct data distribution ( Kairouz et al. , 2019 ; Li et al. , 2020a ) . To model the ( possibly ) varying data distributions on each client , it is natural to instead consider learning a separate model for each client ’ s local dataset . To this end , a number of recent works have explored multi-task learning as a way to improve the accuracy of learning in federated networks ( Smith et al. , 2017 ; Hanzely & Richtárik , 2020 ; Hanzely et al. , 2020 ; Ghosh et al. , 2020 ; Sattler et al. , 2020 ; Deng et al. , 2020 ; Mansour et al. , 2020 ) . Despite the prevalence of multi-task federated learning , we are unaware of any work that has explored task-level privacy for commonly-used multi-task relationship models ( Section 3 ) in federated settings . Differentially private MTL . Prior work in private MTL differs from our own either in terms of the privacy formulation or MTL objective . For example , Wu et al . ( 2020 ) explore a specific MTL setting where a feature representation shared by all tasks is first learned , followed by task-specific models on top of this private representation . We instead study multi-task relationship learning ( Section 3 ) , which is a general and widely-used MTL framework , particularly in federated learning ( Smith et al. , 2017 ) . While our work focuses on task-level privacy , there has been work on data-level privacy for MTL , which aims to protect any single piece of local data rather than protecting the entire local dataset . For example , Xie et al . ( 2017 ) propose a method for data-level privacy by representing the model for each task as a sum of a public , shared weight and a task-specific weight that is only updated locally , and Gupta et al . ( 2016 ) study data-level privacy for a mean estimation MTL problem . Finally , Li et al . ( 2019 ) studies multiple notions of differential privacy for meta-learning . Although similarly motivated by personalization , their framework does not cover the multi-task setting , where there exists a separate model for each task . 3 MULTI-TASK LEARNING AND PRIVACY FORMULATION . In this section , we first formalize our multi-task learning objective , which is a form of meanregularized multi-task learning ( Section 3.1 ) , and then provide our privacy formulation ( Section 3.2 ) . 3.1 PROBLEM SETUP . In the classical setting of multi-task relationship learning ( Zhang & Yang , 2017 ; Zhang & Yeung , 2010 ) , there are m different task learners with their own task-specific data . The aim is to solve : min W , Ω { F ( W , Ω ) = { 1 m m∑ k=1 nk∑ i=1 lk ( xi , wk ) +R ( W , Ω ) } } , ( 1 ) where wk is model for task learner k ; { x1 , . . . , xnk } is the local data for the kth task ; lk ( · ) is the empirical loss for task k ; W = [ w1 ; · · · ; wm ] ; and Ω ∈ Rm×m characterizes the relationship between every pair of task learners . A common choice for setting the regularization termR ( W , Ω ) in previous works ( Zhang & Yeung , 2010 ; Smith et al. , 2017 ) is : R ( W , Ω ) = λ1tr ( WΩWT ) , where Ω can be viewed as a covariance matrix , used to learn/encode positive , negative , or unrelated task relationships ( Zhang & Yeung , 2010 ) . In this paper , we focus on studying the mean-regularized multi-task learning objective ( Evgeniou & Pontil , 2004 ) : a special case of ( 1 ) where Ω = ( Im×m − 1 m1m1 T m ) 2 is fixed . Here Im×m is the identity matrix of size m×m and 1m ∈ Rm is the vector with all entries equal to 1 . By picking λ1 = λ2 , we can rewrite the objective as : min W { F ( W ) = { 1 m m∑ k=1 λ 2 ‖wk − w̄‖2 + nk∑ i=1 lk ( xi , wk ) } } , ( 2 ) where w̄ is the average of task-specific models : w̄ = 1m ∑m i=1 wk . Note that w̄ is shared across all tasks , and each wk is kept locally for task learner k. During optimization , each task learner k solves : min wk { fk ( wk ; w̄ ) = λ 2 ‖wk − w̄‖2 + nk∑ i=1 lk ( xi , wk ) } . ( 3 ) Despite the prevalence of this simple form of multi-task learning and its recent use in applications such as federated learning with strong privacy motivations ( e.g. , Hanzely & Richtárik , 2020 ; Hanzely et al. , 2020 ; Dinh et al. , 2020 ) , we are unaware of prior work that has formalized task-level differential privacy in the context of solving Objective ( 2 ) .

Summary: In this paper, the authors formalize a notion of task-level differential privacy using joint differential privacy, a concept known in the differentially private mechanism design literature. They formulate the problem of multitask learning under differential privacy under this relaxed definition. The motivation is that there is no need for privacy of the data of a particular task from the task itself, but since you are sharing information in multitask learning, you want to protect against other tasks learning about your private information. They provide an algorithm with convergence guarantees to do multitask learning in this framework and perform some experiments to test this algorithm in the application of federated learning