Private Multi-Task Learning: Formulation and Applications to Federated Learning

1 INTRODUCTION . Multi-task learning ( MTL ) aims to solve multiple learning tasks simultaneously while exploiting similarities/differences across tasks ( Caruana , 1997 ) . Multi-task learning is commonly used in applications that warrant strong privacy guarantees . For example , MTL has been used in healthcare , as a way to learn over diverse populations or between multiple institutions ( Baytas et al. , 2016 ; Suresh et al. , 2018 ; Harutyunyan et al. , 2019 ) ; in financial forecasting , to combine knowledge from multiple indicators or across organizations ( Ghosn & Bengio , 1997 ; Cheng et al. , 2020 ) ; and in IoT computing , as an approach for learning in federated networks of heterogeneous devices ( Smith et al. , 2017 ; Hanzely & Richtárik , 2020 ; Hanzely et al. , 2020 ; Ghosh et al. , 2020 ; Sattler et al. , 2020 ; Deng et al. , 2020 ; Mansour et al. , 2020 ) . While MTL can significantly improve accuracy when learning in these applications , there is a dearth of work studying the privacy implications of multi-task learning . In this work , we develop and theoretically analyze methods for MTL with formal privacy guarantees . Motivated by applications in federated learning , we aim to provide task-level privacy1 , where each task corresponds to a client/device/data silo , and the goal is to protect the sensitive information in each task ’ s data ( McMahan et al. , 2018 ) . We focus on incorporating differential privacy ( DP ) ( Dwork et al. , 2006 ) , which ( informally ) requires an algorithm ’ s output to be insensitive to the change of any single entity ’ s data . For MTL , using task-level DP directly would require the entire set of predictive models across all tasks to be insensitive to changes in the private data of any single task . This requirement is too stringent for most applications , as it implies that the predictive model for task k must have little dependence on the training data for task k , thus preventing the usefulness of the model ( see Figure 1 ) . To circumvent this limitation , we leverage a meaningful relaxation of DP known as joint differential privacy ( JDP ) ( Kearns et al. , 2014 ) , which requires that for each task k , the set of output predictive models for all other tasks except k is insensitive to k ’ s private data . As a consequence , the client ’ s private data in task k is protected even if all other clients/tasks collude and share their private data and output models ( as long as client k keeps their data private ) . In contrast to standard DP , JDP allows the predictive model for task k to depend on k ’ s private data , helping to preserve the task ’ s utility . Using JDP , we then develop new learning algorithms for MTL with rigorous privacy and utility guarantees . Specifically , we propose Private Mean-Regularized MTL , a simple framework for learning 1In federated learning applications , where each MTL task typically corresponds to a client ’ s local training task , this can be equivalently viewed as ‘ client-level ’ or ‘ user-level ’ privacy ( McMahan et al. , 2018 ) . multiple tasks while ensuring task-level privacy . We show that our method achieves ( , δ ) -JDP . Our scalable solver builds on FedAvg ( McMahan et al. , 2017 ) , a common method for communicationefficient federated optimization . We analyze the convergence of our solver on both nonconvex and convex objectives , demonstrating a tradeoff between privacy and utility , and evaluate this trade-off empirically on multiple federated learning benchmarks . We summarize our contributions below : • Our work is the first we are aware of to provide formal definitions of task-level differential privacy for multi-task learning objectives ( Section 3 ) . Our definitions rely on joint differential privacy and are applicable to commonly-used multi-task relationship learning objectives . • Using our privacy definitions , we propose Private Mean-Regularized MTL , a simple MTL framework that provides task-level privacy ( Section 4 ) . We prove that our method achieves ( , δ ) -JDP , and we analyze the convergence of our communication-efficient solver on convex and nonconvex objectives . Our convergence analysis extends to non-private settings with partial participation , which may be of independent interest for problems in cross-device federated learning . • Finally , we explore the performance of our approach on common federated learning benchmarks ( Section 5 ) . Our results show that it is possible to retain the accuracy benefits of MTL in these settings relative to global baselines while still providing meaningful privacy guarantees . Further , even in cases where the MTL objective achieves similar accuracy to the global objective , we find that privacy/utility benefits exist when employing the private MTL formulation . 2 BACKGROUND AND RELATED WORK . Multi-task learning . Multi-task learning considers jointly solving multiple related ML tasks . Our work focuses on the general and widely-used formulation of multi-task relationship learning ( Zhang & Yeung , 2010 ) , as discussed in Section 3 . This form of MTL is particularly useful in privacy-sensitive applications where datasets are shared among multiple heterogeneous entities ( Baytas et al. , 2016 ; Smith et al. , 2017 ; Ghosn & Bengio , 1997 ) . In these cases , it is natural to view each data source ( e.g. , financial institution , hospital , mobile phone ) as a separate ‘ task ’ that is learned in unison with the other tasks . This allows data to be shared , but the models to be personalized to each data silo . For example , in the setting of cross-device federated learning , MTL is commonly used to train a personalized model for each device in a distributed network ( Smith et al. , 2017 ; Liu et al. , 2017 ) . Federated learning . A motivation for our work is the application of federated learning ( FL ) , in which the goal is to collaboratively learn from data that has been generated by , and resides on , a number of private data silos , such as remote devices or servers ( McMahan et al. , 2017 ; Kairouz et al. , 2019 ; Li et al. , 2020a ) . To ensure client-level differential privacy in FL , a common technique is to learn one global model across the distributed data and then add noise to the aggregated model to sufficiently mask any specific client ’ s update ( Kairouz et al. , 2019 ; McMahan et al. , 2018 ; Geyer et al. , 2017 ) . However , a defining characteristic of federated learning is that the distributed data are likely to be heterogeneous , i.e. , each client may generate data via a distinct data distribution ( Kairouz et al. , 2019 ; Li et al. , 2020a ) . To model the ( possibly ) varying data distributions on each client , it is natural to instead consider learning a separate model for each client ’ s local dataset . To this end , a number of recent works have explored multi-task learning as a way to improve the accuracy of learning in federated networks ( Smith et al. , 2017 ; Hanzely & Richtárik , 2020 ; Hanzely et al. , 2020 ; Ghosh et al. , 2020 ; Sattler et al. , 2020 ; Deng et al. , 2020 ; Mansour et al. , 2020 ) . Despite the prevalence of multi-task federated learning , we are unaware of any work that has explored task-level privacy for commonly-used multi-task relationship models ( Section 3 ) in federated settings . Differentially private MTL . Prior work in private MTL differs from our own either in terms of the privacy formulation or MTL objective . For example , Wu et al . ( 2020 ) explore a specific MTL setting where a feature representation shared by all tasks is first learned , followed by task-specific models on top of this private representation . We instead study multi-task relationship learning ( Section 3 ) , which is a general and widely-used MTL framework , particularly in federated learning ( Smith et al. , 2017 ) . While our work focuses on task-level privacy , there has been work on data-level privacy for MTL , which aims to protect any single piece of local data rather than protecting the entire local dataset . For example , Xie et al . ( 2017 ) propose a method for data-level privacy by representing the model for each task as a sum of a public , shared weight and a task-specific weight that is only updated locally , and Gupta et al . ( 2016 ) study data-level privacy for a mean estimation MTL problem . Finally , Li et al . ( 2019 ) studies multiple notions of differential privacy for meta-learning . Although similarly motivated by personalization , their framework does not cover the multi-task setting , where there exists a separate model for each task . 3 MULTI-TASK LEARNING AND PRIVACY FORMULATION . In this section , we first formalize our multi-task learning objective , which is a form of meanregularized multi-task learning ( Section 3.1 ) , and then provide our privacy formulation ( Section 3.2 ) . 3.1 PROBLEM SETUP . In the classical setting of multi-task relationship learning ( Zhang & Yang , 2017 ; Zhang & Yeung , 2010 ) , there are m different task learners with their own task-specific data . The aim is to solve : min W , Ω { F ( W , Ω ) = { 1 m m∑ k=1 nk∑ i=1 lk ( xi , wk ) +R ( W , Ω ) } } , ( 1 ) where wk is model for task learner k ; { x1 , . . . , xnk } is the local data for the kth task ; lk ( · ) is the empirical loss for task k ; W = [ w1 ; · · · ; wm ] ; and Ω ∈ Rm×m characterizes the relationship between every pair of task learners . A common choice for setting the regularization termR ( W , Ω ) in previous works ( Zhang & Yeung , 2010 ; Smith et al. , 2017 ) is : R ( W , Ω ) = λ1tr ( WΩWT ) , where Ω can be viewed as a covariance matrix , used to learn/encode positive , negative , or unrelated task relationships ( Zhang & Yeung , 2010 ) . In this paper , we focus on studying the mean-regularized multi-task learning objective ( Evgeniou & Pontil , 2004 ) : a special case of ( 1 ) where Ω = ( Im×m − 1 m1m1 T m ) 2 is fixed . Here Im×m is the identity matrix of size m×m and 1m ∈ Rm is the vector with all entries equal to 1 . By picking λ1 = λ2 , we can rewrite the objective as : min W { F ( W ) = { 1 m m∑ k=1 λ 2 ‖wk − w̄‖2 + nk∑ i=1 lk ( xi , wk ) } } , ( 2 ) where w̄ is the average of task-specific models : w̄ = 1m ∑m i=1 wk . Note that w̄ is shared across all tasks , and each wk is kept locally for task learner k. During optimization , each task learner k solves : min wk { fk ( wk ; w̄ ) = λ 2 ‖wk − w̄‖2 + nk∑ i=1 lk ( xi , wk ) } . ( 3 ) Despite the prevalence of this simple form of multi-task learning and its recent use in applications such as federated learning with strong privacy motivations ( e.g. , Hanzely & Richtárik , 2020 ; Hanzely et al. , 2020 ; Dinh et al. , 2020 ) , we are unaware of prior work that has formalized task-level differential privacy in the context of solving Objective ( 2 ) .

Summary: The paper considers the problem of multi-task learning  (MTL) with task-level differential privacy (TLDP) constraints. The authors propose using joint differential privacy (JDP) for MTL and provide a JDP MTL algorithm, which is a DP variation of FedAvg. They prove the privacy of their algorithm and provide convergence results for smooth nonconvex and strongly convex losses. They then provide numerical experiments.