Abstract:
Many problems in machine learning rely on multi-task learning (MTL), in which the goal is to solve multiple related machine learning tasks simultaneously. MTL is particularly relevant for privacy-sensitive applications in areas such as healthcare, finance, and IoT computing, where sensitive data from multiple, varied sources are shared for the purpose of learning. In this work, we formalize notions of task-level privacy for MTL via joint differential privacy (JDP), a relaxation of differential privacy for mechanism design and distributed optimization. We then propose an algorithm for mean-regularized MTL, an objective commonly used for applications in personalized federated learning, subject to JDP. We analyze our objective and solver, providing certifiable guarantees on both privacy and utility. Empirically, we find that our method allows for improved privacy/utility trade-offs relative to global baselines across common federated learning benchmarks.

heading: 1 INTRODUCTION
text: Multi-task learning (MTL) aims to solve multiple learning tasks simultaneously while exploiting similarities/differences across tasks (Caruana, 1997). Multi-task learning is commonly used in applications that warrant strong privacy guarantees. For example, MTL has been used in healthcare, as a way to learn over diverse populations or between multiple institutions (Baytas et al., 2016; Suresh et al., 2018; Harutyunyan et al., 2019); in financial forecasting, to combine knowledge from multiple indicators or across organizations (Ghosn & Bengio, 1997; Cheng et al., 2020); and in IoT computing, as an approach for learning in federated networks of heterogeneous devices (Smith et al., 2017; Hanzely & Richtárik, 2020; Hanzely et al., 2020; Ghosh et al., 2020; Sattler et al., 2020; Deng et al., 2020; Mansour et al., 2020). While MTL can significantly improve accuracy when learning in these applications, there is a dearth of work studying the privacy implications of multi-task learning.
In this work, we develop and theoretically analyze methods for MTL with formal privacy guarantees. Motivated by applications in federated learning, we aim to provide task-level privacy1, where each task corresponds to a client/device/data silo, and the goal is to protect the sensitive information in each task’s data (McMahan et al., 2018). We focus on incorporating differential privacy (DP) (Dwork et al., 2006), which (informally) requires an algorithm’s output to be insensitive to the change of any single entity’s data. For MTL, using task-level DP directly would require the entire set of predictive models across all tasks to be insensitive to changes in the private data of any single task. This requirement is too stringent for most applications, as it implies that the predictive model for task k must have little dependence on the training data for task k, thus preventing the usefulness of the model (see Figure 1).
To circumvent this limitation, we leverage a meaningful relaxation of DP known as joint differential privacy (JDP) (Kearns et al., 2014), which requires that for each task k, the set of output predictive models for all other tasks except k is insensitive to k’s private data. As a consequence, the client’s private data in task k is protected even if all other clients/tasks collude and share their private data and output models (as long as client k keeps their data private). In contrast to standard DP, JDP allows the predictive model for task k to depend on k’s private data, helping to preserve the task’s utility.
Using JDP, we then develop new learning algorithms for MTL with rigorous privacy and utility guarantees. Specifically, we propose Private Mean-Regularized MTL, a simple framework for learning
1In federated learning applications, where each MTL task typically corresponds to a client’s local training task, this can be equivalently viewed as ‘client-level’ or ‘user-level’ privacy (McMahan et al., 2018).
multiple tasks while ensuring task-level privacy. We show that our method achieves ( , δ)-JDP. Our scalable solver builds on FedAvg (McMahan et al., 2017), a common method for communicationefficient federated optimization. We analyze the convergence of our solver on both nonconvex and convex objectives, demonstrating a tradeoff between privacy and utility, and evaluate this trade-off empirically on multiple federated learning benchmarks. We summarize our contributions below:
• Our work is the first we are aware of to provide formal definitions of task-level differential privacy for multi-task learning objectives (Section 3). Our definitions rely on joint differential privacy and are applicable to commonly-used multi-task relationship learning objectives.
• Using our privacy definitions, we propose Private Mean-Regularized MTL, a simple MTL framework that provides task-level privacy (Section 4). We prove that our method achieves ( , δ)-JDP, and we analyze the convergence of our communication-efficient solver on convex and nonconvex objectives. Our convergence analysis extends to non-private settings with partial participation, which may be of independent interest for problems in cross-device federated learning.
• Finally, we explore the performance of our approach on common federated learning benchmarks (Section 5). Our results show that it is possible to retain the accuracy benefits of MTL in these settings relative to global baselines while still providing meaningful privacy guarantees. Further, even in cases where the MTL objective achieves similar accuracy to the global objective, we find that privacy/utility benefits exist when employing the private MTL formulation.

heading: 2 BACKGROUND AND RELATED WORK
text: Multi-task learning. Multi-task learning considers jointly solving multiple related ML tasks. Our work focuses on the general and widely-used formulation of multi-task relationship learning (Zhang & Yeung, 2010), as discussed in Section 3. This form of MTL is particularly useful in privacy-sensitive applications where datasets are shared among multiple heterogeneous entities (Baytas et al., 2016; Smith et al., 2017; Ghosn & Bengio, 1997). In these cases, it is natural to view each data source (e.g., financial institution, hospital, mobile phone) as a separate ‘task’ that is learned in unison with the other tasks. This allows data to be shared, but the models to be personalized to each data silo. For example, in the setting of cross-device federated learning, MTL is commonly used to train a personalized model for each device in a distributed network (Smith et al., 2017; Liu et al., 2017).
Federated learning. A motivation for our work is the application of federated learning (FL), in which the goal is to collaboratively learn from data that has been generated by, and resides on, a number of private data silos, such as remote devices or servers (McMahan et al., 2017; Kairouz et al., 2019; Li et al., 2020a). To ensure client-level differential privacy in FL, a common technique is to learn one global model across the distributed data and then add noise to the aggregated model to sufficiently mask any specific client’s update (Kairouz et al., 2019; McMahan et al., 2018; Geyer et al., 2017). However, a defining characteristic of federated learning is that the distributed data are likely to be heterogeneous, i.e., each client may generate data via a distinct data distribution (Kairouz et al., 2019; Li et al., 2020a). To model the (possibly) varying data distributions on each client, it is natural to instead consider learning a separate model for each client’s local dataset.
To this end, a number of recent works have explored multi-task learning as a way to improve the accuracy of learning in federated networks (Smith et al., 2017; Hanzely & Richtárik, 2020; Hanzely
et al., 2020; Ghosh et al., 2020; Sattler et al., 2020; Deng et al., 2020; Mansour et al., 2020). Despite the prevalence of multi-task federated learning, we are unaware of any work that has explored task-level privacy for commonly-used multi-task relationship models (Section 3) in federated settings.
Differentially private MTL. Prior work in private MTL differs from our own either in terms of the privacy formulation or MTL objective. For example, Wu et al. (2020) explore a specific MTL setting where a feature representation shared by all tasks is first learned, followed by task-specific models on top of this private representation. We instead study multi-task relationship learning (Section 3), which is a general and widely-used MTL framework, particularly in federated learning (Smith et al., 2017). While our work focuses on task-level privacy, there has been work on data-level privacy for MTL, which aims to protect any single piece of local data rather than protecting the entire local dataset. For example, Xie et al. (2017) propose a method for data-level privacy by representing the model for each task as a sum of a public, shared weight and a task-specific weight that is only updated locally, and Gupta et al. (2016) study data-level privacy for a mean estimation MTL problem. Finally, Li et al. (2019) studies multiple notions of differential privacy for meta-learning. Although similarly motivated by personalization, their framework does not cover the multi-task setting, where there exists a separate model for each task.

heading: 3 MULTI-TASK LEARNING AND PRIVACY FORMULATION
text: In this section, we first formalize our multi-task learning objective, which is a form of meanregularized multi-task learning (Section 3.1), and then provide our privacy formulation (Section 3.2).

heading: 3.1 PROBLEM SETUP
text: In the classical setting of multi-task relationship learning (Zhang & Yang, 2017; Zhang & Yeung, 2010), there are m different task learners with their own task-specific data. The aim is to solve:
min W,Ω
{ F (W,Ω) = { 1
m m∑ k=1 nk∑ i=1 lk(xi, wk) +R(W,Ω)
}} , (1)
where wk is model for task learner k; {x1, . . . , xnk} is the local data for the kth task; lk(·) is the empirical loss for task k;W = [w1; · · · ;wm]; and Ω ∈ Rm×m characterizes the relationship between every pair of task learners. A common choice for setting the regularization termR(W,Ω) in previous works (Zhang & Yeung, 2010; Smith et al., 2017) is:
R(W,Ω) = λ1tr(WΩWT ) ,
where Ω can be viewed as a covariance matrix, used to learn/encode positive, negative, or unrelated task relationships (Zhang & Yeung, 2010). In this paper, we focus on studying the mean-regularized multi-task learning objective (Evgeniou & Pontil, 2004): a special case of (1) where Ω = (Im×m − 1 m1m1 T m)
2 is fixed. Here Im×m is the identity matrix of size m×m and 1m ∈ Rm is the vector with all entries equal to 1. By picking λ1 = λ2 , we can rewrite the objective as:
min W
{ F (W ) = { 1
m m∑ k=1 λ 2 ‖wk − w̄‖2 + nk∑ i=1 lk(xi, wk)
}} , (2)
where w̄ is the average of task-specific models: w̄ = 1m ∑m i=1 wk. Note that w̄ is shared across all tasks, and each wk is kept locally for task learner k. During optimization, each task learner k solves:
min wk
{ fk(wk; w̄) = λ
2 ‖wk − w̄‖2 + nk∑ i=1 lk(xi, wk) } . (3)
Despite the prevalence of this simple form of multi-task learning and its recent use in applications such as federated learning with strong privacy motivations (e.g., Hanzely & Richtárik, 2020; Hanzely et al., 2020; Dinh et al., 2020), we are unaware of prior work that has formalized task-level differential privacy in the context of solving Objective (2).

heading: 3.2 PRIVACY FORMULATION
text: We start by introducing the definition of differential privacy (DP) before discussing its generalization to joint differential privacy (JDP). In the context of multi-task learning, each of the m task learners owns a private dataset Di ∈ Ui ⊂ U . We define D = {D1, · · · , Dm} and D′ = {D′1, · · · , D′m}. We call two sets D,D′ neighboring sets if they only differ on the index i, i.e., Dj = D′j for all j except i. With this setup in mind, we define differential privacy more formally below. Definition 1 (Differential Privacy (DP) for MTL (Dwork et al., 2006)). A randomized algorithm M : Um → Rm is ( , δ)-differentially private if for every pair of neighboring sets that only differ in arbitrary index i: D,D′ ∈ U and for every set of subsets of outputs S ⊂ R,
Pr(M(D) ∈ S) ≤ e Pr(M(D′) ∈ S) + δ. (4) In the context of MTL, a learning algorithm outputs one model for every task learner. As mentioned previously, since the output of MTL is a collection of models, traditional DP would require that all the models produced by an MTL learning algorithm are insensitive to changes that happen to the private dataset of any single task.
In this work we are interested in studying task-level privacy, where the purpose is to protect one task learner’s data from leakage to any other task learners. In this setting, DP incurs an additional restriction that the model of any task learner should also be insensitive to changes in its own data, which would render each of the models useless. To overcome this limitation of DP, we suggest employing joint differential privacy (JDP) (Kearns et al., 2014), a relaxed notion of DP, to formalize the guarantee that an MTL algorithm should provide in order to protect task-level privacy. Intuitively, JDP requires that for each task k, the set of output predictive models for all other tasks except k is insensitive to k’s private data. We provide a formal definition below. Definition 2 (Joint Differential Privacy (JDP) (Kearns et al., 2014)). A randomized algorithm M : Um → Rm is ( , δ)-joint differentially private if for every i, for every pair of neighboring datasets that only differ in index i: D,D′ ∈ Um and for every set of subsets of outputs S ⊂ Rm,
Pr(M(D)−i ∈ S) ≤ e Pr(M(D′)−i ∈ S) + δ, (5) whereM(D)−i represents the vectorM(D) with the i-th entry removed. JDP allows the predictive model for task k to depend on the private data of k, while still providing a strong guarantee: even if all the clients from all the other tasks collude and share their information, they still will not be able to learn much about the private data in the task k. JDP has mostly been used in applications related to mechanism design (Hsu et al., 2016a; Kannan et al., 2015; Hsu et al., 2016b; Cummings et al., 2015; Rogers & Roth, 2014; Kearns et al., 2014). Although it is a natural choice for achieving task-level privacy in MTL, we are unaware of any work that studies MTL subject to JDP.
We also note that we can naturally connect joint differential privacy to standard differential privacy. Informally, if we take the output of a differentially private process and run some algorithm on top of that locally for each task learner without communicating to the global learner or other task learners, this whole process can be shown to be joint differentially private. This is formalized as the Billboard Lemma (Kearns et al., 2014), presented in Lemma 1 below. Lemma 1 (Billboard Lemma). SupposeM : D →W is ( , δ)-differentially private. Consider any set of functions: fi : Di×W →W ′. The composition {fi(ΠiD,M(D))} is ( , δ)-joint differentially private, where Πi : D → Di is the projection of D onto Di.
With the Billboard Lemma, we are able to obtain joint differential privacy by first training a differentially private model with data from all tasks, and then finetuning on each task with its local data. We formally introduce our algorithm and corresponding JDP guarantee by using Lemma 1 in Section 4.
Finally, note that our privacy formulation itself is not limited to the multi-task relationship learning framework. For any form of multi-task learning where each task-specific model is obtained by training a combination of global component and local component(e.g. Li et al. (2021)), we can provide a JDP guarantee for the MTL training process by using a differentially private global component.

heading: 4 PMTL: PRIVATE MULTI-TASK LEARNING
text: We now present PMTL, a method for performing joint differentially-private MTL (Section 4.1). We provide both a privacy guarantee (Section 4.2) and utility guarantee (Section 4.3) for our approach.
Algorithm 1 PMTL: Private Mean-Regularized MTL 1: Input: m, T , λ, η, {w01, · · · , w0m}, w̃0 = 1m ∑m k=1 w 0 k
2: for t = 0, · · · , T − 1 do 3: Global Learner randomly selects a set of tasks St and broadcasts the mean weight w̃t 4: for k ∈ St in parallel do 5: Each task updates its weight wk for E iterations, ok is the last iteration task k is selected
wt+1k = ClientUpdate(w ok k )
6: Each task sends gt+1k = w t+1 k − wtk back to the global learner. 7: end for 8: Global Learner computes a noisy aggregator of the weights
w̃t+1 = w̃t + 1 |St| ∑ k∈St gt+1k min ( 1,
γ
‖gt+1k ‖2
) +N (0, σ2Id×d)
9: end for 10: return w1, · · · , wm as differentially private personalized models
11: ClientUpdate(w) 12: for j = 0, · · · , E − 1 do 13: Task learner performs SGD locally w = w − η(∇wlk(w) + λ(w − w̃t)) 14: end for

heading: 4.1 ALGORITHM
text: We summarize our solver for private multi-task learning in Algorithm 1. Our method is based off of FedAvg (McMahan et al., 2017), a communication-efficient method widely used in federated learning. FedAvg alternates between two steps: (i) each task learner selected at one communication round solves its own local objective by running stochastic gradient descent for E iterations and sending the updated model to the global learner; (ii) the global learner aggregates the local updates and broadcasts the aggregated mean. By performing local updating in this manner, FedAvg has been shown to empirically reduce the total number of communication rounds needed for convergence in federated settings relative to baselines such as mini-batch FedSGD (McMahan et al., 2017). Our private MTL algorithm differs from FedAvg in that: (i) instead of learning a single global model, all task learners collaboratively learn separate, personalized models for each task; (ii) each task learner solves the local objective with the mean-regularization term; (iii) individual model updates are clipped and random Gaussian noise is added to the aggregated model updates to ensure task-level privacy.
To aggregate updates from each task, we assume that we have access to a trusted global learner, i.e., it is safe for some global entity to observe/collect the individual model updates from each task. This is a standard assumption in federated learning, where access to a trusted central server is assumed in order to collect client updates (Kairouz et al., 2019). However, even with this assumption, note that it is possible for any single task learner to infer information about other tasks from the global model, since it is a linear combination of all task specific models and is shared among all task learners.
There are several ways to overcome this privacy risk and thus achieve ( , δ)-differential privacy. In this paper, we use the Gaussian Mechanism (Dwork & Roth, 2014) during global aggregation as a simple yet effective method, highlighted in the red portion of line 8 in Algorithm 1. In this case, each task learner receives a noisy aggregated global model, making it difficult for any task to leak private information to the others. To apply the Gaussian mechanism, we need to bound the `2-sensitivity of each local model update that is communicated to lie in B = {∆w|‖∆w‖2 ≤ γ}, as highlighted in the blue part of line 8 in Algorithm 1. Hence, at each communication round, the global learner receives the model updates from each task, and clips the model updates to B before aggregation. Note that different from DPSGD (Abadi et al., 2016), when we solve the local objective for each selected task at each communication round, our algorithm doesn’t clip and perturb the gradient used to update the task-specific model. Instead, since the purpose is to protect task or client-level privacy in multi-task learning, we perform standard SGD locally for each task and only clip and perturb the model update that is sent to the global learner. We formalize the privacy guarantee of Algorithm 1 in Section 4.2.

heading: 4.2 PRIVACY ANALYSIS
text: We now rigorously explore the privacy guarantee provided by Algorithm 1. In our optimization scheme, for each task k, at the end of each communication round, a shared global model is received. After that the task specific model is updated by optimizing the local objective. We formalize this local task learning process as hk : Dk ×W →W . Here we simply assumeW ⊂ Rd is closed. Define the mechanism for communication round t to be
Mt({Di}, {hi(·)}, w̃t, σ) = w̃t + 1 |St| ∑ k∈St hk(Dk, w̃ t) + βt, (6)
where βt ∼ N (0, σ2Id×d). Note thatMt characterizes a Sampled Gaussian Mechanism given w̃t as a fixed model rather than the output of a composition ofMj for j < t. To analyze the privacy guarantee of Algorithm 1 over T communication rounds, we define the composition ofM1 toMT recursively asM1:T =MT ({Di}, {hi(·)},MT−1, σ). Theorem 1. Assume |St| = q for all t and the total number of communication rounds is T . There exists constants c1, c2 such that for any < c1 q 2 m2T , the mechanismM 1:T is ( , δ)-differentially
private for any δ > 0 if we choose σ ≥ c2 γ √ T log(1/δ) m . When q = m,M 1:T is ( , δ)-differentially
private if we choose σ = 4γ √ T log(1/δ)
m .
Theorem 1 provides a provable privacy guarantee on the learned global model. When all tasks participate in every communication round, i.e. q = m, the global aggregation step in Algorithm 1 reduce to applying Gaussian Mechanism without sampling rather than Sampled Gaussian Mechanism on the average model updates. We provide a detailed proof of Theorem 1 in Appendix A.1.
Note that Theorem 1 doesn’t rely on how task learners optimize their local objective. Hence, Theorem 1 is not limited to Algorithm 1 and could be generalized to other local objectives and other global aggregation methods that produce a single model aggregate.
Now we show that Algorithm 1, which outputs m separate models, satisfies joint differential privacy. Given w̃t for any t ≤ T , we formally define the process that each task learner k optimize its local objective to be h′k : Dk ×W →W . Note that h′k is not restricted to be hk and could represent the optimization process for any local objective. In order to show that Algorithm 1 satisfies JDP, we would apply the Billboard Lemma introduced in Section 3. In our case, the average model that is broadcast by the global learner at every communication round is the output of a differentially private learning process. Task learners then individually train their task specific models on the respective private data to obtain personalized models. We now present our main theorem of the JDP guarantee provided by Algorithm 1:
Theorem 2. There exists constants c1, c2, for any 0 < < c1 q 2 m2T and δ > 0, let σ ≥ c2γ √ T log(1/δ)
m . Algorithm 1 that outputs h′k(Dk,M1:T ) for each task is ( , δ)-joint differentially private.
From Theorem 2, for any fixed δ, the more tasks involved in the learning process, the smaller σ we need in order to keep the privacy parameter the same. In other words, less noise is required to keep the task-specific data private. When we have infinitely many tasks (m → ∞), we have σ → 0, in which case only a negligible amount of noise is needed to add to the model aggregates to make the global model private to all tasks. We provide a detailed proof in Appendix A.1.
Remark. Note that privacy guarantee provided by our Theorem 2 is not limited to mean-regularized multi-task learning. For any form of multi-task relationship learning with fixed relationship matrix Ω, as long as we fix the `2-sensitivity of model updates and the noise scale of the Gaussian mechanism applied to the statistics broadcast to all task learners, the privacy guarantee induced by this aggregation step is fixed, regardless of the local objective being optimized. For example, as a natural extension of our mean-regularized MTL objective, consider the case where task learners are partitioned into fixed clusters and optimize the mean-regularized MTL objective within each cluster, as in Evgeniou et al. (2005). In this scenario, Theorem 2 directly applies to the algorithm run on each cluster.

heading: 4.3 CONVERGENCE ANALYSIS
text: As discussed in Section 3, we are interested in the following task-specific objective:
fk(wk; w̃) = lk(wk) + λ
2 ‖wk − w̃‖22 (7)
where w̃ is an estimate for the average model w; lk(wk) is the empirical loss for task k; wk ∈ Rd. Here, we analyze the convergence behavior in the setting where a set St of q tasks participate in the optimization process at every communication round. Further, we assume the number of local optimization steps E = 1. We present the following convergence result: Theorem 3 (Convergence under nonconvex loss). Let fk be (L+λ)-smooth. Assume γ is sufficiently large such that γ ≥ maxk,t ‖∇wtkfk(w t k; w̃ t)‖2. Further let f∗k = minw,w̄ fk(w; w̄) and p = q m . If we use a fixed learning rate ηt = η = 1pL+(p− 1p )λ , Algorithm 1 satisfies:
1
mT T−1∑ t=0 m∑ k=1 ‖∇fk(wtk; w̃t)‖2 ≤ O ( 1 mT ) + O ( L+ λ+ λp2 )∑T−1 t=0 Bt+1 T +O ( dσ2 ) . (8)
where
Bt = max k
fk(w t k; w̃ t). (9)
Let σ chosen as we set in Theorem 2. Take T = O (
m λdγ2
) , the right hand side is bounded by
λdγ2
m2 T−1∑ t=0 m∑ k=1 ‖∇fk(wtk; w̃t)‖2 ≤ O ( dγ2 m2 ) +O ( dγ2 ) T−1∑ t=0 Bt+1 +O ( 1 m ) log(1/δ) 2 . (10)
We provide a formal statement and full proof of Theorem 3 in Appendix A.2. The upper bound in (30) consists of two parts: error induced by the gradient descent algorithm and error induced by the Gaussian Mechanism. When σ = 0, Algorithm 1 recovers a non-private mean-regularized multi-task learning solver. Corollary 4. When σ = 0, Algorithm 1 with (L+ λ)-smooth and nonconvex fk satisfies
1
mT T−1∑ t=0 m∑ k=1 ‖∇fk(wtk; w̃t)‖2 ≤ O ( 1 mT ) + O ( L+ λ+ λp2 )∑T−1 t=0 Bt+1 T . (11)
By Theorem 2, given fixed , σ2 grows linearly with respect to T . Hence, given the same privacy guarantee, larger noise is required if the algorithm is run for more communication rounds. Note that in Theorem 3, the upper bound consists of O( 1m 2 ), which means when there are more tasks, the upper bound becomes smaller while the privacy parameter remains the same. On the other hand, Theorem 3 also shows a privacy-utility tradeoff using our Algorithm 1: the upper bound grows inversely proportional to the privacy parameter . We also provide a convergence analysis of Algorithm 1 with strongly-convex losses in Theorem 5 below (formal statement and proof in Appendix A.3). Theorem 5 (Convergence under strongly-convex loss). Let fk be (L + λ)-smooth and (µ + λ)strongly convex. Assume γ is sufficiently large such that γ ≥ maxk,t ‖∇wtkfk(w t k; w̃
t)‖2. Further let w∗k = arg minw fk(w; w̄ ∗), where w̄∗ = 1m ∑m k=1 w ∗ k and p = q m . If we use a fixed learning rate ηt = η = c
L−2 p +λp
for some constant c such that 0 ≤ ηp(c− 2)(µ+ λ) ≤ 1, Algorithm 1 satisfies:
1 m ∆T ≤ (1− ηp(c− 2)(µ+ λ))T
( 1
m ∆0 − C
) + C, (12)
where ∆t = ∑m k=1 fk(w t k; w̃ t) − fk(w∗k; w̃∗), C = O
((√ dσ+ √ 2 λB )2
η
) , B =
maxt maxk fk(w t k; w̃ t). Let σ be chosen as in Theorem 2, then there exists T = O ( m(c2−2c)(µ+λ) λ ( L−2 p2 +λ ) dγ2 ) such that
1 m ∆T ≤ (1− ηp(c− 2)(µ+ λ))T
( 1
m ∆0 −
log(1/δ)
m 2 −O
( B
η
)) + log(1/δ)
m 2 +O
( B
η
) .
(13)
As with Corollary 4, we recover the bound of the non-private mean-regularized MTL solver for σ=0. Corollary 6. When σ = 0, Algorithm 1 with (L+λ)-smooth and (µ+λ)-strongly convex fk satisfies
1 m ∆T ≤ (1− ηp(c− 2)(µ+ λ))T
( 1
m ∆0 −
B
ηp(c− 2)(µ+ λ)
) +
B
ηp(c− 2)(µ+ λ) . (14)

heading: 5 EXPERIMENTS
text: In this section, we empirically evaluate our private MTL solver on several federated learning benchmarks. Specifically, we demonstrate the privacy-utility trade-off of training a MTL objective compared with training a global model. We then compare results of performing local finetuning after learning an MTL objective with local finetuning after learning a global objective.

heading: 5.1 SETUP
text: For all experiments, we evaluate the test accuracy and privacy parameter of our private MTL solver given a fixed clipping bound γ, variance of Gaussian noise σ2, and communication rounds T . All experiments are performed on common federated learning benchmarks as a natural application of multi-task learning. We provide a detailed description of datasets and models in Appendix A.4. Each dataset is naturally partitioned among m different clients. Under such a scenario, each client can be viewed as a task and the data that a client generates is only visible to the local task learner.

heading: 5.2 PRIVACY-UTILITY TRADE-OFF OF PMTL
text: We first explore the training loss (orange) and privacy parameter (blue) as a function of communication rounds across three datasets (Figure 2). Specifically, we evaluate the average loss for all the tasks and given a fixed δ after each round, where δ is set to be 1m for all experiments. In general, for a fixed clipping bound γ and σ, we see that the method converges fairly quickly with respect to the resulting privacy, but that privacy guarantees may be sacrificed in order to achieve very small losses.
To put these results in context, we also compare the test performance of our private MTL solver with that of training a global model. In particular, we use FedAvg (McMahan et al., 2017) to train a global model. At each communication round, task learners solve their local objective individually. Assuming the global learner is trustworthy, while aggregating the model updates from all tasks, the global learner applies a Gaussian Mechanism and sends the noisy aggregation back to the task learners. As a result, private FedAvg differs from our private MTL solver in the following two places: (i) the MTL objective solved locally by each task learner has an additional mean-regularized term; (ii) the MTL method evaluates on one task-specific model for every task while the global method evaluates all tasks on one global model. For each dataset, we select privacy parameter ∈ [0.05, 0.1, 0.2, 0.4, 0.8, 1.6, 2.0, 4.0]. For each , we select the γ, σ, and T that result in the best validation accuracy for a given and record the test accuracy. A detailed description of hyperparameters is listed in Appendix A.5. We plot the test accuracy with respect to the highest validation accuracy given one for both private MTL model and private global model. The results are shown in Figure 3.
In all three datasets, our private MTL solver achieves higher test accuracy compared with training a private global model with FedAvg given the same . Moreover, the proposed mean regularized MTL solver is able to retain an advantage over global model even with noisy aggregation. In particular, for small < 1, adding random Gaussian noise during global aggregation amplifies the test accuracy
difference between our MTL solver and FedAvg. Under the StackOverflow task, both methods obtain test accuracy close to the non private baseline for large . To demonstrate that applying private MTL has an advantage over private global training more generally, we also compared our PMTL method with private FedProx (Li et al., 2020b). The results (which mirror Figure 3) are in Appendix A.6.

heading: 5.3 PMTL WITH LOCAL FINETUNING
text: Finally, in federated learning, previous works have shown local finetuning with different objectives is helpful for improving utility while training a differentially private global model (Yu et al., 2020). In this section, after obtaining a private global model, we explore locally finetuning the task specific models by optimizing different local objective functions. In particular, we use common objectives which (i) naively optimize the local empirical risk (Vanilla Finetuning), or (ii) encourage minimizing the distance between local and global model under different distance metrics (Mean-regularization, Symmetrized KL, EWC (Kirkpatrick et al., 2017; Yu et al., 2020)). The results are listed in Table 1. When =∞ (the non-private setting), global with mean-regularization finetuning outperforms all MTL+finetuning methods. However, when we add privacy to both methods, private MTL+finetuning has an advantage over global with finetuning on different finetuning objectives. In some cases, e.g. using Symmetrized KL as the finetuning objective, the test accuracy gap between private MTL with fintuning and private global with finetuning is amplified when is small compared to the case where no privacy is added during training.

heading: 6 CONCLUSION AND FUTURE WORK
text: In this work, we define notions of task-level privacy for multi-task learning and propose a simple method for differentially private mean-regularized MTL. Theoretically, we provide both privacy and utility guarantees for our approach. Empirically, we show that private mean-regularized MTL retains advantages over training a private global on common federated learning benchmarks. In future work, we are interested in building on our results to explore privacy for more general forms of MTL, e.g., the family of objectives in (1) with arbitrary matrix Ω. We are also interested in studying how task-level privacy relates to algorithmic fairness in the MTL setting.
Ethics Statement. In this work we aim to raise awareness about privacy issues that potentially exist when performing multi-task learning, which is commonly used to model heterogeneous data in areas such as healthcare, finance, and IoT computing. We are particularly motivated by the application of cross-device federated learning, where numerous forms of multi-task learning have been proposed to improve accuracy, but the privacy implications remain unclear. To ensure common notions of task or client-level privacy in this setting while retaining the demonstrated accuracy benefits of MTL, we have provided formal privacy definitions for multi-task learning. Building on these definitions, we proposed a practical algorithm and studied its performance both theoretically and empirically for the special case of mean-regularized multi-task learning objective. We hope this work will be a starting point for future extensions that study the issue of privacy in multi-task learning.
Reproducibility Statement. To enable reproducibility in our theoretical and empirical results, we provide:
• Proofs of Theorem 1, 2, 3, 5 in Appendix A.1, A.2, A.3.
• Dataset and model details in Appendix A.4.
• Hyperparameters for tuning in Appendix A.5.
• Code used for running experiments in Supplementary Material.

Review summaries:
In this paper, the authors formalize a notion of task-level differential privacy using joint differential privacy, a concept known in the differentially private mechanism design literature. They formulate the problem of multitask learning under differential privacy under this relaxed definition. The motivation is that there is no need for privacy of the data of a particular task from the task itself, but since you are sharing information in multitask learning, you want to protect against other tasks learning about your private information. They provide an algorithm with convergence guarantees to do multitask learning in this framework and perform some experiments to test this algorithm in the application of federated learning

Many problems in machine learning rely on multi-task learning (MTL), in which the goal is to solve multiple related machine learning tasks simultaneously. In this work, authors formalize notions of task-level privacy for MTL via joint differential privacy (JDP). They propose an algorithm for mean-regularized MTL, an objective commonly used for applications in personalized federated learning, subject to JDP. Then analyze objective and solver, providing certifiable guarantees on both privacy and utility. The method allows for improved privacy/utility trade-offs relative to global baselines across common federated learning benchmarks. 

The paper considers the problem of multi-task learning  (MTL) with task-level differential privacy (TLDP) constraints. The authors propose using joint differential privacy (JDP) for MTL and provide a JDP MTL algorithm, which is a DP variation of FedAvg. They prove the privacy of their algorithm and provide convergence results for smooth nonconvex and strongly convex losses. They then provide numerical experiments.