Private Multi-Task Learning: Formulation and Applications to Federated Learning
Authors: Shengyuan Hu
Steven Wu
Virginia Smith
Venue: Accepted by TMLR ()

Abstract: Many problems in machine learning rely on multi-task learning (MTL), in which the goal is to solve multiple related machine learning tasks simultaneously. MTL is particularly relevant for privacy-sensitive applications in areas such as healthcare, finance, and IoT computing,
where sensitive data from multiple, varied sources are shared for the purpose of learning. In this work, we formalize notions of client-level privacy for MTL via billboard privacy (BP), a relaxation of differential privacy for mechanism design and distributed optimization. We then propose an algorithm for mean-regularized MTL, an objective commonly used for applications in personalized federated learning, subject to BP. We analyze our objective and solver, providing certifiable guarantees on both privacy and utility. Empirically, we find that our method provides improved privacy/utility trade-offs relative to global baselines across common federated learning benchmarks.

Published in Transactions on Machine Learning Research (04/2023)
Private Multi-Task Learning: Formulation and Applications to Federated Learning
Shengyuan Hu
shengnuanhu@cmuuco edu
Cornegie Mellon University
Zhiwei Steven Wu
zstevenwu@cmmu edu
Cornegie Mellon University
Virginia Smith
smithw@@ema.c edu
Carnegie Mellon University
Reviewed on OpenReview: https: // openreview. net/ bet/forunt forum? id= onuf dyHlug
Abstract
Many problems in machine learning rely multi- learning (MTL), in which the goal is to solve multiple related machine Icarning tasks simultancously. MTL is particularly relevant for privacy-sensitive applications areas such as healthcare, finance, and loT computing, where sensitive data from multiple, varied sources are shared for the purpose of learning. Tn this work, we formalize notions of client-level privacy for MTL via billboard privacy (BP), relaxation of differcntial privacy for mechanism design and distributed optimization. We then propose an algorithm for mean-regularized MTL, objective commonly used for applications in personalized federated learning, subject to BP. We analyze our objective and solver, providing certifiable guarantees on both privacy and utility. Empirically, we find that our method provides improved privacy/utilitss trade-offs relative to global baselines across common federated learning benchmarks.
1 Introduction
Multi-lask learning (MTL) aims to solve multiple learning tasks simultaneously while exploiting similari- ties/dii hissdiffeeences across tasks (Caruana, 1997). MTL is commonly used in applications that warrant strong privacy guarantecs. For example, MTL has been used healthcare, as a way to learn over diverse populations or between multiple institutions (Baytas al., 2016; Suresh et al., 2018; Harulyunyan et al., 2019); in financial forecasting, combine knowledge from multiple indicators or across organizations (Ghosn & Bengio, 1997; Cheng al., 2020); and in loT computing, as an approach for personalized federated learning (Smith et al., 2017; Hanzely & Richtárik, 2020; Hanzely et al., 2020; Ghosh et al., 2020; Sattler et al., 2020; Deng et al., 2020; Mansour et al., 2020). While MTL can significantly improve accuracy when learning in these applications, there is a dearth of work studying the privacy implications of multi-task learning.
In this work, we develop and theoretically analyze methods for MTL. with formal privacy guarantees. Motivated by applications in fedcrated learning, we aim to provide client-level privacy, where each task corresponds to client/ /user/daaa silo, and the goal is to protect sensitive information in each task's data (McMa- han et al., 2018). We focus on ensuring differential prinacy (DP) (Dwork et al., 2006), which (informally) requires an algorithm's output to be insensitive to changes in any single entity's data.
BAMIL lou esprouto
Ghaal
DP.TTL
50
Figure 1: Naively using current client-level DP formulations with MTL results in models that are no better than a random guess.
For MTL, where separate model is generated for each client, using client-level DP directly would require the entire set of predictive models across all tasks to be insensitive to changes in the private data of any single task. This

---

Published in Transactions on Machine Learning Research (04/2023)
requirement is too stringent for most applications, as it implies that the predictive model for task k must have little dependence on the training data for task k, thus preventing the usefulness of the model (see Figures 1).
To address this issue, we leverage a privacy model known as the billboard model (Hsu et al., 2016b). The billboard model is built using: (1) a global signal from a differentially private process that is public to all the clients, and (2) every client i's private data. Unlike DP, the billboard model ensures that for each task k; the set of output predictive models for all other tasks except k is insensitive to k's private data." 1 Therefore, it allows the predictive model for task & to depend on k's private data, helping to preserve each task's utility.
In this work, we develop new learning algorithms for MTL that satisfy the billboard model with rigorous privacy and utility guarantees. Specifically, we propose Private Mean-Regularized MTL, a simple framework for learning multiple tasks while ensuring client-level privacy. We show that our method achieves (e, obbillboordd privacy (BP) (Defined in Section 3.2). Our scalable solver builds on FedAAgg (McMahan et al., 2017), a common mothod for communication-cffiiant foderated optimization. We analyze the convergence of our solver on both nonconvex and convex objectives, demonstrating a tradeoff between privacy and utility, and evaluate this trade-off empirically multiple federated learning benchmarks. We summarize our contributions below:
We propose Private Mean-Regularized MTL, simple MTL framework that provides client-level bill- board privacy (BP) (Section 4). We prove that our method achieves (c,
We analyze the convergence of our communication--fffcieet solver on convex and nonconvex objectives. Our convergence analysis extends to non-private settings with partial participation, which may be of independent interest. for problems in cross-device federated learning.
Finally, we explore the performance of our approach on common federated learning benchmarks (Section 5). Our results show that we can retain the accuracy benefits of MTL in these settings relative to global baselines while stiill providing meaningful privacy guarantees. Further, even in cases where the MTL. and global objectives achieve similar accuracy, we find that privacy/utiin benefits exist when employing our private MTL formulation compared to privately learning a single global model.
2 Background and Related Work
Multi-task learning. Multi Multi-task learning considers jointly solving multiple related ML tasks. Our work focuses on the general and widely-used formulation of multi-task relationship learning (Zhang & Yeung, 2010), as detailed in Section 3. This form of MTL is particularly useful in privacy-sonsitive applications where dalasets are split among multiple helerogeneous entities (Baylas el al., 2016; Smith et al., 2017; Ghosn & Bengio, 1997). In these cases, it is natural to view each data source (e.g., financial institution, hospital, mobile phone) as a separate 'task' that is learned in unison with the other tasks. This allows learning to be performed jointly, but the models to be personalized to each data silo. For example, in the setting of cross-device federated learning, MTT, is commonly used to train a personalized model for each device in a distributed network (Smith al., 2017; Liu et al., 2017).
Federated learning. A motivation for our work is the application of federated learning (FL), in which the goal is to collaboratively learn from number of private data silos, such as remote devices or servers (McMahan et al., 2017; Kairouz et al., 2019; Li et al., 2020a). To ensure client-level DP in FL, a common technique is to learn one global model across the distributed data and then add noise to the aggregated model to sufficiently mask any specific client's 's update (e.g., Kairouz et al., 2019; McMahan et al., 2018; Geyer et al., 2017; Levy et al., 2021; Lowy & Razaviyayn, 2021; Lowy et al., 2022). However, a defining characteristic of fodcrated learning is that the distributed data are likely to be heterogeneous, i.e., each client may generate data via distinct data distribution (Kairouz et al., 2019; Li et al., 2020a). To model the (possibly) varying data distributions on each client, it is natural to instead consider learning a separate model for each client's 'ss local datasct. To this end, a number of recent works have explored multi-task learning as way to improve the accuracy of learning in federated works (Smith et al., 2017: Hanzely & Richtárik, 2020; Hanzely et al., 2020; Ghosh et al., 2020; Sattler et al., 2020; Deng et al., 2020; Mansour et al., 2020). Despite the
This privacy guarantee known as joint differential privacy (JDP) Kearns et (2014), and billboard privacy is common way achieve JDP.
2

---

Published in Transactions on Machine Learning Research (04/2023)
prevalence of multi-task federated learning, unaware of any work that has explored client-level privacy for commonly-used multi-task relationship models (Section 3) in federated settings.
Differentially private MTL. Prior work in private MTL differs from our own either in terms of the privacy formulation or MTL objective. For example, Wu et al. (2020) explore a specific MTL setting where a shared private feature representation first learned, followed by task-specific models. We instead study multi-task relationship learning (Section 3), which a general and widely-used MTL framework, particularly in federated learning (Smith et al., 2017). While our work focuses on client-level privacy, there has been work on data-level privacy for MTL, which aims to protect any single sample of local data rather than protecting the entire local dataset. For example, Xie et al. (2017) propose method for data-level privacy by modeling cach task as a sum of a public shared weight and a task-specific weight that is only updated locally, and Gupta et al. (2016) study data-level privacy for a mean estimation MTL problem. Li et al. (2019) study multiple notions of DP for meta-learning. Although similarly motivated by personalization, their framework does not cover the multi-task setting, where there exists a separate model for each task. Hu et al. (2020) studied example-level private multi-task learning but only their method is restricted to small scale convex lask, which is different from our focus client-level privacy. More closely related to our work, Jain et al. (2021) study a personalization method that learns a private shared representation. Although they similarly leverage the billboard model, their formulation cannot be applied to the general form of multi-task learning in this work. Their results are also limited to the special case of linear regression, unlike the broad set of convex and nonconvex objectives considered herein. Finally, Bietti et al. (2022) similarly propose a personalized federated learning method using the billboard model. Although both methods train a global model shared across tasks, the concrete algorithms are rather different. The main algorithm of Bietti et al. (2022) builds Federated Residual Learning (Agarwal ct al., 2020) while our main algorithm builds on mean-regularized multi-task learning. We provide an in-depth discussion and empirical comparison to this method in Appendix A.8.
3
Multi- Task Learning Setup and Privacy Formulation
In this section, we first formalize the multi-task learning objectives of interest (Section 3.1), and then discuss our proposed privacy formulation (Section 3.2).
3.1 Problem Setup
Multi-task learning aims to improve generalization by jointly solving and exploiting relationships between multiple tasks Caruana. (1997); Ando & Zhang (2005). The classical setting of multi-task relationship
learning (Zhang& & Yang, 2017; Zhang & Yeung, 2010) considers m different lasks with their own task-specific data, learned jointly through the following objective:
ΣΣΣDN 4k,, wx) R(W.0)) }}
1
min W.R.
F(W,
Here we is model for task k fz1,..., Inz } is the local data for the thhh task, lx()) is the empirical loss, W = [wn::.. ; Wm], and Ω E Rmxm characterizes the relationship between every pair of tasks. A common choice for setting the regularization term R(W, () in prior works (Zhang & Young, 2010; Smith et al., 2017) is:
R(W,R)= = Xntr(WSW"")
where Ω can be viewed as a covariance matrix, used to learn/encoen /encode positive, negative, or unrelated task relationships. In this paper, we focus primarily on the regaa-eegulrrizdd multi-task learning objective (Evgeniou & Pontil, 2004): a special case of (1) where Ω (Imxm m Im 1 is fixed. Here Imxm is the identity matrix of size m x m and 1m E Rm is the vector with all entries equal to 1. By picking A = A we can rewrite Objective 1 as:
nk be (1 wk)
min F(W)
m 2 1wk- 1/
(2)
where u is the average of task-specific models, i.c., 1 -= m Σn wk. Note that is shared across all tasks, and each we is kept. locally for task learner k. During optimization, each task learner k solves:
3

---

Published in Transactions on Machine Learning Research (04/2023)
Ske(u;: (6) 11 ma Σ
min 10%
10
4. (1 wk)
Application to Federated Learning. In federated learning, where the goal is to learn over set of clients in a privacy-proseving manner, initial approaches focused on learning a single global model across the data McMahan et al. (2017). However, as data distributions may differ from one client to another, MTL has become a popular alternative that enables every client to collaborate and learn a separate, personalized model of its own Smith et al. (2017). Specifically, in the case mean-regularized MTL, each client solves Equation 3 and utilizes wy as its final personalized model. Unlike finetuning from a global model, MTL itself learns a separate model for each client. by solving Objective 1 in order to improve the generalization performance (Zhang & Yang, 2017; Zhang & Yeung, 2010), which is not equivalent to simple finctuming from global model (see Section 5). Despile the prevalence of mean-regularized multi-lask learning and its recent use in applications such as federated learning with strong privacy motivations (e.g., Smith et al., 2017; Hanzely & Richtárik, 2020; Hanzely et al., 2020; Dinh et al., 2020), we are unaware of prior work that has formalized client-level differential privacy in the context of solving Objective 2.
3.2 Privacy Formulation
To consider privacy for MTL, we start by introducing the definition of differential privacy (DP) and then discuss its generalization to joint differential privacy (JDP). In the context of multi-task learning, each of the m task learners owns a private dataset D; E U, U. We define D = {Di,... Dm3 and = (D',.... D'm and call two sets D, D' neighboring sets if they only differ on the index i, i... D2 D' for all j except
Definition 1 (Differential Privacy (DP) for MTL. (Dwork et al., 2006)). A randomized algorithm M:um :u" + R" is (e, didiferrreel private if for every pair of neighboring sets that only differ in arbitrary inder i: D, D' E and for every set of subsets of outputs S c R,
Pr(M(D) E S) S e (PM(D')ES) E 5
(4)
In the context of MTL, an algorithm outputs one model for every task. In this work we are interested in studying client-level differential privacy, where the purpose to protect one task's data from lcakage to any other task McMahan al. (2017). As mentioned previously and illustrated in Figure 2, since the output of MTL is collection of models, traditional client-level DP would require that all the models produced by MTL algorithm are insensitive to changes that happen in the private dataset of any single client/task
Why can't we apply traditional client-level DP? With the above definition in mind, note that DP has a severe restriction: the model of any task learner must also be insensitive to changes in its own data, effectively rendering each model useless. Although it is intuitive that this would result in unacceptable performance, we verify it empirically in Figure 1. For a common federated learning benchmark (FEMNIST, discussed in Section 5), we apply DPSGD the joint model that concatenates all clients' parameters. We compare MTL with vanilla client-level DP relative to training global model with client-level DP and our proposed MTL approach using BP (below). With the naive DP formulation, MTL is significantly worse than the other approaches improving only marginally upon random guessing.
Billboard Privacy. To overcome this limitation of traditional DP, motivated by the billboard model (Hsu et al., 2016b), we propose billboard privacy (BP), a relaxed notion of DP, to formalize the client-level privacy guarantees for MTL algorithm. We provide the formal definition below.
Definition 2 (Billboard Privacy (BP) (Hsu al., 2016b)). Consider any set of functions: ft : U4 x R R' and g 44 + R, a randomized algorithm M : um + Rm represented as DI, (D))) is (c, bbbblbboardd privale is for every i, for every puir of neighboring dalasels that only differ in inder i: D, D' E um and for every set of subsets of outputs S c CR"
PRM(D) E) < E
(5)
where I1,D is D's projection onto the i-th inder and M(D)-i represents the vector M(D) with the i-th entry removed.
4

---

Published in Transactions on Machine Learning Research (04/2023)
BP allows the predictive model for task & to depend on the private data of k, while still providing strong guarantee. BP provides m - 1-out-of-m privacy under Shamir's scheme of secret sharing (Shamir, 1979): even if all the other m - 1 collude and share their information, they still will not be able to learn much about the private data in the task k. BP has mostly been used in applications related to mechanism design (Hsu et al., 2016a; Kannan el al., 2015; Hsu et al., 2016b). Although is a natural choice for achieving client-level privacy in MTL, we are unaware of any work that studies the general MTL formulations considered herein subject to BP. We also note that we can naturally connect billboard privacy to standard differential privacy. Informally, if g is deffediiferenntially private, then [TI, (D))) TM is (c. o-billboordd private for arbitrary [fihe(tm) by definition of billboard privacy. In other words, if we take the output of a differentially private process and run some algorithm on top of that locally for each task learner without communiattng to the global learner or other lask learners, this whole process can be shown to be BP.
Remark (Generality of Privacy Formulation). Finally, note that our privacy formulation itself is not limited to the multi-task relationship learning framework. For any form of multi-task learning where cach lask-specific model is obtained by training combination of global component and local component(e.g Li et al. (2021)), we can provide a BP guarantee for the MTL training process by using differentially private global component.
Connection to Joint Differential Privacy. Compared to BP, a weaker yet more general privacy formula- tion is known as joint differential privacy (JDP) (Kearns et al., 2014) defined formally below. By definition, (e, 5)BB implies (e, d)JDP. Different from billboard privacy where a global differentially private message g(D) is needed, JDP does not need any global information shared across all the clients. Compared to JDP, achieving BP is harder problem since it requires learning a shared message (in the case of MTL. private global model) that could be used for all clients while a JDP mechanism does not necessarily produce such message.
Definition 3 (Joint Differential Privacy (JDP) (Kearns et al., 2014)). A randomized algorithm M: : 2m + Rm is (c, (-jonnnint differentially private if for every i, for every pair of neighboring datasets that only differ in index i: D,D' E 2 and for every set of subsets of outputs S C RR,
Pr((D)- S e Pr(MID') ES 6
(6)
where M(D)-; represents the vector M(D) with the i-th entry removed.
4 PMTL: Private Multi- Task Learning
We now present PMTT., a method for joint differentially-private MTL. (Section 4.1). We provide both a formal privacy guarantee (Section 4.2) and utility guarantee (Section 4.3) for our approach.
4.1
Algorithm
We summarize our solver for private multi-task learning in Algorithm Our method is based off of FedAvg (McMahan et al., 2017), a communccation-effiient method widely used in federated learning. FedAvg alternates between two steps: (i) cach task learner selected one communication round solves its own local objective by running stochastic gradient descent for E iterations and sending the updated model to the global learner; (ii) the global learner aggregates the local updates and broadcasts the aggregated mean. By performing local updating in this manner, FedAAg has been shown to empirically reduce the total nunber of communication rounds needed for convergence in federated settings relative to baselines such as mini-batch FedSGD (McMahan et al., 2017). Our private MTL algorithm differs from FedAvg that: (i) instead of lcarning a single global model, all task Icarners collaboratively learn scparate, personalized models for each (ask; (ii) each task learner solves the local objective with the mean-regularization term; (iii) individual model updates are clipped and random Gaussian noise is added to the aggregated model updates to ensure client-level privacy.
In this work, we focus on providing global client-level billboard privacy. Therefore, we assume that we have access to a trusted global learner while aggregating updates from each task, i.e., it is safe for some global entity to observe/collect the individual model updates from each task. This is a standard assumption in foderated learning, where access to a trusted server assumed in order to collect client updates (Kairouz et al., 2019). To add an additional layer on security, our method has a natural extension to support secure
5

---

Published in Transactions on Machine Learning Research (04/2023)
Algorithm 1 PMTL: Private Mean-Regularize MTL
Input: m, T, X, 7, fwl. ........ w w.J. i m >k= w for t 0.... .T do
Global Learner randomly selects set of tasks S and broadcasts the mcan weight w for k E St in parallel do Each client updates its weight wy for E iterations, is the last iteration task & is selected
5:
Each client sends 4+1 wtt1 wf back to the global learner.
end for
Global Learner computes noisy aggregator of the weights
W
wi
9k
min 1,
NN o laxd)
S
9k
KES.
end for Output w1,"." ; tim as differentially private personalized models
10:
11: ClientUpdate(w)
12: for j = 0,... F do
13:
Task learner performs SGD locally
== 17 (Vuddl( N(w xt))
14:
end for
aggregation, common cryptographic primitive used federated lcarning (Bonawitz et al., 2016; 2019; Kairouz et al., 2021). While SA/MPC important to ensure secrecy while communicating model updates, neither trusted server assumption nor secure aggregation protects client's private data from leakage to other clients by observing the model output. Thus, our algorithm focuses on addressing privacy concern for model personalization in federated learning.
There are several ways to overcome this privacy risk and thus achieve (e, diiffferrett privacy. In this paper, we use the Gaussian Mechanism (Dwork Roth, 2014) during global aggregation as a simple yet effective method, highlighted in red in line 8 of Algorithm 1. In this case, each client receives a noisy aggregated global model, making it difficult for any task to leak private information to the others. To apply the Gaussian mechanism, need to bound the éz-sensitivity of each local model update that is communicated to lie in B = [Aw]]/Aw/2 & 7) as highlighted in blue in line 8 of Algorithm 1. Hence, at each communication round, the global learner receives the model updates from each clients, and clips the model updates to B before aggregation. Note that different from DPSGD (Abadi et al., 2016), when we solve the local objective for each selected task at each communication round, our algorithm doesn't 't clip and perturb the gradient used update the task-sp model. Instead, since the purpose is to protect task or client-level privacy in multi-task learning, we perform standard SGD locally for each task and only clip and perturb the model update that is sent to the global learner. We formalize the privacy guarantee of Algorithm in Section 4.2.
4.2 Privacy Analysis
We now rigorously explore the privacy guarantee provided by Algorithm 1. In our optimization scheme, for each task k at the end of each communication round, a shared global model is received. After that the task specific model is updated by optimizing the local objective. We formalize this local task learning process he De x W + W. Here we simply assume W c. Re is closed. Define the mechanism for communcattin round & to be
Σ (D( S.S.| KES.
1
where ge (N Taxx).. Note that M' characterizes a Sampled Gaussian Mechanism given ã² a fixed model rather than the output of composition of M for t. To analyze the privacy guarantee

---

Published in Transactions on Machine Learning Research (04/2023)
of Algorithm 1 over T commmncatiin rounds, we define the composition of M' to M recursively M = ((P:), (la()),,M--1,
Theorem 1. Assume |St] q for all t and the total number of communication rounds is T. There erists constants C1, C2 such that for any e < c1 To T, the mechanism M is (c,5) client-level differentially private TVoloo(1/
for any 8 > if we choose a 2 C2
When q = = M is (c,8) client-level differentially private
cm
if we choose 4og/log(11/)
em
Theorem 1 provides provable privacy guarantee the learned model. When all tasks participate in every communication round, i.e. q= m, the global aggregation step in Algorithm 1 reduce to applying Gaussian Mechanism without sampling rather than Sampled Gaussian Mechanism on the average model updates. We provide a detailed proof of Theorem in Appendix A.1. Note in particular that Theorem 1 doesn't rely on how task learners optimize their local objective. Hence, Theorem is not limited to Algorithm and could be generalized to other local objectives and other global aggregation methods that produce a single model aggregate.
Now we show that Algorithm 1, which outputs m separate models, satisfies BP. Given T for any t S T, we formally define the process that each task learner k optimize its local objective to be h4g Da W + W. Note that N/g is not restricted to be h2 and could represent the optimization process for any local objective. In our MR-MTT. problem, the average model broadcast by the global learner at every communication round is the output of a differentially private learning process. Task learners then individually train their modcls on the respective private data to obtain personalized models. By definition of billboard privacy in Section 3.2, we are able to show that Algorithm 1 satisfies BP:
There exists constants c1. c22 for any 0 < <2 I and ó > O, let o > T Algorithm 1 that outputs 12g (DD,,MMTTT for each task is bebblbbill private.
Corollary 2.
From Theorem 2, for any fixed 6, the more tasks involved in the learning process, the smaller a we need in order to keep the privacy parameter e the same. In other words, less noise is required to keep the task-specific data private. When we have infinitely many tasks (m + oo), we have o + 0, in which case only a negligible amount of noise is nceded for the model aggrogates to make the global model private all tasks. We provide detailed proof in Appendix A.
Remark (Generality of Corollary 2). Note that the privacy guarantee provided by Corollary 2 is not limited to regulregglarized MTL. For any form of multi-task relationship learning with fixed relationship matrix Ωt as long as we fix the fz-sensitivity of model updates and the noise scale of the Gaussian mechanism applied to the statistics broadcast to all task learners, the privacy guarantee induced by this aggregation step is fixed, regardless of the local objective being optimized. For example, as natural extension of mean-regularized MTL, consider the case where task learners are partitioned into fixed clusters and optimize the mean-regularized MTL objective within each cluster, in Evgeniou et al. (2005). Tn this scenario, Theorem 2 directly applies to the algorithm run on each cluster.
4.3 Convergence Analysis
As discussed in Section 3, we are interested in the following task-specific objective: fu(wa; () = (w(w) + > -タ | - -l1
(8)
where w is an estimate for the average model w; (www) = Emh la (ara, wx) is the empirical loss for task k; Wh& E Rd.
Here, we analyze the convergence behavior in the setting where a set S, of q tasks participate in the optimization process at every communication round. Further, we assume the total number of communication round T is divisible by the number of local optimization steps E: T 0 mod E. We present the following convergence result:
Theorem 3 (Convergence under nonconvex loss (Informal)). Let fx be (L + X)-smooth. Assume fx is G-Tápschitz in l norm such that 7 G. Further let St min w fe(w; (w) p = g. and B
7

---

Published in Transactions on Machine Learning Research (04/2023)
O= 0.01, y=0.5
0=0.01, y=0.2
a=0.02, y=1
155
15
6.0
Privacy parameteriaa Loskes
12.5
Privacy parameterie
3-
10.0
3:
Losses
10
10
5.5
7.5
5.0
5.0
2.5
1-
0.0
200
400
800
0
200
400
600
800
1000
Communicatiin rounds
Communication rounds
Communication rounds
(a) StackOverflow tag prediction
(b) FEMNIST
(c) CelebA
Figure 2: Loss and privacy parameter communication rounds for PMIL. The blue line shows the change of in terms of number of communication rounds during training. The orange line shows the average training loss.
max; max; fü (w w If we use fized learning rate Th 11 pL+(D+
Algorithm 1 satisfies:
AB
dDoa²
ΣΣ
fu w\)l 5
+
+O
+
O
(9)
mf k=
T
E
E'
Let a chosen as we sel in Theorem
Take T 0
m Xdp?
the right hand side is bounded by
m
d
AB
log(1/8)
s
O
+
(10)
mT
m
E
mE
We provide formal statement and proof of Theorem 3 in Appendix A.2. The upper bound in Equation 9 consists of two parts: error induced by the gradient descent algorithm and error induced by the Gaussian Mechanism. When o = 0, Algorithm recovers a non-private MR-MTL solver.
Corollary 4. When 0, Algorithm 1 with (L + X) -smooth and nonconeer fx satisfies
V (w T || S O +
+
(11)
mT
F
k=1
By Theorem 2, given fixod e, o² grows linearly with respect to T. Hence, given the same privacy guarantee, for more communication rounds. Note that in Theorem 3, the
larger noise is required if the algorithm upper bound consists of O ((,)),, ), which means when there are more tasks, the upper bound becomes smaller while the privacy parameter remains the same. On the other hand, Theorem 3 also shows privacy-utility tradeoff using our Algorithm 1: the upper bound grows inversely proportional to the privacy parameter e. We also provide a convergence analysis of Algorithm with strongly-convex losses in Theorem below (formal statement and proof in Appendix A.3).
Theorem 5 (Convergence under strongly-convex loss (Informal)). Let SK be (L A)-smooth and (n + A)- strongly conver. Assume 2 max | = fa (wfs 122 Let wi arg min,. fa (w; w²) and 2, If we set m7 + 2X for some constant e such that I mplcc 2)(μ + A) < 1, we have:
m
op
1
2
=ddo
fo w¹) fa(wfi w²) <
+ 0
O
(12)
m
2
2
2B
Let be chosen in Theorem then there exists T O
such that
Adps
m
(wff:w)) ff((w; tx) 00 2Tm
25
)
+
mm2 -1)==1))))
2
)
log(1/8)
(13)
m
2 -
e²
8

---

Published in Transactions on Machine Learning Research (04/2023)
0.30
0.9
077
Test accuracy
26
Test accuracy
0.20
0.
0.15
04
0.7
2.1
02
0.05
non-private Global private
0.1
6.00
2.0
0.0
1.0
1.5
2.0
25
3.5
0.0
05
1.0
1.5
2.0
2.5
3.0
3.5
4.0
0.6
0.5
1.0
155
2.0
25
3.0
33
40
(a) SverkOverflow tag prediction
(b) FEMNIST
(c) CelebA
Figure 3: Comparison of training PMTL private global model. PMTL is able to retain advantages global approaches in private settings. In addition, even in settings where the non-private MTL and global baselines are similar (e.g., FEMNIST, CelebA), there exist utility benefits at all levels of when using PMTL.
As with Corollary 4, recover the bound of the non-private mean-regularized MTL solver for o=0. We provide convergence under convex loss in non-private scenario in Appendix A.3.
Convergence to a neighborhood of the optimal. It is worth noting that for both convex and non- convex case, given the nature of the mean-regularized MTL objective, the local objective will only converge to within a neighborhood example. Assume that we have m different clients/tasks, each with local data 2; and local model wy The mean-regularized MTL objective for this problem would be 1 (x; x= m -, (w, which is greater than 1 (x w; Wl w) I Ex w)?. Note that this lower bound neither converges to 0 w changes time diminishes with increasing in line with other previous works that study the same objective but with different solvers, where similar dependencies on and B can be seen (Hanzely & Richtárik, 2020).
the optimal. Consider the following simple mean-estimation problem as an
This result is therefore espected/standaat and is
5 Experiments
We empirically evaluate private MTL solver common federated learning benchmarks Caldas al. (2018). We first demonstrate the superior privacy-utility trade-off that exists when training our private MTL method compared with training a single global model (Section 5.2). We also compare our method with simple cectuuinnngexpplorrieee the results of performing local finetuning after learning an MTL objective vs. a global objective (Section 5.3). Our code is publicly available at: sttps://githun com/ cow/s-huu/pe hhu//sT
5.1 Setup
For all experiments, we evaluate the test accuracy and privacy parameter of our private MTL solver given a fixed clipping bound 7, variance of Gaussian noise o², and communication rounds T. All experiments are performed on common federated learning benchmarks a natural application of multi-task learning. We provide detailed description of datasets and models in Appendix A.4. Each dataset is naturally partitioned among different clients. Under such scenario, each client can be viewed a task and the data that client generates is only visible locally.
FEMNIST
0.1
0.8
2.0
Global
Vanilla Finetuming
0.645 = 0.013
0.606 :: 0.017
0.600 i: 0.016
0.618 = 0.017
0.677 1 0.008
0.653 3: 0.010
0.832
0.005
0.812 0.009
Mlenn-rogllarzzation
0. 6.608 :: 0.011
0.581
à 0.011
0.605
0.008
0.574 = 0.006
0.656
1 0.009
0.633 0.003
0.633
0.826 0.011
0.839 0.006
Symmetriaed KL
0 188 : 0.012
0.318
0.005
0.584
0.012
0.181
0.016
0.662
1 0.016
0.565
1: 0.019
0.839
0.006
0.829 0.015
EWC
0. 0.663 :: 0.002
0.558 à: 0.001
0.595 0.004
0.607 = 0.007
0.681 1: 0.002
0.666 1: 0.001
0.837 0.001
0.823 0.005
Table 1: Comparison of PMTL vs. a private global model with different local finetuning methods, e = oo corresponds no noise and clipping, i.c., training non-privately. The higher accuracy between MTL and Global given the same e and finetuning method is bolded.
5.2 Privacy-Utility Trade-off of PMTL
We first explore the training loss (orange) and privacy parameter e (blue) as function of communication rounds across three datasets (Figure 2). Specifically, we evaluate the average loss for all the tasks and given

---

Published in Transactions on Machine Learning Research (04/2023)
fixed ó after each round, where 8 is set to be for all experiments. In general, for fixed clipping bound y and a, we see that the method converges fairly quickly with respect to the resulting privacy, but that privacy guarantees may be sacrificed in order to achieve very small losses.
To put these results in context, we also compare the test performance of our PMTL solver with that of training a global model. In particular, we use FedAvg (McMahan et al., 2017) to train a global model. At each communication round, clients solve their local objective individually. While aggregating the model updates, the global learner applies Gaussian Mechanism and sends the noisy aggregation back to the clients. As a result, private FedAvg differs from our PMTL solver in the following two places: (i) the MTL objective solved locally by cach task learner has a mean-rogularized term; (ii) the MTL method evaluates on one task-specific model for every task while the global method evaluates all tasks on one global model. For each dataset, we select privacy parameter e E [0.05, 0.1, 0.2, 0.4, 0.8, 1.6, 2.0, 4.0]. For each e, we select the y, σ, and T that result. in the best validation accuracy for a given c and record the test accuracy. A detailed description of hyperparameters is listed in Appendix A.5. We plot the test accuracy with respect to the highest validation accuracy given one e for both privale MTL. model and private global model. Results are shown in Figure 3.
In all three datasets, our private MTL solver achieves higher test accuracy compared with training private global model with FedAvg given the same c. Moreover, the proposed mean regularized MTL solver is able to retain an advantage over global model even with noisy aggregation. In particular, for small 6 < 1, adding random Gaussian noise during global aggregation amplifies the test accuracy difference between our MTL solver and FedAvg. Under the StackOverflow task, both methods obtain test accuracy close to the non private baseline for large e. To demonstrate that applying private MTL has an advantage over private global training more generally, also compared our PMTL method with private FedProx (Li et al., 2020b). The results (which mirror Figure 3) are in Appendix A.6.
5.3 PMTL with local finetuning
Finally, in federated learning, previous works have shown local finetuning with different objectives is helpful for improving utility while training differentially private global model (Yu et al., 2020). In this section, after obtaining private global model, explore locally finetuning the task specific models by optimizing different local objective functions. In particular, we use common objectives which (i) naively optimize the local empirical risk (Vanilla Finetuming), or (ii) encourage minimizing the distance between local and global model under different distance metrics MMean-regullrrzztoon,, Symmetrized KL, EWC (Kirkpatrick et al., 2017; Yu et al., 2020)). The results are listed in Table When = oo (the non-private setting), global with rean-regullarizttinn finetuning outperforms all MiLtfinetunig methods. However, when we add privacy both methods, private fint+fnnttuing has an advantage over global with finetuning on different finctuning objectives. In some and private cases, e.g. using Symmetrized KL, the test accuracy gap between private MTL++inluuing is amplified when is small compared to the case where no privacy is added during training. We also compare our MMTTL+finnttnig with training pure local model in Appendix A.7.
6 Conclusion and Future work
In this work, we defined notions of client-level differential privacy for multi-task learning and proposed a simple method for private mean-regularized MTL. Theoretically, we provided both privacy and utility guarantees for our approach. Empirically, we showed that private MTL retains advantages over training a private global model on common federated learning benchmarks. In future work, we are interested in building on our results to explore privacy for more general forms of MTL, e.g., the family of objectives in (1) with arbitrary Ω as well as studying how client-level privacy relates to issues of fairness in multi-task settings.
7
Acknowledgements
ZSW was supported in part by the NSF FAI Award # #1999606, NSF Award #2120667, a Google Faculty Research Award, J.P. Morgan Faculty Award, Meta Research Award, and Cisco research grant.
10

---

Published in Transactions on Machine Learning Research (04/2023)
References
Tensorflow federated: federated.
Machine learning on decentralized data. URL https::/www.tennoorfft
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Nla Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In ACM SIGSAC Conference Computer and Communicaatios Security, 2016.
Alekh Agarwal, John Langford, and Chen- Yu Wei. arXiv:2003.1988 2020.
Federated residual learning.
orXie preprinl
Rie Kubota Ando and Tong Zhang. framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6:1817-1853, 2005.
Inci M Baytas, Ming Yan, Anil Jain, and Jiayu Zhou. Asynchronous multi-task learning. In International Conference on Data Mining (TCDM), 2016.
Alberto Bietti, Yuen-Yu Wei, Miroslav Dudik, John Langford, and Steven Wu. Personalization improves privacy-accuracy tradcoffs foderated learning. In Proceedings of the 39th International Conference on Machine Leurning, volume 162 of Proceedings of Machine Learning Research, pp. 1945 1962. PMLR, 17 Jul 2022.
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for federated learning on user-held data. Xin preprint wxXi::661..444 2016.
Keith Bonawitz, Fariborz Salehi, Jakub Konečný, Brendan McMahan, and Marco Gruteser. Federated learning with autotuned communication-efieee secure aggregation. In 2019 53rd Asilomar Conference on Signals, Systems, and Computers, 2019.
Sebastian Caldas, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings, https://leat: cmu, edu/. arXiv preprint arXiu: 1812.081000007 2018.
Rich Caruana. Multitask learning. Machine Learning, 28:41-75, 1997.
Yong Cheng, Yang Liu, Tianjian Chen, and Qiang Yang. Federated learning for privacy-preserving ai. Communications of the ACM, 63(12), 2020.
Gregory Cohen, Saced Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist
handwritten letters. In 2017 International Joint Conference Neural Networks (IJCNN), pp. 2921-2926, 2017.
Yuyang Deng, Mohammad Mahdi Kamani, and Mchrdad Mahdavi. Adaptive personalized foderated learning. arXiv preprint Xix:2003.13466 2020.
Canh T Dinh, Nguyen Tran, and Tuan Dung Nguyen. Personalized federated learning with moreau envelopes. In Advances in Neural Information Processing Systems, 2020.
Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 933-4):211-477 2014.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to scnsitivity in private data analysis. In Conference on Theory of Cryptography (TCC), 2006.
Theodoros Evgeniou and Massimiliano Pontil. Regularized multi-task learning. In Conference Knowledge Discovery and Data Mining, 2004.
Theodoros Evgeniou, Charles A Micchelli, Massimiliano Pontil, and John Shawe-Taylor. Learning multiple tasks with kernel methods. Journal of Machine Learning Research, 6(4), 2005.
11

---

Published in Transactions on Machine Learning Research (04/2023)
Robin C Geyer, Tassilo Klein, and Moin Nabi. Differentially private federated learning: A client level perspective. arxiv preprint or arxi 712. 07557, 2017.
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. In Advances Neural Information Processing Systems, 2020.
Joumana Ghosn and Yoshua Bengio. Multi-task learning for stock selection. In Advances in Neural Information Processing Systems, 1997.
Sunil Kumar Gupta, Santu Rana, and Svetha Venkatesh. Differentially private multi-task learning. In Pacific-Asia Workshop on Intelligence and Security Informatics, pp. 101-113. Springer, 2016.
Filip Hanzely and Peter Richtárík. Federated learning of mixture of global and local models. Xiv preprinl arXiv:2002.05516, 2020.
Filip Hanzely, Slavomír Hanzely, Samuel Horváth, and Peter Richtarík. Lower bounds and optimal algorithms for personalized federated learning. Advances in Neural Information Processing Systems, 2020.
Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific data, 6(1):1-18, 2019.
Justin Hsu, Zhiyi Huang, Aaron Roth, Tim Roughgarden, and Zhiwei Steven Wu. Private matchings and allocations. STAM Journal of Computing, 45(6), 2016a.
Justin Hsu, Zhiyi Huang, Aaron Roth, and Zhiwei Steven Wu. Jointly private convex programming. In Symposium Discrete Algorithms, SODA, 2016b.
Rui Hu, Yuanxiong Guo, Hongning Li, Qingqi Pei, and Yanmin Gong. Personalized federated learning with differential privacy. IEEE Internet of Things Journal, 7(10):9530 9539, 2020.
Prateek Jain, John Rush, Adam Smith, Shuang Song, and Abhradecp Guha Thakurta. Differentially private model personalization. In Neural Information Processing Systems, 2021.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977 2019.
Peter Kairouz, Ziyu Liu, and Thomas Steinke. The distributed discrete gaussian mechanism for foderated learning with secure aggregation. In International Conference on Machine Learning, pp- 3201-5212 PMLR, 2021.
Sampath Kannan, Jamie Morgenstern, Aaron Roth, and Zhiwei Steven Wu. Approximately stable, school optimal, and student-truthful many-to-one matchings (via differential privacy). In Symposium on Discrete Algorithms,, SODA, 2015.
Michael J. Kearns, Mallesh M. Pai, Aaron Roth, and Jonathan R. Ullman. Mechanism design in large games: incentives and privacy. In Moni Naor (ed.), Innovations in Theoretical Computer Science, ITCS114, Princeton, NJ, USA, January 12-14, 2014, pp. 403-410. ACM, 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Acodemy of Sciences, 114(13):338 (1113)))321133333 1326,, 2017.
Yassine Laguel, Krishna Pillutla, Jérôme Malick, and Zaid Harchaoui. A superquantile approach to federated learning with heterogeneous devices. In 55th Annual Conference on Information Sciences and Systems, CISS 2021, Baltimore, MD, USA, March 24-26, 2021, pp. 1-6. IEEE, 2021.
Daniel Levy, Ziteng Sun, Karcem Amin, Satyen Kale, Alex Kulesza, Mehryar Mohri, and Ananda Theertha Suresh. Learning with user-level privacy. Advances in Neural Information Processing Systems, 34, 2021.
12

---

Published in Transactions on Machine Learning Research (04/2023)
Jeffrey Li, Mikhail Khodak, Sebastian Caldas, and Ameet Talwalkar. Differentially private meta-learning. In International Conference on Learning Representations, 2019.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37 (7):5:0 100000000--0 2020a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In Conference on Machine Learning and Systems, 2020b.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pp. 6357-636660 6368. PMI.R 2021.
Sulin Liu, Sinno Jialin Pan, and Qirong Ho. Distributed multi-task relationship learning. In International Conference on Knowledge Discovery and Data Mining (ICDM), pp. 937-946 2017.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In International Conference on Compuler Vision, 2015.
Andrew Lowy and Meisam Razaviyayn. Private federated learning without trusted server: Optimal algorithms for convex losses. arXiv preprinl arXiv:2106.0977 2021.
Andrew Lowy, Ali Ghafelebashi, and Mcisam Razaviyayn. Private non-convex federated learning without trusted server. arXiv preprinl ar arXiv:20330077 2022.
Yishay Mansour, Mchryar Mohri, Jac Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications to federated learning. ar preprinl arXiv:2002. 10619, 2020.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Commuicctionn efficient. learning of deep networks from decentralized data. In Arlificial Intelligence and Slalistics, pp- 1273-1282. PMLR, 2017.
H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang, Learning differentially privale recurrent language models. In International Conference on Learning Representations, 2018.
Nya Mironov. Rényi differential privacy. In 2017 IEEE 30th Compuler Securily Foundations Symposium (CSF), pp. 263 225.275.22 IEEE, 2017.
Felix Sattler, Klbus-Robert Müller, and Wojeiech Samek. Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints. IEEE Transactions on Neural Networks and Learning Systems, 2020.
Adi Shamir. How to share a secret. Commun. ACM, 22(11):612 111-613,,1661111, nov 1979. ISSN 0001-0782.
Virginia Smith, Chaokai Chiang, Maziar Sanjabi, and Ameet Talwalkar. Federated multi-task learning, In Advances in Neural Information Processing Systems, 2017.
Harini Suresh, Jen J Gong, and John V Gullag. Learning tasks for multitask learning: Heterogenous patient populations in the icu. In International Conference on Knowledge Discovery & Data Mining, 2018.
Huiwen Wu, Cen Chen, and Li Wang. A Theoretical Perspective on Differentially Private Federated Multi-task Learning, arXiv e-prints, art. MrXiv:2011.07117) November 2020.
Liyang Xie, Inci M Baytas, Kaixiang Lin, and Jiayu Zhou. Privacy-preserving distributed multi-task learning with asynchronous updates. In International Conference on Knowledge Discovery and Data Mining, 2017.
Tao Yu, Eugene Bagdasaryan, and Vitaly Shmatikov. Salvaging federated learning by local adaplation. arXiv preprint arXi::2002.0440 2020.
Yu Zhang and Qiang Yang. A survey on multi-task learning, arXiv preprint arXiv:1707.08114, 2017.
Yu Zhang and Yan Yeung. A convex formulation for learning task relationships in multi-task learning. In Conference on Uncerlainly in Artificial Intelligence, 2010.
13

---

Definition 4 (fz-sensitivity). Let f u + Rd be some arbitrary function, the ez-sensitivity of f is defined as Published in Transactions on Machine Learning Research (04/2023)
A Appendix
A.1 Privacy Analysis: Proof for Theorem and 2
In the proofs of Theorem 1 and 2 we follow the line of reasoning in Abadi et al. (2016), which analyzes the privacy DPSGD. We first state the following lemma from Abadi et al. (2016).
Lemma (Abadi et al., 2016, Theorem 1) There exists constants er and 02 such that given the sampling probability p 2 and the number of steps T, for any cppT, DPSGD is diffferrreyt private for PVTlog(1/6)
m
any 5 if choose c2
To prove Theorem 1, we also need the following definitions and lemmas.
max A adjacent D.D'Eu
IIf(D)
(14)
Definition 5 (Rényi Divergence). (Mironov, 2017, Definition 3) Let P. Q be two probability distribution over the same probability space, and let p. q be the respective probability density function. The Rényi Divergence with finile order + is:
D.(II1 - In / ((x)
p(1)
dx
(15)
g(x))
Definition 6 ((a, e)-Rényi Differential Privacy). (Mironov, 2017, Definition 4) A randomized mechanism f:D D R. is said to have (a, -)RRégi Differential Privacy if for all adjacent D, D' ED il. holds that:
D&L((D))((D') S..
(16)
Lemma 2. (Mironou, 2017, Corollary 3) The Goussian mechanism is (a, (22(22)))))/))) - Renyi Differen- tially Privale.
Lemma 3.
(Mironor, 2017, Proposition 3) I f is (a, RRDDP, then it ( a-1 8)P for all 6 O.
log(1/6)
We begin by proving the first part of Theorem 1, where g + m.
Proof for Theorem 1: g o m. Note that aggregation step in line 8 of Algorithm 1 can be rewritten
St
9k
min
1,
11.9 ++1
?
l12
ON
(17)
kes,
w
9k
min
1,
110/t1112/
?
+ 0,
7 Laad
(18)
kES.
9k 1+1 min
(1 "111tt11
(1,
?
+ N
0,, ( q0
2 " Illaxa )).
(19)
LES
From here, can directly apply Lemma 1 with set to be 4. Hence, conclude that when q + m, there exists constants ea and C2 such that given the number of steps T, for any ci L M is (e, diffferreetl private for any 8 > 0 if we choose 02 TTggo
j
This proof can extend to the case where gives more specific bound on the variance o² in the case where g m.
In the remainder of this section, we provide proof that
Proof for Theorem 1: m. Define H' - D. x W + W as
= wt + k(D,,,").
(20)
14

---

Published in Transactions on Machine Learning Research (04/2023)
As result, we have (fD.)),, w² = (ID:), (() T²) B
By Lemma 2, M is (a, /ddoaaaa/)o//o))oaoooaaaa)))
Differentially Private. Note that
(A2H²)² = max max adjucent DDDD
(HD,,, ... DD."" , }) (HD,, ...
Dmn Dmn ,
(21)
(2 w³) (D w²) 2
max
max ] adjacent Dy.Dj ED)
(22)
|1 w²) hi (D', ||
(23)
m2 adjacent Dj, DED
m² 3 max
(24)
Hence, by sequential composition of Rényi Differential Privacy (Mironov, 2017, Proposition 1), M is
(a, E. 20 max,(4 (Ag)))))))) / --DDP.
By Lemma we know that M is EE 2a max, (A2h)))) /m22 + log(1/8) a-1 DP.
Plugging in a 1log(1/8)
47y Tlo((/88
we have
cm
T
2a max 2 /m3 + log(1/0)
log(1/6)
S
207 2 /mm +
(25)
1-1
1
i=1
24
(log(1/5)
log (1g(1/))
+
(26)
47 T log(1// 1/8)
(18(1188
m2
y2
C
(27)
2
(28)
Hence, MIT is (c,6)-DP if we choose 47 4T T log(11/)
em
By Theorem 1 and Billboard Lemma, it directly follows that Algorithm 1 is (c, OJDP.
Proof for Theorem 2. Theorem 1 shows that Algorithm 1 consists of a (c, 6D process to produce global model. After that each task learner trains local model with the DP global model and its private data. By definition of BP, it directly follows that Algorithm is (e, o)-B..
A.2 Convergence Anooosis(non
We first present the formal statement of Theorem 3.
Theorem (Convergence under nonconvex loss). Let fü be (L A)- Assume y is sufficiently large such that 7 2 max |/ fo (wff; ll2. Further let fx min fn(w; w) and p 2. If we use fired learning rate 7/t ()+ Algoritho 1 satisfies:
m
(4 ( 22) }
2A + E(Lp +Xp=
Em (fa(w w0) - f2)
V (
mT
(29)
o (L+ 1
p²
Ete
TIE BiE
LdX+ dx +d dX2//p2 <ddx+d>+dd>>ppp 00.
Ldx
+
T
mE
where
B, = max fi (x(f;;
(30)
15

---

Published in Transactions on Machine Learning Research (04/2023)
Take T ===0 =0
Let a chosen as we set in Theorem
[ddz
m
the right hand side is bounded by
o (1 ( 122) +
f 1²1 5
mf
(31)
L -+
L L+1+ 1
log(1/0) e2
+0
D B+O
E
mE
Proof for Theorem 6. Let w = arg min,, fa(w;; w*). Let I2 be the random variable indicaling whether task k is selected in communicttio round t. Note that the probability task learner is selected in any arbitrary
m
communication round ps
q
Thus E[[ = Pk -. By L-smoothness of fk. we have 2.
m
q
L - w
E [ffx( ; ) fu( w-)] s E
f ( w w w
(32)
L
(V ) me T V La (wfs vtt)
+ マ
|1 V S& (=(
(33)
x M.pi n:Pk
(34)
2
In the case where t # 0 mod E, i.c + not a communication round, ū++1 w Thercfore, we have
L 1+
E[fa(utt) ++1 ; a - fu(wi: <
2
TpPk)
(35)
2
In the case where t + 1= mod E, we have
L + X
(Ick +1;
-) ( Sa(wi; x') < E[ ; w Sa Su + [0'
11 P
1:Pk
[V 7')?.
2
(36)
It suffices bound B:
X
> 11 11
B
1w 10 112
(37)
2
/3++||||
| ++1 W w1+1|||
(38)
2
A
[[[|
g 1|2 B
2w 1+1
कt 12
(39)
2
A
E|| - . ||
(
wh) + 2( 1)
(40)
2
A
[| - -
a+++ Te [2| 4|l
f+1
7 + 4E ++++ Th
||||
++1
+++1
(41)
2
A
E :ott - + 1
E Ell ( + || ++1 - w
+4
E
||||
e+1
112
2
C1
C2
(42)
16

---

Published in Transactions on Machine Learning Research (04/2023)
where the first and third incquality follows from Cauchy-Schwarts Inequality: B[XY] can then upper bound C1 and C2-
YY We
22
C = E
9k
min
[1. 11ob 11
7
+
3
(43)
KESt
2
E
m
+++ 9k
min
(1.T0e=-11
(I,
11o 112
?
+ 1
(44)
k=1
E
m
7+ 9k
min
(I, 11// 11
(1100--11
+ vod
(45)
k=1
1+1 min (1, 1, "116=t1112 )(
11 vida
[ 9k
(46)
k=1
m
E
1t 1 me ff(((() min (I V
vddo
(47)
my V min
min (
+ vdo
(48)
(1.
Denote h(11
Em=
11 [((of(( min
We have:
2
C2 -
110 ++1 11
(49)
x 2
2
fu +
(50)
2
B1+1
(51)
X
Plugging the bounds for C₁ and C2 into B yields:
A (h(t) + vdo) (M(O)+ + vdo v22+++ - B B B
(52)
2
Denote the right hand side as B(t), we have
L X
EIfa (w (1tt1, 1 - Se(wh; 5 3(1)+ +
11
T/:Pk
(53)
2
Let & = REff (1ff; - f' (wrt; 77)),, we have
L +
8 S 6, +(() +
Mi TIPk
|V
(54)
2
In the nonconvex case, have
L X
mpPk
2 2 M
VV
(w
( 12 B(t) - fa( σ.) -
(55)
2
17

---

Published in Transactions on Machine Learning Research (04/2023)
Summing over & on the left handed side, when y is large enough so that no clipping happens have
Σ
m
Σ
(( ThPPss
L+ A
2 M1
[u wn - s(t))
Pk
11111 mod E
(56)
+
Σ
N1Ps
11
2
1+1 mod E
Σ
T&P
L A
V (u x²)| m1t(t)
2
[+1== mod E
R=
(57)
L +
+
Σ
TPP
2
2
++120 mod E
1=1
Σ
m&P
L
+
X
nEp
2
V ffc
2
++1=0 mod
A
2
2d doz 201
(l² +
2V + -2/
Br41
ml(l) + m
B2+1
(58)
x
X
L L+X
+
Σ
7&P
p2pp
Vfa(w[: w1)l
2
1+170 mad E
T-1
L+1
2
Ct
1/P
2
(59)
Σ
A
2 tt
G A/mm (vāo+ + / Bi+1 n.r
Am
do² - 201 a
2d
Be+1
2
1+1 mod E
fu( O. T
(60)
k=1
where Ge = 2
V 13.
Picking 1)1
p + yields
k=
Aym Vdo + / P
Σ
p²
G
Gr 2
Am
do + +201
2d
Be+1
(61)
2 + 2 + 1)1)
p (p2 +11
X
#+10 mod E
p' (p² + 22)
Σ
Cf
(62)
#1 mod 2( I + (p2 + 1)1)2
m
Σ (u(u, w²) fz.
(63)
This is equivalent to
T-1
Σ + + Σ
-2
2 Vdo + V B
m G. - L + +
A
Am
do² + =20
2d
x Bi+1
q
1=0
++10 mod E
(64)
+2 F-1 1 ( )
1 222)
2
L X
fa(w w.)
(65)
E
E L+>+2 32
2 1+ 1 x)) +
f A
La (u w0 - fī-
(66)
E
L
+ X+ +
31
18

---

Published in Transactions on Machine Learning Research (04/2023)
Hence, we have
1)
1)2
2
E
G. G. Am
(Vdo +
[vdo+V3
+V>
V Be+1
2
1
+
Σ
G
(67)
p
++1=0 mod E
t+1 mod E
# a + P
1 ++7+
L+
1² x² +
fu(wg;
w²)
f%z
(68)
E
1=1
Σ
(LAm + mX2
do² + +201
2d
x Be+1
+ mdBB+11
(69)
D2
(+130 mod
This implies
T-1
L+ + ) +
Σ G 2
fu(u ū0) - fb
(70)
E
L 32
1=0
k=1
(LAm (LAm+mX²+ _
2d
MABB+1
+2
Σ
do²² + 201
Be+1
(71)
p2
1+120 mud
Hence, we conclude that
fie T-)12
(72)
mT
1=0 1=1
L x+ x
Ek=1 (fa(wl; w0) f%)
+2p
m.
(73)
OLA 12 12+ 82
ETE
do² + +200 2
2d Bie + 2Bix
A
T
(1 122)
Ek=I (f - fF)
+hpp"
mT
(74)
o o }
IT/E
(Vito + v2BiE 2
T
(4 (L+ 12) +
Exar - O O X -
dm? 9²
ETIE BiE
mT
T
(75)
Ldd - dx2 + dx2/p²
+
F
Taking a 227V T (0)
and T =0
m xxyy>
we have
me
(4 + ( + B +2x)
2A
Σm (Je(W];; 50) - -52)
Vf(
(76)
mf
mf
o 0I+
2
IT-1 B
pd
+ = [+++ A log(1/5)
(77)
T
/3 2
19

---

Published in Transactions on Machine Learning Research (04/2023)
A.3 Convergence Analysis (Convex):
We first present the formal statement of Thcorem 5.
Theorem 7. Let fü be (L+ Xm-smooth and (μ + s)-strongly conver. Assume ry is sufficiently large such that
Y 2 maN V/ fu 1l2 Further let w arg min fw(w; w*), where x 1 m 2 wx and p = 2. 1f
fixed learning rate TJ 17 -p*+Xp for some constant c such that 0 S 1 Tp 2) + S 21
Algorithm satisfies:
AT S 27
1
2 + 2 ) Do md do? + 2Vdo V
+
md +22do B }B)
(78)
1
where A. = - /x(wx
- Jn (103s w*) and B = max, maxk Jn (wo w+)
Let a be chosen as in Theorem 2, then there erists T = O
m Adj?
such that
1 (LLA0 S 2
log(1/0) me²
log(1/8)
+ +O
AT
O (B)
(79)
m
m
2
Proof for Theorem Let w = arg min, fww w*) Let I be the random variable indicating whether task l is selected in communication round t. Thus E[[ = Pk. By L + X-smoothness and μ + A-strong convexity of fk, we have
L V (u Tt) w w1) + 2/2 l -- w2
( +1 ;,6'' fu wh) < E
(80)
2
E
(V I'm σh)) fu w")?
[
(81)
+ X
722 nhPk
[V 1²
(82)
2
+ X 2
MePk
1ePk
2(μ ()(f() σt) - f(wf; w²))
(83)
2
+
X 2
2 1EPK
1pPk
2(μ + ()(f( wh) - f(w t w²)).
(84)
In the case where 1+1 1 # 0 mod E, i.e. L + is not a communication round, 71+1 = w.. Therefore, we have
L + X
[[fe(wtt) ( fa [u wh)) <
T.P% 2(p + A)(f(w); w²) f(w w))). mip
(85)
In the case where L + ++1 mod E, we have
Elfa (lfa ++1, :ooft:/att) ( - fo(wi; [ E( w fo(wwtt w
(86)
+ ((L + 2)1ir - 22ppk)) (p + 2)({(u(: -(E)-
20

---

Published in Transactions on Machine Learning Research (04/2023)
It suffices to bound B:
X
B
E
[wtt 2 112 11 111
(87)
2
E|| 1
+1
2 k
1+1 2" 11
(88)
2
A
B
20 Th x 1|]
(89)
2
A
क || v Bill (ijt+1 w²) + 2( [+1 [)+2]
(90)
2
A
Bl||
- क ||3| v El| 2ح - क. |2| 4 ++1 + 2|2 + 4B||| wh |||| ++1
(91)
2
A
BI 1 - 13
El| 19| + 1u T 112 +4, Ell "6|13]|| : 1
112
2
C1
(92)
where the first and third inequality follows from Cauchy-Schwartz Inequality: B[XY] S VE It. suffices to find the upper bound of C₁ and C2.
min (1. 119h
C = E
gh
+ Br
(93)
KESt
2
|11gtt1112 ll2 )
E
+++1
gk
s+1 min
(1,
?
+ 1
(94)
E
m
9k
min
1 112 (11--==12))
7
+ vdd
(95)
m
)((
11"
m
[++ gk
min
1. "11
7
+ Vido
(96)
11" Vdlo
1 n/ fu( min
(
(97)
)
2
n fu min 1, (4 (1.,
+ voo
(98)
Denote h(()
Em
(((() min (M
On the other hand,
2 2
C2 S
1
s+1
(99)
A 2
2
fulwtt (1tt1)
(100)
2
x B
(101)
21

---

Published in Transactions on Machine Learning Research (04/2023)
Plug the bounds for C1 and C2 into B:
B X (h(t) vdo) [())) vdd a +2 / 2
Bi+1
(102)
x
s A (tt)) =(t)(t) + do² + +2ddy/
2 B 2 Bett
(103)
Denoting the right hand side as B(1), we have
(lfx( - : - f Th) x')] s p(t) + ((L + A)miP' -2mpk) (μ+ A)(f(u);;
w²) fx)).
(104)
Letting 8 Elfa(uo(s - (w n8 ) have
o $ (1 ((L ADriPt - 2m.Pk) (n + X)) 8% + 3(t)
(105)
Summing over & on the left handed side, when is large enough so that no clipping happens we have
g++ < (1- ((L+ Nymip> - 2nx)) (a+ - 2))
Σ o% + m3(()
(106)
R=1
1=1
do² + +2Vooo / Be+i + x Bit1
= (1 - ((Tp² + xp² - -2Amn 2mp) (n + 2))
Σ
8y + =nd
k=1
(107)
Let A, =It 8%. Assume max B, = B. Pick C mn dod / B 3
(1Lp² (1L=++ -21/ A1+ have
- ((Lp² + Ap² 22)m 2mp) (μ + A)) (A. - -O).
(108)
Note that in the case where + +1 0 mod F, we have
A+11 I (1 ((Lp² + >p - 2n.p) (μ + A)) D.
(109)
< (1 ((Lp² xx² -2)) 21:p)) (μ 2)) A
(110)
Choose 7/- = 1 22
for some constant such that 0 < (1 ((Lp² + >p2 - 2X)n? - 2mpp) (n + X))
<
2- We have
1.
(6- (62-20) (0) +1
-4
(AI-C)
(111)
L+ 21
1102-20)(u
(62 (1 + 11
(62 -2-10-- (11-10-
-2) (2-)0 + A)
E-1
ACB
1
(112)
-+ 21 #
L+ X
23
This is equivalent to
E (62 2c) (c)
A41 - D
1
(Ai-Et1-))
(113)
I a
22 x
22

---

Published in Transactions on Machine Learning Research (04/2023)
where
(c2=2c)(++))
D =
(114)
1 (6t (c²=20)(u+A)
1
L+d- 3
md do² + 2v / B
(115)
- (1-
(02 20)-222(+2)))
L+7 3
md (do² 22dd v 2 }
2 B + - E md do + 2V
(116)
1
Apply recursively to all t, obtain
T (2-))= + N
(c2 2c)
AT v
-) D)+
(117)
L + x - 23
p
Do mX (do=
12 B B 2 (1)) )
md (do² + +2vdo B }
S 27
1
do²
2Vdo V
x
(118)
2
Take c Tlo((1/)
and we can find T = 0 Adps
such that,
log(1/0)
log(1/5)
+ (mB)
AT < 2T
Au
O (mB)) +
(119)
e²
1
Divide both side by we have
1
1
log(1/6)
log(1/8) + +0(B)
Ar
Do
O (B)
(120)
m
2T
m
me²
2
In the non-private case, our Theorem could reduce to the following corollary, which is of independent. interest.
Corollary 8. When o = 0, Algorithm 1 with (L + Assmohhh and (μ + X)-strongly conver fo satisfies
AA m
S 2T
-oo
1 + 1
B
(121)
m
2
A.4 Datasets and Models
We summarize the details of the dalasets and models we used in our empirical study in Table 2. Our experiments include both convex (Logistic Regression) and non-convex (CNN) loss objectives on both text (StackOverfloww) and image (CelebA and FEMNIST) datasets. We provide anonymized code in the supplementary material for reproducibility Our code makes use of the FL implementation from the public repo of Laguel et al. (2021) and Li et al. (2021).
23

---

Published in Transactions on Machine Learning Research (04/2023)
Table 2
Dataset
Number of tasks
Model
Task Type
FEMNIST (Cohen et al., 2017; Caldas al., 2018)
205
4-layer CNN
62-class image classification
StackOverflow (tff) CelebA (Liu et al,, 2015; Caldas et al., 2018)
400
Logistic Regression
500-class tag prediction
515
4-layer CNN
Binary image classification
A.5 Hyperparameters
Each fixed privacy parameter could be computed by different combinations of noise scale d, clipping norm 7, number of communication rounds T, and subsampling rate 4. Tn all our experiments, we subsample 100 different tasks for each round, ie. q = 100, to perform local training as well as involved in global aggregation. For FEMNIST and CelebA, we choose a E {0.02, 0.05, 0.13 and y E f0.2, 1}. For StackOverflow, we choose a e {0.01, 0.05, 0.1} and E 0.5, 1}. We summarize both utility and privacy performance for different hyperparamcters below.
FEMNIST, y=
FEMNIST, 0
FEMNIST, y=
0.8
0.8
Accuracy
Test Accuracy
Test Accuracy
0.6
0.6
0.4
0= 0.1
0.4
0= 0.1
0.4
0.1
0=
0.05
0= 0.05
0= 0.05
0.2
0.02
0.2
d= 0.02
0.2
0.02
non-private
non-private
non-private
0.0
0.0
0.0
200
400
600 800
1000
200
400
600
800
1000
400
600
800
1000
Communication rounds
Communication rounds
Communication rounds
FEMNIST, y= 0.2
FEMNIST, y= 0.5
FEMNIST, y
5
0.1
0=1
a= 0.1
4
a= 0.05
15
0= 0,05
0= 0.05
a= 0.02
0.02
40
000.2
2
20
5
1
0-
200
400
600
800
1000
200
600
1000
800
1000
Communication rounds
Communication rounds
Communication rounds
Figure 4: FEMNIST results
A.6 Comparison with FedProx
Besides FedAvg, we also compared private mean-regularized MTL with other methods that aims to train a global model privately. In particular, we studied private FedProx (Li al., 2020b) as an alternative global baseline. Note that although the local objective being solved in FedProx is similar to that in mean-regularized MTL, FedProx is fundamentally different method to handle data heterogeneity in FL from MTL. Specifically, FedProx learns a global model where cach client solves an incxact minimizer by optimizing local empirical risk with a regularization term. We instead explore learning a mulli-lask objective where each client solves a regularguel objective and learns a seperate, client-specific model. The results are shown in Figure 7. In all three datascts, private FedProx very similar to private FodAvg under different private parameters and worse than private MTL. particular, FEMNIST and Stackoverflow, private MTL significantly outperforms training a private global model (FedArg and FedProx), for all e's.
24

---

Published in Transactions on Machine Learning Research (04/2023)
CelebA, y= 0.2
CelebA, y= 0.5
CelebA, y=1
0.9
0.9
Accuracy
0.8
Test Accuracy
0.8
Test Accuracy
0.8
0=0.02
0.02
0.02 non-private
0.7
non-private
0.7
non- poinppivvte
0.7
0.5
0.5
0.5
200
400
600 800
1000
0
200
400
600 800
1000
200
400
600 800
1000
Communication rounds
Communication rounds
Communication rounds
CelebA, y 0.2
CelebA, y= 0.5
CelebA, y =1
0,1
1.5
a= 0.05
0.05
a= 0.05
0.02
o= 0.02
a= 0.02
1.0
0.5
0.0
200
400
600
800
1000
0
200
600
800
1000
200
400
600
800
1000
Communication rounds
Communication rounds
Communication rounds
Figure 5: CelebA results
StackOverflow. y= 0.5
StackOverflow, y=1
StackOverflow, y=
0.3
Accuracy
Test Accuracy
Test Accuracy
0.2
0.2
0.2
0= 0.05
0= 0.05
0.05
0.1
0.1
0.1
== 0.01
0= 0.01
0.01
non-private
non-private
non-private
0.0
0.0
200
400
600
0
200
400
600
200
400
600
Communication rounds
Communication rounds
Communication rounds
StackOverflow, y= 0.5
0.5
StackOverflow, y=1
StackOverflow, y
15
50
250
d=
0.1.1
a= 0.1
40
200
a= 0.05
10
a= 0.01
a 0.01
a= 0.01
100
50
0
0
200
400
600
200
400
600
0
200
400
600
Communication rounds
Communication rounds
Communication rounds
Fig Figure 6: StackOverflow results
Gittal hodPrix
3.5
0.5
0.20
0.4
0.2
0.350
Global Fechvox
Gatel NosToo
3.0
03
0.5
3.0
3.5
1.5
63
35
1.6
3.5
25
85
10
(a) StackOverflor tag prediction
(b) FEMNIST
(c) CelebA
Figure 7: Comparison of PMTL and training private global model[FedAve) Boee//////////r////////////
25

---

Published in Transactions on Machine Learning Research (04/2023)
Local
PMTL
PMTL+best finetuning
0.1c
0.8c
2.0c = 0.1c 0.8 c 2.0
SterkOverflow
.318
.305
.323
.324
FEMNIST
.618
.371
.498
.621
.663
.640
.681
CelebA .694
.633
.641
.667
.801
.817
.818
Table 3: Comparison between PMTL and Local training, For the problems, we pick the finetuning method that yields highest test accuracy from all the methodd introduced Section 5.3.
results on the non-convex
A.7
Comparison with pure local baseline
While federated learning could yield better utility performance compared to pure local training, this is not always true when we apply client-level DP during federated learning. When a small e is enforced, accuracy for federated learning could drastically drop (see Section 5). In this section, we compare our PMTL with training pure local model. In addition, since local finetuning does not incur additional privacy cost in our sccnario to protect client-level privacy, also compare finttuinnnnun with local training. We present the results in Table 3. For StackOverflow where convex model is used, finetuning with sulficiently many rounds should be the same as training the local model. For the other two datasets where neural network is trained, training a purely local model performs worse than PMTL under large and PMTL with the best local finetuning objective under all e we evaluated. We note that the goal of our work is not to argue that MTT. better than global/loca baselines all scenarios, but rather to show that it possible to provide effective private training methods for commonly-used MTL objectives.
A.8 Comparison to PP-SGD (Bietti et al., 2022)
In this section, we compare PP-SGD (Bietti et al., 2022), similar form of model personalization in federated learn- ing, with our proposed PMTL Stackoverflow tag predic- tion. PP-SGD aims to solve the following local objective: min fi fg v)) := E(y, (w + 8,)Tz) x). It is worth noting that when the model is neural network, there isn't a straightforward extension for this method to sup- port personalized model weights for each layer, contrast our method where the mean-regularization term is calculated by taking the difference of the entire model parameter vector of global and local model. Therefore, for fair comparison, we run PP-SGD on the logstic regression Stackoverflow tag prediction task. Recall that different from the stackoverflow task in the original Bictti et al. (2022) paper, we look at a slightly different setting where feature dimension is 10000 and number classes is 500 (instead of 5000 features dimension and 80 classes in Bietti et al. (2022)). The results are shown Figure 8. As we see, when we require strong privacy < 1), PP-SGD gives worse privacy-utility trade-off compared to method. When privacy is weak, our method and PP-SGD achieves similar utility under same privacy budget.
0.30
accuracy
0.15
PMTL
PP-SGD
0.00
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.3
4.0
Figure Comparison with PP-SGD on Stackover- flow tag prediction
26


Reviews:
[{"review_id": "dGc7xhEQSA", "reviewer": "Reviewer_kwvr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper seems well-written, with substantial theoretical study. However, I have the following concerns about the paper,\n\nThe authors claim that differential privacy is not applicable to multi-task learning due to reduced performance. If this is true, then is the solution to consider a weaker privacy notion? Since here we are learning m separate models, why not each client train a differentially private model using local data without interacting with others? Do we really need to give up on differential privacy and adopt JDP?\n\nWhy does satisfying JDP make sense? To me, it does not make sense. Consider this scenario that each client trains a model independently without differential privacy. In this case, JDP is satisfied.\n\nThe convergence bound in (9) does not make sense. As the T goes to infinity, the bound does not go to zero.\n\nDue to the above concerns, the paper needs substantial revision.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this work, the authors study the multi-task learning problem in a distributed setting under privacy constraints. They (empirically) show that the original definition of differential privacy is not applicable to this problem. So they use a weaker version of differential privacy and developed an algorithm with a convergence guarantee to solve this learning problem.\n\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Include the Answer to the questions asked in the Strengths And Weaknesses section. ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vJwegpOV0Z", "reviewer": "Reviewer_iY8Y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\nThis paper brings the billboard privacy for MTL which is an interesting concept to describe the privacy objective.\n\n#### Weakness\nMy main concern is that the proposed approach is known in the literature while the main technical difficulty is largely ignored in the paper. Please correct me if I am wrong or misunderstand your point.\n\n1. Billboard privacy does not bring much knowledge compared to existing federated learning implementations. The billboard privacy can be seen as a combination of differential privacy and input privacy which has already been implemented in the current federated learning systems (Bonawitz et al., 2019). The input privacy can be achieved through multiparty computation (MPC) protocols or secure aggregation protocols in the FL setting (Bonawitz et al., 2017). The input privacy and output privacy, e.g. differential privacy, are orthogonal concepts and their implementations can be easily combined.\n\n2. This paper assumes there to be a \"trusted global learner\" which is a very strong assumption. However, this paper only aims to privately optimize objectives like (3). This can be easily done with multiparty computation (MPC) protocols or secure aggregation protocols in federated learning without a \"trusted global learner\". (And we can add client DP to defend m-1 out of m collusion). The BP concept provides no more insights on how to implement this crucial part which means one can not still circumvent input privacy.\n\n3. The authors claimed that traditional client-level DP does not work, but their proposed algorithm PMTL is essentially traditional client-level DP. In Algorithm 1, one can move the clipping and add noise in line 8 to line 6 and make it client-level DP. It seems to me that the authors are arguing that in order to satisfy Definition 1 you have to choose a very large noise such that the algorithm is useless while in order to reach Definition 2 you can use reasonably small noise. However, definition 1 is certainly not a good privacy concept in the MTL case and is not used in existing implementation ---- again we are using input privacy + differential privacy. \n\n#### Reference\nBonawitz K, Eichner H, Grieskamp W, et al. Towards federated learning at scale: System design[J]. Proceedings of Machine Learning and Systems, 2019, 1: 374-388.\nBonawitz K, Ivanov V, Kreuter B, et al. Practical secure aggregation for privacy-preserving machine learning[C]//proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. 2017: 1175-1191.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies multi-task learning (MTL) in a distributed environment with privacy guarantees. They show that the standard differential privacy concept does not capture the MTL and propose to use billboard privacy (BP), a variant of differential privacy that fits the scenario. They propose a private multi-task learning algorithm (PMTL) which is shown to satisfy BP and enjoy a standard convergence rate. Empirical evaluation is provided. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My concerns mentioned above are relatively high level and may be difficult to address without a significant change of paper. However, I would like the authors to make the following changes/clarifications:\n- Is the algorithm a weaker version of existing implementations of secure aggregation + client DP? E.g. this paper replaces secure aggregation with a trusted global learner. \n- Could you explain or convince me the value of introducing BP in addition to existing input privacy + differential privacy?\n- A large body of MPC/secure aggregation protocols literature are not discussed in the paper.\n", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "K51JccP8Sp", "reviewer": "Reviewer_C5Fx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n-The problem is well-motivated and interesting \n\n-The BP notion seems practical for their MTL problem, although there is some confusion around their definition (see below)\n\n-The paper is well-written for the most part, but there are some aspects that are unclear \n\n-The numerical experiments seem relevant \n\n**Weaknesses** \n\n*Main weaknesses*:  \nThe two main weaknesses that I see are: 1) Def. 2 is not exactly clear, nor is justification for the claim that BP provides strong and meaningful privacy guarantee for MTL (even if other clients collude). 2) Convergence guarantees: the nonconvex one seems to be vacuous (and the strongly convex one is difficult to decipher). I'll start by discussing these points and then list other (more minor) weaknesses. \n\n1) In Def. 2, the notations $f_i$, $g$, and $\\Pi_i D$ are never defined as far as I could tell. (Some of these notations become apparent later in the paper.) \n\nThis makes the definition unclear and leads me to wonder about the validity of the claim that Def 2 provides strong privacy. For example, if $m=2$, then Def. 2 seems to mean that $\\mathcal{M}_1$ is insensitive w.r.t. changes in data set $D_2$? However, if client 2 were an adversarial eavesdropper, then client 2 could use its knowledge of $D_2$ to try to decode $D_1$ since $\\mathcal{M}_1$ might depend heavily on $D_1$ (as Def. 2 does not require $\\mathcal{M}_1$ to be insensitive to D_1$). Thus, (my understanding of) Def. 2 does not seem to provide a strong guarantee against collusion, as claimed on p.4. \n\n2) In Theorem 3, the assumption that $f_k$ has uniformly $\\gamma$-bounded gradient implies that the trivial (DP) algorithm that simply outputs any collection of $w_k$ independent of the data achieves utility of $\\gamma^2$ (where utility is measured in terms of average squared gradient norm as in Thm. 3). On the other hand, the second term in Eq. 10 is at least as large as $\\gamma^2$ if $B_{i, E} > 1/d$, which would seem to be the case in any interesting/practical application. Thus, it appears that Theorem 3 is vacuous. \n\n*Other weaknesses*\n\n3) last sentence of last bullet on p.2 is a bit unclear: \"privacy/utility benefits\" relative to what? relative to global (private or non-private?) baseline? \n\n4) p.2: paragraph on MTL background is a bit unclear, as the difference between MTL and federated learning (FL) feels a bit murky here. The authors say that MTL is useful in applications where datasets are shared among  different entities/clients. By contrast, in FL, data is typically decentralized and not shared among clients. Thus, the last two sentences of the paragraph about MTL being useful for cross-device FL by allowing data to be shared was a bit confusing to me. \n\n5) p.2: paragraph about FL: it would be good to cite other works on DP FL here. For example, I suggest citing the following works which are very relevant and seeing the citations therein: \n\nLowy, Andrew, and Meisam Razaviyayn. \"Private federated learning without a trusted server: Optimal algorithms for convex losses.\" arXiv preprint arXiv:2106.09779 (2021). \n\nLowy, A., Ghafelebashi, A., & Razaviyayn, M. (2022). Private Non-Convex Federated Learning Without a Trusted Server. arXiv preprint arXiv:2203.06735.\n\n6) brackets in Eq. 2 (inside the sum over $k = 1, \\cdots, m$) would clarify \n\n7) Theorem 1: what notion of DP is being claimed here? client-level DP? Be specific. \n\nAlso, I was expecting BP here. Maybe lead with Cor. 2 as your main privacy result and mention that you also provide ___ DP..\n\n8) $l_k(w_k)$ in Eq. 8 does not seem to be defined. Earlier, $l_k(x_i, w_k)$ was said to be a loss function but what is $l_k(w_k)$? \n\n9) Theorem 3. Besides being apparently vacuous, it seems that you did not correctly optimize for $T$ in going from Eq. 9 to 10. Also $\\mathbb{E}$ is missing on the LHS. Also, you should explicitly state somewhere that since you are assuming uniform $\\gamma$-Lipschitz continuity of $f_k$, you do not need to clip in Algorithm 1. Also, it is strange that some terms on the RHS of Eq's 9 and 10 (and LHS of Eq 10) have correct units (scaling with squared gradient norm), while other terms (e.g. the first term on the RHS of Eq. 9) have incorrect units: please either be precise and write each term with the correct scaling factor (preferred) or be informal and omit all scaling factors besides $m, d, \\epsilon$. \n\n10) Theorem 5. Similar comments as above with Thm 3. This theorem is strangely written and difficult to parse. I suggest simplifying the bounds to clearly display the dependence on $m, d, \\epsilon, \\lambda, \\mu, B$. Introducing additional notation such as $C$ and $\\Delta_T$ is not necessary and makes things less clear. Instead, I expect that you can properly choose parameters, align units, and write the final bound as a sum of 1 or at most 2 terms.\n\n11) Experiments: comparison vs. Bietti et al 2022 would have been nice, as they seem to be the main algorithmic competitor for DP MTL (based on the introduction). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers multi-task learning (MTL) under differential privacy/billboard privacy (DP/BP) constraints. They consider the mean-regularized MTL objective and propose using BP to protect client/task privacy. They give a BP algorithm for solving their objective and analyze its convergence. Finally, numerical experiments are provided, showing benefits of their approach vs. training a single global model. ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see **Weaknesses** above. ", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In reviewer discussion, all reviewers find the paper somewhat borderline but acceptable as the authors have addressed the concerns raised in the reviews.
Decision: Accept (poster)