Private Multi-Task Learning: Formulation and Applications to Federated Learning
Authors: Shengyuan Hu
Steven Wu
Virginia Smith
Venue: ICLR 2022 Submitted (2022)
Keywords: privacy preserving machine learning
large scale machine learning
multi-task learning

Abstract: Many problems in machine learning rely on multi-task learning (MTL), in which the goal is to solve multiple related machine learning tasks simultaneously. MTL is particularly relevant for privacy-sensitive applications in areas such as healthcare, finance, and IoT computing, where sensitive data from multiple, varied sources are shared for the purpose of learning. In this work, we formalize notions of task-level privacy for MTL via joint differential privacy(JDP), a relaxation of differential privacy for mechanism design and distributed optimization. We then propose an algorithm for mean-regularized MTL, an objective commonly used for applications in personalized federated learning, subject to JDP. We analyze our objective and solver, providing certifiable guarantees on both privacy and utility.  Empirically, we find that our method allows for improved privacy/utility trade-offs relative to global baselines across common federated learning benchmarks.

Under review as a conference paper at ICLR 2022
PRIVATE MULTI-TASK LEARNING: FORMULATION AND APPLICATIONS TO FEDERATED LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Many problems in machine learning rely on multi-task learning (MTL), in which the goal is to solve multiple related machine learning tasks simultaneously. MTL is particularly relevant for privacy-sensitive applications in areas such as healthcare, finance, and IoT computing, where sensitive data from multiple, varied sources are shared for the purpose of learning. In this work, we formalize notions of task-level privacy for MTL via joint differential privacy (JDP), a relaxation of differential privacy for mechanism design and distributed optimization. We then propose an algorithm for mean- regalaregu MTL, an objective commonly used for applications in personalized federated learning, subject to JDP. We analyze our objective and solver, providing certifiable guarantees on both privacy and utility. Empirically, we find that our method allows for improved privacy/utility trade-offs relative to global baselines across common federated learning benchmarks.
INTRODUCTION
Multi-task learning (MTL) aims to solve multiple learning tasks simultancously while exploiting similaritiss/diffcreeee across tasks (Caruana 1997). Multi-task learning is commonly used in applications that warrant strong privacy guarantees. For example, MTL has been used in healthcare, as a way to learn over diverse populations or between multiple institutions (Baytas et al., 2016 Suresh et al. 2018 |Harutyunyan et al. 2019); in financial forecasting, to combine knowledge from multiple indicators or across organizations (Ghosn & Bengio 1997 Cheng et al. 2020); and in IoT computing, as an approach for learning in federated networks of heterogeneous devices (Smith et al.
2017 ct al. Mansour et al. 2020). While MTL can significantly improve accuracy when learning in these applications, there is dearth of work studying the privacy implications of multi-task learning.
Hanzely & Richtárik 2020 Hanzely et al., [2200] Ghosh et al. 2020 Sattler et al., 2020 Deng
In this work, we develop and theoretically analyze methods for MTL with formal privacy guarantees. Motivated by applications in federated learning, we aim to provide task-level privacy[ where each task corresponds to a client/device/data silo, and the goal is to protect the sensitive information in each task's data (McMahan et al. 2018). We focus on incorporating differential privacy (DP) (Dwork et al., 2006), which (informally) requires an algorithm's output to be insensitive to the change of any single entity's data. For MTL, using task-level DP directly would require the entire set of predictive models across all tasks to be insensitive to changes in the private data of any single task. This requirement is stringent for most applications, it implies that the predictive model for task & must have little dependence on the training data for task k thus preventing the usefulness of the model (see Figure T
To cireumvent this limitation, we leverage a meaningful relaxation of DP known joint differential privacy (JDP) (Kearns et al. 2014), which requires that for each task k. the set of output predictive models for all other tasks except is insensitive to k's private data. As a consequence, the client's private data in task & is protected even if all other clients/tasks collude and share their private data and output models (as long as client keeps their data private). In contrast to standard DP, JDP allows the predictive model for task k to depend on k's private data, helping to preserve the task's 's utility.
Using JDP, we then develop new learning algorithms for MTL with rigorous privacy and utility guarantees. Specifically, we propose Private Mean- Regularized MTL, a simple framework for learning
In federated learning applications, where each MTI. task typically corresponds to client's local training task, this be cquivalently viewed as 'client-level' 'user-level' privacy (McMahan al. 2018.

---

Under review as a conference paper at ICLR 2022
Algoniti
Algoritte
Algorithm
D2
D2
Figure 1: An MTL problem consists m different tasks a learning algorithm that jointly produces one model for each task (Left). For example, in cross-device federated learning, each "task" may represent data from mobile phone client (as depicted), and MTL can be used to learn shared, yet personalized models for each client (Smith et al. (017). In traditional differential privacy, if the private data of task it (c.g., Task 1) changes, the models produced by the MTL algorithm should be indistinguishable from the models derived without changing data from task (Middle). In contrast, JDP allows model of task & to be dependent on task k's data while still protecting other tasks from leaking information about their private data (Right).
multiple tasks while ensuring task-level privacy. We show that our method achieves (e, JDP Our scalable solver builds on FedAvg (McMahan et al., (017). common method for communication- efficient federated optimization. We analyze the convergence of our solver on both nonconvex and convex objectives, demonstrating tradeoff between privacy and utility, and evaluate this trade-off' empirically on multiple federated learning benchmarks. We summarize our contributions below:
Our work is the first we are aware of to provide formal definitions of task-level differential privacy for multi-task learning objectives (Section 3p. Our definitions rely joint differential privacy and are applicable to commonly-used multi-task relationship learning objectives.
Using our privacy definitions, we propose Private Regullzeula MTL, a simple MTL frame- work that provides task-level privacy (Section 4). We prove that our method achieves (c, 6JDDP, and we analyze the convergence of our communication-effiiiee solver on convex and nonconvex objectives. Our convergence analysis extends to non-private settings with partial participation, which may be of independent interest for problems in cross-device federated learning.
Finally, explore the performance of our approach common federated learning benchmarks (Section 5). Our results show that it is possible to retain the accuracy benefits of MTL in these settings relative to global baselines while still providing meaningful privacy guarantees. Further, even in cases where the MTL objective achieves similar accuracy to the global objective, we find that privacy/utility benefits exist when employing the private MTL formulation.
2 BACKGROUND AND RELATED WORK
Multi-task learning. Multi-task learning considers jointly solving multiple related ML tasks. Our work focuses on the general and widely-used formulation of multi-task relationship learning (Zhang &
Yeung 2010). as discussed in Section 3 This form of MTL is particularly useful in privacy-sensityy
applications where datasets are shared among multiple heterogeneous entities (Baytas al. [0166] Smith et al. 2017; Ghosn & Bengio 1997). In these cases, it is natural to view each data source (e.g., financial institution, hospital, mobile phone) as a separate 'task' that is learned in unison with the other tasks. This allows data to be shared, but the models to be personalized to each data silo. For example, in the setting of cross-device federated learning. MTL commonly used train personalized model for each device in a distributed network (Smith et al. 2017 Liu et al. 2017).
Federated learning. A motivation for our work is the application of federated learning (FL), in which the goal is to collaboratively learn from data that has been generated by, and resides on, number of private data silos, such as remote devices or servers (McMahan al.) [017 Kairouz et al.)
2019 Li et al. 2020a).. To ensure client-level differential privacy in Fl., common technique is to learn one global model across the distributed data and then add noise to the aggregated model to sufficiently mask any specific client's 's update (Kairouz et al. 2019 McMahan et al. 2018 Geyer cct al. 2017). However, a defining characteristic of federated learning is that the distributed data are likely be heterogeneous, i.e., each client may generate data via distinct data distribution (Kairouz) et al. 2019 Li et al. 2020a). To model the (possibly) varying data distributions on each client, is natural instead consider learning separate model for each client's local dataset.
To this end, a number recent works have explored multi-task learning as a way to improve the accuracy of learning in federated networks (Smith al. 2017 Hanzely & Richtárik 2020 Hanzely
2

---

Under review as a conference paper at ICLR 2022
al., 2020 Ghosh al. 2020 Sattler al. 2020 Deng et al. 2020 Mansour et al. 2020). Despite
the prevalence of multi-tase federated learning, we are unaware of any work that has explored task-level privacy for commonly-use
multi-task relationship models (Section in federated settings.
Differentially private MTL. Prior work in private MTL differs from our own either in terms of the privacy formulation MTL objective. For example, Wu al. (2020) explore specific MTL setting where feature representation shared by all tasks is first learned, followed by task-specific models on top of this private representation. We instead study multi-task relationship learning (Section 3 which is a general and widely-used MTL framework, particularly in federated learning (Smith et al. 2017). While our work focuses on task-level privacy, there has been work on data-level privacy for MTL. which aims to protect any single piece of local data rather than protecting the entire local dataset. For example, Xie et al. (017) propose a method for data-level privacy by representing the model for each task as sum of public, shared weight and task-specifie weight that is only updated locally, and Gupta et al. (2016) study data-level privacy for mean estimation MTL problem. Finally, [L] et al. (2019) studies multiple notions of differential privacy for meta-learning, Although similarly motivated by personalization, their framework does not cover the multi-task setting, where there exists separate model for each task.
3 MULTI-TASK LEARNING AND PRIVACY FORMULATION
In this section, we first formalize our multi-task learning objective, which is a form of mean- regularized multi-task learning [Section3..] and then provide our privacy formulation (Section 3.2).
3.1 PROBLEM SETUP
In the classical setting of multi-task relationship learning (hhag & Yang 2017 Zhang & Yeung 2010), there are m different task learners with their own task-specific data. The aim is to solve:
la (x,,, wx) R(W,O) }
min F(W,S)
where wi is model for task learner k;; {#1...... ......... Tna the local data for the 1tth task; (.) is the empirical loss for task k; W = [w-.... Wm I: and Q E Rm>xm characterizes the relationship between every pair of task learners. A common choice for setting the regularization term R(W, previous works (Zhang& & Yeung 2010 Smith et al., 2017) is:
R(W, . = A1r(WSWWT)
where Ω can be viewed covariance matrix, used learn/encode positive, negative, unrelated task relationships (Zhang & Yeung 2010). In this paper, we focus on studying the mean-regularized multi-task learning objective (Evgeniou Pontil 2004)): special case of (1) where S2 (Amxm 1, 1 1 ) is fixed. Here Imxm is the identity matrix of size x m and 1m E Rm is the vector with all entries equal to 1. By picking \1
2.. we rewrite the objective as:
la (Ta, wk) }]
min F(W)
Wk
17
m
where te is the average of task-specific models: tasks, and each wt is kept locally for task learner k. During optimization, each task learner k solves:
En Wk. Note that w is shared across all
m
mk 401,, wk)
min fa(w:: wT 2 Ilwe w
Despite the prevalence of this simple form of multi-task learning and its recent use in applications such federated learning with strong privacy motivations (e.g.. Hanzely & Richtárik) 2020 Hanzely et al, 2020; Dinh et al. 2220), we are unaware of prior work that has formalized lask- differential privacy in the context of solving Objective 2
3

---

Under review as a conference paper at ICLR 2022
3.2 PRIVACY FORMULATION
We start by introducing the definition of differential privacy (DP) before discussing its generalization to joint differential privacy (JDP). In the context of multi-task learning, each of the m task learners owns a private dataset Dj E U4 C U. We define D = {D1, ... Dm and D' = {Di.... ... D )- We call two sets D, D' neighboring sets if they only differ on the index i.e. D2 Dj for all except i. With this setup in mind, we define differential privacy more formally below.
Definition (Differential Privacy (DP) for MTL (Dwork et al. 2006)). A randomized algorithm
M: 2thh - R is (e. () -differentially private for every pair of neighboring sets that only differ in arbitrary index i: D, D' E Ll and for every set of subsets of outputs S C R.
Pr(M(D) E S) S e' PP(M(DD) E ES) S)++ 6
(4)
In the context of MTL, a learning algorithm outputs one model for every task learner. As mentioned previously, since the output of MTL is a collection of models, traditional DP would require that all the models produced by an MTL learning algorithm are insensitive to changes that happen to the private dataset of any single task.
In this work we are interested in studying task-level privacy, where the purpose is to protect one task learner's data from leakage any other task learners. In this setting, DP incurs an additional restriction that the model of any task learner should also be insensitive to changes in its own data, which would render each of the models useless. To overcome this limitation of DP, we suggest employing joint differential privacy (JDP) (Kearns et al. 2014), relaxed notion of DP, to formalize the guarantee that an MTL algorithm should provide in order to protect task-level privacy. Intuitively, JDP requires that for each task k. the set of output predictive models for all other tasks except l: is insensitive to kts private data. We provide a formal definition below.
Definition 2 (Joint Differential Privacy (JDP) (Kearns et al., (014)). A randomized algorithm M : Um + R. is (e, j-jjinn differentially private if for every i, for every pair of neighboring datasets that only differ in index i: D'D' E Um and for every set of subsets of outputs S C RD,.
Pr(M(D)-<ES)
<P((M(D))> 0
(5)
where M(D) -s represents the vector M(D) with the i-th entry removed.
JDP allows the predictive model for task k to depend on the private data of k2 while still providing a strong guarantee: even if all the clients from all the other tasks collude and share their information, they still will not be able to learn much about the private data in the task k. JDP has mostly been used in applications related to mechanism design (Hsu et al., 2016a] Kannan et al., 2015 | Hssu et al.) (0166) Cummings et al. 2015 Rogers& & Roth 2014 Kearns et al. 2014). Although is natural choice for achieving task-leve. privacy in MTL, we are unaware of any work that studies MTL subject to JDP.
We also note that we can naturally connect joint differential privacy to standard differential privacy. Informally, we take the output of a differentially private process and run some algorithm on top of that locally for each task learner without communicating to the global learner other task learners, this whole process can be shown to be joint differentially private. This is formalized as the Billboard Lemma (Kaans et al. 2014). presented in Lemma [
Lemma 1 (Billboard Lemma). Suppose M D + W is (e, (e, d)differrenliall private. Consider any set of functions: ft : D, x W + W". The composition (f.LL,,,M((D)) is (e, j'joint differentially private, where 11, D + D, is the projection of D onto D;
With the Billboard Lemma, we are able to obtain joint differential privacy by first training a differen- tially private model with data from all tasks, and then finetuning on each task with its local data. We formally introduce our algorithm and corresponding JDP guarantee by using Lemma] iin Section 4
Finally, note that our privacy formulation itself is not limited to the multi task relationship learning framework. For any form of multi-task learning where each task-specific model obtained by training a combination of global component and local component(e.g [Li et al. (202T)), we can provide JDP guarantee for the MTL training process by using a differentially private global component.
4 PMTL: PRIVATE MULTI-TASE LEARNING
We now present PMTL, a method for performing joint differentially-prig -rivaat MTL (Section 4.1). We provide both privacy guarantee (Section 4.2) and utility guarantee (Section 4.3 for our approach.
4

---

Under review as a conference paper at ICLR 2022
Algorithm PMTL: Private Mean- Mean-Regularized MTL
1: Input: m, T, >, 7/, {wi. ........ wn w: m Σx= w
2: for t= 0, .... T do
3: Global Learner randomly selects a set of tasks S, and broadcasts the mean weight W
4: for & E S, in parallel do
5: Each task updates its weight wk for E iterations, og is the last iteration task &s is selected
Wk ++1 ClientUpdate(uq""
6: Each task sends sh
wi (+1 wf. back the global learner.
7: end for
8: Global Learner computes noisy aggregator of the weights wh wi
sh
min
1, 19 6+1 112
+NO0, laxd)
S
KES.
9: end for return
10:
W1
wm as differentially private personalized models
ClientUpdate(w) 12:
for 0, do
Task learner performs SGD locally
W w 7 (V wele(w) + A(w w²))
14: end for
4.1 ALGORITHM
We summarize our solver for private multi-task learning in Algorithm Our method is based off of FedAvg (McMahan et al. 2017). a communication--eficccnn method widely used in federated learning. FedAvg alternates between two steps: (i) each task learner selected at one communication round solves own local objective by running stochastic gradient descent for E iterations and sending the updated model to the global learner; (ii) the global learner aggregates the local updates and broadcasts aggregated mean. By performing local updating this manner, FedAvg has been shown to empirically reduce the total number of communication rounds needed for convergence in federated settings relative to baselines such as mini-batch FedSGD (McMahan et al. 2017). Our private MTL algorithm differs from FedAvg in that: (i) instead of learning single global model, all task Iearners collaboratively learn separate, personalized models for each task; (ii) each task learner solves the local objective with the meaa-regularizztion term; (iii) individual model updates are clipped and random Gaussian noise is added to the aggregated model updates to ensure task-level privacy.
To aggregate updates from each task, assume that we have access to trusted global learner. i.e., is safe for some global entity to observe/collect the individual model updates from each task. This standard assumption in federated learning, where access trusted central server is assumed in order to collect client updates (Kairouz et al. 2019). However, even with this assumption, note that possible for any single task learner infer information about other tasks from the global model, since it is a linear combination of all task specific models and is shared among all task learners.
There are several ways to overcome this privacy risk and thus achieve (c, ddiffferrett privacy. In this paper, we use the Gaussian Mechanism (Dwork & Roth 2014) during global aggregation as simple yet effective method, highlighted in the red portion of line 8 in Algorithm [ In this case, each task learner receives a noisy aggregated global model, making it difficult for any task to leak private information to the others. To apply the Gaussian mechanism, we need to bound the lz-sensitivity of each local model update that is communicated to lie in B { {Awl) [Aw]|Aw]| S <>}, as highlighted in the blue part of line in Algorithm|] [ Hence, at each communication round, the global learner receives the model updates from each task, and clips the model updates to B before aggregation. Note that different from DPSGD (Abadi et 2016). when we solve the local objective for each selected task each communication round, our algorithm doesn't 't clip and perturb the gradient used to update the task-specific model. Instead, since the purpose is to protect task or client-level privacy in multi-task learning, perform standard SGD locally for each task and only clip and perturb the model update that is sent to the global learner. We formalize the privacy guarantec of Algorithm i Section 4.2
5

---

Under review as a conference paper at ICLR 2022
4.2 PRIVACY ANALYSIS
We now rigorously explore the privacy guarantee provided by Algorithm 1 In our optimization scheme, for each task k at the end of each communication round, a shared global model received. After that the task specific model is updated by optimizing the local objective. We formalize this local task learning process as hy: : DE W + W. Here we simply assume W mechanism for communication round to be
C Rd closed. Define the
((D.)), (l((+)) w. .) T²
+
Σ
Ehx(Da (Dx, w²) + 8
(6)
|Sa
RES.
where 8 ~N,00, laxa). Note that M¹ characterizes a Sampled Gaussian Mechanism given of as a fixed model rather than the output of a composition of M for j < t. To analyze the privacy guarantee of Algorithm over T communication rounds, we define the composition of M' to M recursively as M = =M (1D.), (M,,),,P--1
Theorem 1. Assume |Sa] q for all t and the total number of communication rounds is T. There exists constants C12 C2 such that for any < C1 I. I, the mechanism M is (e, diffferrttn
TV TV
private for any > we choose a 2 02
When q = m, M 1:T is (c, 8) ( -)dfffrrretltlllly em
private we choose 4o em
Theorem I provides a provable privacy guarantee on the learned global model. When all tasks participate in every communication round, i.e. q m, the global aggregation step in Algorithm reduce to applying Gaussian Mechanism without sampling rather than Sampled Gaussian Mechanism on the average model updates. We provide detailed proof of Theorem I in Appendix [A.]
Note that Theorem I doesn' it rely on how task learners optimize their local objective. Hence, Theorem iss not limited to Algorithm[1 T and could be generalized to other local objectives and other global aggregation methods that produce a single model aggregate.
Now we show that Algorithm| I which outputs separate models, satisfies joint differential privacy. Given T for any < T, we formally define the process that each task learner & optimize its local objective be M/2 : Dr W W. Note that hi is not restricted to be h2 and could represent the optimization process for any local objective. In order to show that Algorithm satisfies JDP, we would apply the Billboard Lemma introduced in Section 3. In our case, the average model that is broadcast by the global learner at every communication round is the output of a differentially private learning process. Task learners then individually train their task specific models on the respective private data to obtain personalized models. We now present our main theorem of the JDP guarantee provided by Algorithm
Theorem 2. There exists constants c1, for any 0 < E < Ci LT and 8 > O, let 2 227 T (og((1)) Algorithm 7 that outputs W1, (Dx, for each task is (c, o)-joins differentially private.
cm
From Theorem 2 for any fixed d, the more tasks involved in the learning process, the smaller we need order to keep the privacy parameter e the same. In other words, less noise is required to keep the task-specific data private. When we have infinitely many tasks (m -> oo), we have - 0, in which case only a negligible amount of noise is needed to add to the model aggregates to make the global model private to all tasks. We provide detailed proof in Appendix A.
Remark. Note that privacy guarantee provided by our Theorem is not limited to meau-regularized multi-task learning. For any form of multi-task relationship learning with fixed relationship matrix Ω as long as we fix the fz-sensitivity of model updates and the noise scale of the Gaussian mechanism applied to the statisties broadcast all task learners, the privacy guarantee induced by this aggregation step is fixed, regardless of the local objective being optimized. For example, as a natural extension of our mean- anean-regularized MTL objective, consider the case where task learners are partitioned into fixed clusters and optimize the mean-re regularized MTL objective within each cluster, as in Evgeniou et al. (2005). In this scenario, Theorem 2 directly applies the algorithm run on each cluster.
4.3 CONVERGENCE ANALYSIS
As discussed in Section we interested in the following task objective:
(ww) W) = 1g (ww) + >
2 w - 3
(7)

---

Under review as a conference paper at ICLR 2022
where w is estimate for the average model w;; (e(w)) is the empirical loss for task k;; wy R
Here, we analyze the convergence behavior in the setting where a set S of q tasks participate in the optimization process at every communication round. Further, we assume the number of local optimization steps E' 1. We present the following convergence result:
Theorem (Convergence under nonconvex loss). Let f& be (L X)-smooth. Assume y is sufficiently
large such that 2 max,,r VIV ung Su("f; w 1l2 Further let fg mmin,, S& fa(w; 1) and we use a fixed learning rate πι pL+( (p- $)
m If
Algorithm satisfies:
T m
0 O
Ei
B4+1
V Vi x 1|
S
+
p
+ (do?))
(8)
mT
mt
T
1=0 k=1
where
B = max fu w²).
(9)
Let a chosen as we set in Theorem
Take T 0
the right hand side bounded by
Adby
Ady
T-1
m
dy2 m
T-1
B +o log(1/0) (10)
1 m
V w0) )12 S 1=0
+ O (dy²)
Σ t=0
m
22
We provide a formal statement and full proof of Theorem 3 in Appendix A.2
The upper bound in
330 consists of two parts: error induced by the gradient descent algorithm and error induced by the Gaussian Mechanism. When a = 0, Algorithm recovers a non-private mean-ree rean-regulariizd multi-task learning solver.
Corollary 4. When a = 0, Algorithm 7 with (L X)-smooth and nonconvex fa satisfies
O
A
E Be+1
f T7 ) O
(II)
mf
m&
By Theorem 2 given fixed e, 0² grows linearly with respect to T. Hence, given the same privacy guarantee, larger noise is required if the algorithm is run for more communication rounds. Note that in Theorem the upper bound consists of O m ). which means when there bound becomes smaller while the privacy parameter remains the same. On the other hand, Theorem 3 also shows privacy-utility tradeoff using our Algorithm [ the upper bound grows inversely proportional to the privacy parameter e. We also provide a convergence analysis of Algorithm [ with strongly-convex losses in Theorem 5 below (formal statement and proof in Appendix (A.3).
tasks, the upper
Theorem 5 (Convergence under strongly-convex loss). Let fx be (L + A)-smooth and (μ X)- strongly convex. Assume is sufficiently large such that ? Z max,, |I wi (e(we ) Further let w argmin fo (w; w²) where / = En w6 and 2. If we for some constant such that 0 S mplc - 2) + A) S 1, Algorithm satisfies:
fixed learning rate
= 1) I-t C
txp
1 Ar 51 (1- 2))x+)))+++ m
(L c) +C,
(12)
m
fa(ut: wt) c 0
(Vinty 1
where A.
Ele fa fa(uf;
B
max, maxk I' (wts w
Let be chosen as in Theorem then there exists T T=0
(n(oR-20)(++))
such that
Ar < mp(c 2)( + [))² ( log(1/5)
-
B
log(1/6)
+
+ +0
m
me²
me²
(13)

---

Under review as a conference paper at ICLR 2022
As with Corollary 4 recover the bound of the non-privee non-private mean-regularized MTL solver for o=0.
Corollary 6. When a 0, Algorithm
7 Twith (L+ X) X--smothh and (μ+ X) stssongyy convex Su satisfies
1 Ar S (1 -mplm 2) A])" m
1 - mp(c 2) x
B
up(c - 2 (14)
EXPERIMENTS
In this section, we empirically evaluate our private MTL solver on several federated learning bench- marks. Specifically, we demonstrate the privacy-utility trade-off of training MTL objective compared with training global model. We then compare results of performing local finetuning after learning an MTL objective with local finetuning after learning global objective.
5.1 SETUP
For all experiments, we evaluate the test accuracy and privacy parameter of our private MTL solver given fixed clipping bound y, variance of Gaussian noise o², and communication rounds T. All experiments are performed on common federated learning benchmarks as a natural application of multi-task learning. We provide detailed description of datasets and models in Appendix A.4 Each dataset is naturally partitioned among m different clients. Under such scenario, each client can be viewed as task and the data that a client generates is only visible the local task learner.
a=0.1 y=0.5
a=0.11 y=0.2
a=0.02, y=1
Privcce merrmmetertes
12.5
Privacy parametenia)
6.0-
10.0
Losses
5.55
7.5
5.0
5.0
2.5
1.00
200
400
600
200
400
600
800
1000
260
400
600
800
1000
Commurication lon
Communication rounds
Communication rounds
(a) StackOverflow tag prediction
(b) FEMNIST
(c) CelebA
Figure 2: Loss and privacy parameter communication rounds for PMTL. The blue line shows the change of privacy parameter terms of number of communication rounds during training. The orange line shows the average training loss across tasks.
5.2 PRIVACY-UTILITY TRADE-OFF OF PMTL
We first explore the training loss (orange) and privacy parameter e (blue) function of communi- cation rounds across three datasets (Figure 2). Specifically, we evaluate the average loss for all the tasks and given fixed ó after each round, where ò is set to be - for all experiments. In general, for a fixed clipping bound y and a, we see that the method converges fairly quickly with respect to the resulting privacy, but that privacy guarantees may be sacrificed in order to achieve very small losses.
To put these results in context, we also compare the test performance of our private MTL solver with that of training global model. In partieular, use FedAvg (McMahan al. 2017) train a global model. At each communication round, task learners solve their local objective individually. Assuming the global learner is trustworthy, while aggregating the model updates from all tasks, the global learner applies a Gaussian Mechanism and sends the noisy aggregation back to the task learners. As result, private FedAvg differs from our private MTL solver in the following two places: (i) the MTL objective solved locally by each task learner has an additional mean- regularree term; (ii) the MTL method evaluates on one task- specifcc model for every task while the global method evaluates all tasks on one global model. For each dataset, we select privacy parameter E [0.05, 0, 0.2, 0.4, 0.8, 1.6, 2.0, 4.0). For each e, we select the 7, a, and that result in the best validation accuracy for given e and record the test accuracy. A detailed description of hyperparameters is listed in Appendix AA.5 We plot the test accuracy with respect to the highest validation accuracy given one e for both private MTL model and private global model. The results are shown in Figure
In all three datasets, our private MTL solver achieves higher test accuracy compared with training a private global model with FedAvg given the same Moreover, the proposed mean regularized MTL solver is able to retain an advantage over global model even with noisy aggregation. In particular, for small < adding random Gaussian noise during global aggregation amplifies the test accuracy
8

---

Under review as a conference paper at ICLR 2022
0.8
0.9
Test scouracy
0.6
0.2
0.4
0.7
Test
0.1
0.2
Test
0.6
0.0
0.0
2
(a) StackOverflow tag prediction
(b) FEMNIST
(c) CelebA
Figure 3: Comparison of PMTL and training private global model.
difference between MTL solver and FedAvg. Under the StackOverflow task, both methods obtain test accuracy close to the non private baseline for large e. To demonstrate that applying private MTL has an advantage over private global training more generally, we also compared our PMTL method with private FedProx (Li et al. 2020b). The results (which mirror Figure are in Appendix A.6
Table I: Comparison of private MTL and private Global model with different local finetuning methods. é OG corresponds to the case where no noise and clipping happened, i.e., training non-privately, The higher accuracy between MTL and Global given the same and finetuning method is bolded.
FEMNIST
0.1
6 2.0
MIII
Global
MITL
Global
MIL
Global
MTLI
Global
Visilla Finctuning
0.645
0.013
0.000 = 0,017
0,640 1 0.016
0.648 0.017
0.677 1. 0.008
0.653
0.000
0.832 1. 0.0005
0.12
0.00)
Mann-egallaizotion
0.608- 0.011
0.581
= 0.011
0.605
+ 0.008
0.574
0.006
0.666 + 0.009
0.633
0.008
0.825 + 0.001
0.839
0. 6.006
Symmetrized KL
0.886
0.012
0.516
0.006
0.584
0.011
0.681
1 0.0L6
0.682 1: 0.016
0.365
0.010
0.859
1: 0.006
0.500
0.015
EWC
0 0.0833 0.002
0.558
= 0.0011
0.505 0.004
0.807 : 0.007
0.681 1: 0.002
0.660
0.001
0.837 0.0.00
0.533
8: 0.005
5.3 PMTL WITH LOCAL FINETUNING
Finally, in federated learning, previous works have shown local finetuning with different objectives helpful for improving utility while training differentially private global model (Yu et al. 2020). In this section, after obtaining a private global model, we explore locally finetuning the task specific models by optimizing different local objective functions. In particular, which (i) naively optimize the local empirical risk (Vanilla Finetuning). or (ii) encourage minimizing the distance between local and global model under difffrrnt distance metrics (Mean-regullarization Symmetrized KL, EWC (Kirkpatrick et al. 2017: Yu et al. 2020)). The results are listed in Table [] When oc (the non- non-private setting). global with mean- mmean-regularizztioon finetuning outperforms all MTL+fnntuning methods. However, when we add privacy to both methods, private MTL+ finetuunn has an advantage over global with finetuning on different finetuning objectives. In some cases, e.g. using Symmetrized KI. as the finetuning objective, the test accuracy gap between private MTI with fintuning and private global with finetuning is amplified when small compared the case where no privacy is added during training.
use common objectives
CONCLUSION AND FUTURE WORK
In this work, we define notions of task-level privacy for multi-task learning and propose a simple method for differentially private mean-regularized MTL. Theoretically, we provide both privacy and utility guarantees for our approach. Empirically, we show that private mean-regularized MTL retains advantages over training private global on common federated learning benchmarks. In future work, we are interested in building on our results to explore privacy for more general forms of MTL, e.g., the family of objectives in (T) with arbitrary matrix Ω. We are also interested in studying how task-level privacy relates to algorithmic fairness in the MTL setting.

---

Under review as a conference paper at ICLR 2022
Ethics Statement. In this work aim raise awareness about privacy issues that potentially exist when performing multi-task learning. which is commonly used to model heterogencous data in areas such healthcare, finance, and IoT computing. We are particularly motivated by the application of cross-device federated learning, where numerous forms of multi-task learning have been proposed to improve accuracy, but the privacy implications remain unclear. To ensure common notions of task client-level privacy in this setting while retaining the demonstrated accuracy benefits of MTL, we have provided formal privacy definitions for multi-task learning. Building on these definitions, proposed practical algorithm and studied its performance both theoretically and empirically for the special case of mean-regularized multi-tast learning objective. We hope this work will be starting point for future extensions that study the issue of privacy in multi-task learning.
Reproducibility Statement. provide:
To enable reproducibility in our theoretical and empirical results, we
Proofs of Theorem
Appendix A.1
A.2
A.3
Dataset and model details in Appendix A.4
Hyperparameters for tuning in Appendix A.5
Code used for rumning experiments in Supplementary Material.
REFERENCES
Tensorflow federated:
Machine learning on decentralized data. URL http:::::/w...eesoo...oooo
Eederated
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In ACM SIGSAC Conference on Computer and Communications Security, 2016.
Inci M Baytas. Ming Yan, Anil K Jain, and Jiayu Zhou. Asynchronous multi-tase learning. In International Conference on Data Mining (ICDM), 2016.
Sebastian Caldas, Peter Wu, Tian Li, Jakub Konečný, Brendan McMahan, Virginia Smith, Ameet Talwalkar. Leaf: benchmark for federated settings, https:://iee //te/////affff////////ee//eeeee. cmu edu/ arXiv preprint 2018.
Rich Caruana. Multitask learning. Machine Learning, 28:41-75, 1997.
Yong Cheng, Yang Liu, Tianjian Chen, and Qiang Yang. Federated learning for privacy-preseee preserving ai. Commu- nications of the ACM, 63(12). 2020.
Gregory Cohen, Saced Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters, 2017 International Joint Conference on Neural Networks (IJCNN), pp. 2921-222220 -2226, 2017.
Rachel Cummings, Michael J. Kearns, Aaron Roth, and Zhiwei Steven Wu, Privacy and truthful equilibrium selection for aggregative games. In Web and Internet Economics Conference (WINE), 2015.
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint arXiv:2003. 13461, 2020.
Canh T Dinh, Nguyen H Tran, and Tuan Dung Nguyen. Personalized federated learning with moreau envelopes. In Advances Neural Information Processing Systems, 2020.
Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4):21 (002211-1007, 1-407, 2014.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity private data analysis. In Conference on Theory of Cryptography (TCC), 2006.
Theodoros Evgeniou and Massimiliano Pontil. Regularized multi-task learning. In Conference on Knowledge Discovery and Data Mining, 2004.
Theodoros Eygeniou, Charles A Micchelli, Massimiliano Pontil, and John Shawe-Taylor. Learning multiple tasks with kernel methods. Journal of Machine Learning Research, 6(4), 2005.
10

---

Under review as a conference paper at ICLR 2022
Robin C Geyer. Tassilo Klein, and Moin Nabi. Differentially private federated learning: A client level perspective. arXiv preprint arXiv:/712.07557, 2017.
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered
federated learning. In Advances Neural Information Processing Systems, 2020.
Joumana Ghosn and Yoshua Bengio. Multi-task learning for stock selection. In Advances in Neural Information Processing Systems, 1997.
Sunil Kumar Gupta, Santu Rana, and Svetha Venkatesh. Differentially private multi-task learning In Pacific-Asia Workshop on Intelligence and Security Informatics, pp. 101-113. Springer, 2016.
Filip Hanzely and Peter Richtárik. Federated learning of a mixture of global and local models. arXiv preprint arXiv:2002.05516. 2020.
Filip Hanzely, Slavomír Hanzely, Samuel Horváth, and Peter Richtarik. Lower bounds and optimal algorithms for personalized federated learning. Advances in Neural Information Processing Systems, 2020.
Hrayr Harutyumyan, Hrant Khachatrian, David C Kale, Greg Ver Stoeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific data, 601):1-18, 2019.
Justin Hsu, Zhiyi Huang, Aaron Roth, Tim Roughgarden, and Zhiwei Steven Wu. Private matchings and allocations. SIAM Journal of Computing, 45(6). 2016a.
Justin Hsu, Zhiyi Huang, Aaron Roth, and Zhiwei Steven Wu. Jointly private convex programming. In Symposium on Discrete Algorithms, SODA, 2016b.
Peter Kairouz, Н Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. arXiv preprint 1rXiv:1912.0497,, 2019.
Sampath Kannan, Jamie Morgenstem, Aaron Roth, and Zhiwei Steven Wu. Approximately stable, school optimal, and student-truthful many-to-one matchings (via differential privacy). In Symposium on Discrete Algorithmss SODA, 2015.
Michael Kearns, Mallesh M. Pai. Aaron Roth, and Jonathan R. Ullman, Mechanism design in large games: incentives and privacy. In Moni Naor (ed.). Innovations in Theoretical Computer Science, ITCS' 14, Princeton, NJ, USA, January 12-14, 2014. pp. 403-410. ACM, 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Barkkkssa, et al. Overcoming catastrophic
Kieran Milan, John Quan, Tiago Ramalho, Agnieszka forgetting in neural networks. Proceedings of the National Academy of Sciences, 1 114(13):55113322 2017.
Jeffrey Li, Mikhail Khodak, Sebastian Caldas, and Ameet Talwalkar. Differentially private meta-learning. In International Conference on Learning Representations, 2019.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 3703-500 2020a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In Conference on Machine Learning and Systems, 2020b.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pp. 6357-6368 PMLR, 2021.
Sulin Liu, Sinno Jialin Pan, and Qirong Ho. Distributed multi-task relationship leamiin. In International Conference on Knowledge Discovery and Data Mining (ICDM), pp. 937-946, 2017.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In International Conference on Computer Vision, 2015.
Yishay Mansour, Mehryar Mohri, Jac Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications to federated learning. arXiv preprint arXiv:2002.1061 2020.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication- efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pp. 1273-1282. PMLR, 2017.
H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. In International Conference Learning Representations, 2018
11

---

Under review as a conference paper at ICLR 2022
Πya Mironov. Rényi differential privacy. In 2017 IEEE 30th Computer Security Foundations Symposium (CSF). pp. 263-275. IEEE, 2017.
Ryan M. Rogers and Aaron Roth. Asymptotically truthful equilibrium selection in large congestion games. In ACM Conference on Economics and Computation, 2014.
Felix Sattler, Klaus-Robert Müller, and Wojciech Samek. Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints. IEEE Transactions on Neural Networks and Learning Systems, 2020.
Virginia Smith. Chaokai Chiang. Maziar Sanjabi, and Ameet Talwalkar. Federated multi-task learning. In Advances Neural Information Processing Systems, 2017.
Harini Suresh, Jen Gong. and John Guttag. Learning tasks multitask learning: Heterogenous patient populations in the icu. In International Conference on Knowledge Discovery Data Mining, 2018.
Huiwen Wu, Cen Chen, Learning. arXiv e-prints, art. arXiv:201 1071 179, November 2020.
Wang. A Theoretical Perspective on Differentially Private Federated Multi-task
Liyang Xie, Baytas, Kaixiang Lin, and Jiayu Zhou. divacy-preserving distributed multi- learning with asynchronous updates. In International Conference on Knowledge Discovery and Data Mining, 2017.
Yu, Eugene Bagdasaryan, Vitaly Shmatikov. Salvaging federated learning by local adaptation. arXiv preprint arXiv:2002.00 04758, 2020.
Yu Zhang Qiang Yang. A survey on multi-task learning. arXiv preprint arXiv: 14, 2017.
Yu Zhang and Dit-Yar Yeung. A convex formulation for learning task relationships in multi-task learning. In Conference on Uncertainty Artificial Intelligence, 2010.
12


Reviews:
[{"review_id": "_H8jbk7UuJd", "reviewer": "Reviewer_fBjV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThis paper proposes a new idea of using a relaxation of differential privacy in the multitask learning framework.\nIt motivates work in the direction of looking at application specific relaxations of differential privacy, to give more realistic algorithms which perform better than the ones which satisfy the strong differential privacy constraint.\n\nWeaknesses/Questions:\n\n\t\n1. Is $x_i$ the data? Is $l_k(w_k)$ in eq. (7) the population loss or the empirical loss?\n\n2. Some more results like comparison of rates to algorithms that are not only joint differentially private, but differentially private would be useful. Are the improvements only in constant factors or are the improvements in terms of dimension dependent or number of samples dependent terms?\n\n3. Can the authors give some sense of what are the best rates you can under joint differential privacy and how does that compare with the given algorithm?\n\n4. Is there any motivation as to why the number of local optimization steps was chosen to be 1? In more realistic scenarios, each task would take multiple descent steps to personalize better, right?\n\n5. Can you show a short calculation to ensure the constant multiplying the exponentially decaying term is always positive?\n\n6. The last two terms in equation (13) seem fishy, does it mean the error doesn’t decay by increasing the number of iterations?\n\n7. The values of epsilon in figure 2, seem very high for the loss to decay enough.\n\n8. It’s not surprising that private MTL does better than the private global models, by the sheer number of parameters that private MTL allows (it’s ~m times the private global model parameters). A more fair comparison would be to compare against an algorithm that is globally differentially private instead of joint differentially private.\n", "summary_of_the_paper": "In this paper, the authors formalize a notion of task-level differential privacy using joint differential privacy, a concept known in the differentially private mechanism design literature. They formulate the problem of multitask learning under differential privacy under this relaxed definition. The motivation is that there is no need for privacy of the data of a particular task from the task itself, but since you are sharing information in multitask learning, you want to protect against other tasks learning about your private information. They provide an algorithm with convergence guarantees to do multitask learning in this framework and perform some experiments to test this algorithm in the application of federated learning", "summary_of_the_review": "The idea to use joint differential privacy instead of differential privacy in MTL is interesting. The privacy analysis and the convergence rates presented are fairly standard. The theory doesn’t seem to stand on it’s own and more experiments can be done to rigorously test the introduction of this idea. Hence, I believe in it’s current form the paper is not ready for publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PIwwUkaZdyL", "reviewer": "Reviewer_C2cr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "They give a task-level private algorithm for multi-task learning, which is very important in machine learning, and show the theoretical convergence analysis. The structure is clear and writing is good. The experiment results can also support their theory.\nTypo. Section 5.3 “common” rather than “commons”\n", "summary_of_the_paper": "Many problems in machine learning rely on multi-task learning (MTL), in which the goal is to solve multiple related machine learning tasks simultaneously. In this work, authors formalize notions of task-level privacy for MTL via joint differential privacy (JDP). They propose an algorithm for mean-regularized MTL, an objective commonly used for applications in personalized federated learning, subject to JDP. Then analyze objective and solver, providing certifiable guarantees on both privacy and utility. The method allows for improved privacy/utility trade-offs relative to global baselines across common federated learning benchmarks. ", "summary_of_the_review": "The work is complement, although only one algorithm for one problem, the analysis and experiments are sufficient.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "NogzuELiSwI", "reviewer": "Reviewer_T1p3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I find Sections 1-3 to be strong: the paper is well-written and the problem is well-formulated and nicely described. My main problem is with the convergence results, which are difficult to interpret due to too many parameters and no informal bound is provided. Additionally, it is not clear that the bounds are even non-trivial (see below for more discussion). My other main point of confusion is that the authors say they consider JDP as a relaxation of TLDP to provide better utility, but then their algorithm actually appears to provide the stronger (by Billboard Lemma) notion of TLDP (due to Theorem 1); so what is the role of JDP in their results? \n\nDetailed Review: \np.1: \n\nis task-level DP essentially the same as user-level DP/client-level DP (which is considered in McMahan et al 2018)? If so, state that--these terms are more familiar to the DP community than task-level DP. \n\nemphasize \"as long as client k keeps their data private\" \n\np.3 \n\nMotivate Eq (1). Goal is to train a model that has small average loss over all tasks, right? So why use a regularizer? \n\nDefine $\\ell_k$ and $x_i$ \n\nMissing parentheses in Eq (2)\n\np. 4\n\nclarify that strong guarantee of JDP hinges on each client keeping its own model private \n\nI believe Kearns et al 2014 is wrong citation for Lemma 1. Shouldn't it be Hsu et al? \n\nLemma 1: $D_k$ ->  $D_i$\n\np.6\n\nTheorem 1: missing restriction on $\\epsilon$. Proof of Abadi et al (2016) Theorem 1 shows that some restriction (roughly $\\epsilon \\leq \\ln(1/\\delta)$) is needed; also, you should refer to what you provide as TLDP or user-level/client-level DP (in Theorem 1 and in Definition 1)--DP is too vague, as there are many notions of DP \n\n**Why study JDP if your algorithm satisfies the stronger notion of TLDP?**\n\np.7\n\nIf you assume $\\gamma \\geq \\| \\nabla f_k(w) \\|$ (i.e. loss is effectively Lipschitz), then it seems you do not need to clip the gradients (and clipping will never actually occur in the algorithm)--am I right? \n\nWhy assume loss is $L + \\lambda$-smooth and not $L$ smooth? \n\nShould replace min by inf unless you are assuming compactness of domain; are you? I don't the domain was formally defined. I don't remember seeing $d$ defined either. \n\n**Convergence bounds: if $f_k \\leq 1$ so $B_t = 1$, then the second term in Eq (11) becomes essentially trivial $O(L + \\lambda + ...)$** And that is the non-private case, so the DP bounds can only get \"worse\". \n\nAlso, **the bounds in (8)-(10) and (12) are too complicated: should provide simplified version with dependence on key parameters m, d, epsilon, delta displayed**; and even the formal version should not be simplified to include only the key parameters as well as the smoothness and boundedness parameters, and non-dominant terms should be omitted. \n\n\n ", "summary_of_the_paper": "The paper considers the problem of multi-task learning  (MTL) with task-level differential privacy (TLDP) constraints. The authors propose using joint differential privacy (JDP) for MTL and provide a JDP MTL algorithm, which is a DP variation of FedAvg. They prove the privacy of their algorithm and provide convergence results for smooth nonconvex and strongly convex losses. They then provide numerical experiments. ", "summary_of_the_review": "The paper considers/formulates an interesting problem and is generally well-written. Unfortunately, the convergence results do not appear to be strong (or clear). Additionally, there is confusion about the notion of privacy that their algorithm provides and the role that JDP is playing in the paper. For these reasons I cannot recommend acceptance for the current form of the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Many problems in machine learning rely on multi-task learning (MTL), in which the goal is to solve multiple related machine learning tasks simultaneously. In this work, authors formalize notions of task-level privacy for MTL via joint differential privacy (JDP). They propose an algorithm for mean-regularized MTL, an objective commonly used for applications in personalized federated learning, subject to JDP. Then analyze objective and solver, providing certifiable guarantees on both privacy and utility. The main results, namely the convergence rate results, are hard to parse and hard to interpret. For example, as one reviewer pointed out, it is bounded below by a constant which is not properly explained. Further, comparisons to the literature in user-level privacy (which is equivalent as the task-level privacy) is not provided enough. Significant improvement in the presentation of the main results, along with an interpretable explanation of the contribution, is necessary for this manuscript.
Decision: Reject