Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a task-level differentially private meta-learning algorithm based on the meta-regularization method. The authors provide the privacy and utility guarantees based on convex and Lipschitz loss assumptions. They also present a clustering-based method to deal with the case where the tasks are sampled from multi-modal distributions. They evaluate the performance of the algorithms in the case of linear regression tasks with mean square loss.
strengths_and_weaknesses: 1.	This paper studies the problem of meta-learning with task-level differential privacy. However, the motivation of this paper is not clear. The authors claim that the record-level is not sufficient in many scenarios (line 9, 53) and they present an “email spam filter” example (line 53-56) to support this point. However, they do not clearly explain what kind of “sensitive information” (line 56) can be protected by task-level DP instead of record-level DP, which makes the example unconvincing and the paper not well-motivated.

2.	The proposed approach is conceptually simple. They apply noisy SGD [14] to meta-level updates of the meta-regularization method [3,4] to get their task-level private meta-learning algorithm. The analysis of privacy and utility guarantees also follows [14], which is quite standard.

3.	The assumptions in Assumption 1 are too strong for meta-learning scenarios. Meta-learning methods are typically applied in reinforcement learning and deep learning scenarios [1-6], which generally do not satisfy Assumption 1. As the privacy guarantee analysis of the proposed algorithms relies on assumption 1, the proposed algorithms are not practical in meta-learning scenarios.

4.	The experiments demonstrate the effectiveness of the proposed methods. However, the experiments are also very toyish (only in the case of linear regression task with MSE loss on toy dataset), which reduces the practical values. This could be also because of the overly strong assumptions (as mentioned in the third review).

questions: none
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Meta learning has been applied successfully in the Machine learning area recently to generalize new task with new small datasets efficiently including meta-initialization and meta-regularization methods. Meanwhile, meta learning may suffer from leakage of sensitive information and privacy part of meta learning has not been well studied. Although a work with meta-initialization method designs a record-level differential privacy approach, it is still too weak to be against aggregate statistics and has potential leakage as well as unrealistic (strict) assumption. To fill the gaps, this work proposes a meta-regularization-based task-level DP algorithm and guarantee transfer risk as well as accuracy with less strict assumption.
strengths_and_weaknesses: Strength:
1. Assumption of this work is less strict as existing works (only assume convex and Lipschitz loss).
2. Existing DP optimization (Noisy SGD) can be applied to this algorithm as almost “free lunch”.
3. Design a novel task-level DP clustering-based meta learning algorithm for multiple meta models as well.

Weakness:
1. Like existing work based on meta-initialization only, this work is based on meta-regularization only. Since this work has mentioned those two methods could be connected by framework of online convex optimization, it would be nice to consider meta-initialization as well under the mentioned framework.
questions: 1. Since the manuscript mentioned meta learning in the distributed setting, did you consider secure multiparty computation as another feasible approach to tackle privacy issue during meta learning?
limitations: No potential negative societal impact found.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: In this work, the authors propose meta learning algorithms with task-level differential privacy to protect the entire dataset for each task. The authors also conduct experiments to empirically evaluate the effectiveness of our proposed algorithms.
strengths_and_weaknesses: Pros

[+] The authors propose a task-level differentially private meta learning algorithm based on the meta-regularization approach.

[+] The authors propose a private clustering based meta learning algorithm to enable private meta learning of multiple meta models.



Cons

[-] The contribution of this paper seems limited. For example, in this paper, the authors directly apply existing differential privacy (DP) optimization techniques like Noisy SGD (e.g., [1]) on the newly defined "data point" and loss function achieves task-level differential privacy naturally. In addition, the proposed meta-clustering algorithm is directly derived from the HYP CLUSTER algorithm in [2].

[-] The authors make strong assumptions, e.g., the assumption of the strongly convexity of the base learner $R_{Z,\lambda}$.

[-] The experiment evaluation is limited. Firstly, there are some existing DP meta learning algorithms and task-level DP algorithms, and the authors fail to compare the proposed algorithms with those baselines. In addition, the influence of different task distributions is unexplored in the experiments. Lastly, the size of the adopted dataset is small.

[1] Bassily, Raef, Vitaly Feldman, Kunal Talwar, and Abhradeep Guha Thakurta. "Private stochastic convex optimization with optimal rates." Advances in neural information processing systems 32 (2019).

[2] Mansour, Yishay, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. "Three approaches for personalization with applications to federated learning." arXiv preprint arXiv:2002.10619 (2020).

questions: [1] Can the proposed algorithm be applied to other losses, e.g., the general convex algorithms?

[2] The influence of the task difficulty and the meta models’ weights on the task similarity is not studied. Theoretical and empirical analysis should be given.

[3] It is not clear about the derivation of the set of optimal parameters for the tasks in Eqn. (8). The definition of optimality is undefined in the paper. In addition, the difference comparison between $V_{c}$ and $V$ is not given.

[4] It would be better if the authors conduct experiments on the large-scale real-world datasets to evaluate the performance of the proposed methods.

[5] More discussions on the personalized privacy-preserving requirements of different tasks should be given.
limitations: Yes
review_title: null

Meta-review: metareview: The paper resulted in lukewarm support for the paper, although they were positive. There were some concerns regarding the practicality of the algorithms mentioned in the paper. Also, I took a quick look at the paper. The setup seems similar to model personalization (https://papers.nips.cc/paper/2018/hash/aa97d584861474f4097cf13ccb5325da-Abstract.html), where folks have analyzed non-convex models with DP. The paper misses citing this line of work. I will request the authors to include the discussions from the rebuttal phase, and also add a discussion to compare with the work along the lines of DP personalization (mentioned above).
confidence: Less certain
Decision: Accept