Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper provides task-aware privacy preservation to improve the task performance for multi-dimensional user data for deploying a trained model. The paper’s spirit is to use an encoder-decoder framework with a Laplace noise added to learn a task-relevant but LDP privacy preserved latent representation of user data. The author also provides an analytical near-optimal solution for a linear setting with mean-squared error task loss. They show the effectiveness of their method via experiments on three real-world datasets.
main_review:  Strengths
+ The problem formulation is simple and easy to follow to get the idea of the proposed method. 
+ They provide an intensive and rigorous analysis for an analytical near-optimal solution for a linear setting and MSE task loss in terms of task-aware privacy preservation analysis.

Weaknesses
-	First, the general idea of the paper is interesting, but it seems to be incremental and at an early stage of this work. The novelty is limited. The idea of redistributing the noise across input features has been extensively studied, for instance, with autoencoder-decoder and layer-wise relevant propagation, and forward derivatives (Phan et al., 2017; 2019). These references should be considered as baselines for comparison to highlight the novelty of the proposed approach. However, these critical references are missing. 
-	Second, the reviewer does not understand why we need LDP here and why not centralized DP? What is the threat model? Training the autoencoder-decoder would need a centralized server to gather all the data tuples together. If so, there is no need for LDP.
-	Third, how practical is Assumption 1? It is not clear at all in the current writing. Also, it is unclear to the reviewer that why Propositions 1-4 result in an optimal encoder and decoder design that preserves $\epsilon$-LDP.
-	In the analysis, a linear model with an MSE task loss is used in the analysis and experiments, which may not be easy to extend to other complex tasks, such as deep learning. 
-	Experimental results can be improved. The datasets used in the experiments still have a limited number of dimensions, such as dim=24 in power consumption, dim=6 in real estate valuation, and dim=30 in breast cancer detection. It would be interesting if the work can be done with the more complex datasets and learning tasks, such as image datasets with thousands of dimensions and deep learning tasks for example. Adaptive mechanisms such as (Phan et al., 2017; 2019; 2020) and other related works listed below can be considered as baseline approaches for comparison. 
-	The compared benchmark approaches are unclear. There is no place to define or refer to the compared method. The reviewer needs to guess what it means. Also, there is a limited number of comparisons, which makes the paper unconvincing. 

NhatHai Phan, Xintao Wu, Han Hu, Dejing Dou. Adaptive Laplace Mechanism: Differential Privacy Preservation in Deep Learning. IEEE ICDM'17.

NhatHai Phan, Minh Vu, Yang Liu, Ruoming Jin, Xintao Wu, Dejing Dou, and My T. Thai. Heterogeneous Gaussian Mechanism: Preserving Differential Privacy in Deep Learning with Provable Robustness. IJCAI'19.
summary_of_the_review: The general idea of the paper is interesting, but it seems to be an early stage of this work. Investigating the work with more complex datasets and tasks and more baseline comparisons would make the paper more convincing. There are a lot baseline of LDP or task-aware work to compare with the work, for example
[1] Wang, N., Xiao, X., Yang, Y., Zhao, J., Hui, S. C., Shin, H., ... & Yu, G. (2019, April). Collecting and analyzing multidimensional data with local differential privacy. In 2019 IEEE 35th International Conference on Data Engineering (ICDE) (pp. 638-649). IEEE.
[2] Liu, R., Cao, Y., Yoshikawa, M., & Chen, H. (2020, September). Fedsel: Federated sgd under local differential privacy with top-k dimension selection. In International Conference on Database Systems for Advanced Applications (pp. 485-501). Springer, Cham.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper addresses the problem of large loss in utility due to the addition of noise in a local-DP setup, especially when working with high dimensional data. To mitigate this problem, the paper proposes an encoder-decoder setup in which noise is applied through the encoder, in a lower-dimension to decrease its effect on utility. Then  Laplace random noise is applied to each point's encoding in the latent space. The training  of the autoencoder is done over an offline phase on public data and with respect to the main task that the data is to be used for.  The approach is then evaluated empirically.
main_review: The curse of dimensionality and the problems it creates for utility of DP mechanisms is an important problem that has been studied for a long time. The approach this paper take's creates the following concerns:

1. The main idea of this  paper  which is dimensionality reduction, has already been proposed before, in numerous contexts such as random projections [1] or sparse regression [2]. The idea of mixing this with task-awareness and considering how a model would use features and adding noise only to features that are used has also been studied before [3]. It would be great if the paper quantifies its differences with these prior work/missing citations better.

2. How much overhead does this add to the user's side, in terms of latency and computation complexity? It will help to know how much extra latency gets added compared to doing normal local DP. The user seems to need to be able to carry out the encoder operation. 

3. How much of this method's success depend on a correct analysis of what feature is related to the  task? How easy or hard is it to get such details? Some more clarity in explaining this would certainly help readability. It seems like the effectiveness of this method relates to how well related features can be extracted, and what this model is learning and if it really relates to how this data would be used in the future for other training tasks. This could even circle back to interpretability and how different models might use different features. 

4. Although the paper claims effectiveness in high-dimensonal cases, the experiments  create the question "how well would this method scale in the presence of real-world, high dimensional data?", like medical images which are often large in size. The datasets used here are very small compared to real-life scenarios.  

[1] Kenthapadi K, Korolova A, Mironov I, Mishra N. Privacy via the Johnson-Lindenstrauss Transform. Journal of Privacy and Confidentiality. 2013;5(1):39-71.

[2] Thakurta AG, Smith A. Differentially private feature selection via stability arguments, and the robustness of the lasso. InConference on Learning Theory 2013 Jun 13 (pp. 819-850). PMLR.

[3] Mireshghallah F, Taram M, Jalali A, Elthakeb AT, Tullsen D, Esmaeilzadeh H. A principled approach to learning stochastic representations for privacy in deep neural inference. arXiv preprint arXiv:2003.12154. 2020 Mar.
summary_of_the_review: 
The main reasons for my recommendation are:

1. similarity to prior work, the novelty of this work and it's benefits are not well identified

2. How useful the extracted dimensions would be in high dimensions and for models that might use different set of features is not clear.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a task-aware local DP method to improve the privacy andutility trade-off for multi-dimensional user data. Also, a analytical near-optimal solution for a general linear encoder-decoder model and MSE loss is provided. For neural network cases (i.e., nonlinear encoder-decoder), the authors present a heuristic learning algorithm to get the model parameters. The experiments results demonstrate the effectiveness of the proposed methods. 
main_review: Strengths:
1. The problem of improving utility and privacy trade-off in local differentially private ML is an important practical problem.
2. For linear encoder and decoder, and MSE loss setting, an analytical near-optimal solution is provided. The comparisons among task-aware approach and the benchmark approaches give an intuitive interpretation of why the parameters obtained from task-aware approach are better. 
 

My main concern in this paper is whether the proposed method can preserve local DP. 
1. The local DP setting is not clearly given, which I mean, interactive [1] or non-interactive setting [2]? 
2. Based on my understanding of this paper,  Algorithm 1 needs $N_{epochs}$ to find the encoder, decoder and sensitivity. During this interactive proposes, the algorithm continues to query the sensitive user data samples. I think the composition theorem should be considered. 
3. For achieving private local data representation,  Wang et al. in [3] also proposed a lightweight privacy-preserving mechanism consisting of arbitrary data nullification and DP noise addition. Compared with [3], I find Algorithm 1 follows a quite similar process when learning the model parameters. Since [3] introduces data nullification and adversary training, their method may be more efficient and better than this paper in terms of communication and utility. Please provide a comparison with [3]. 


[1] Joseph, Matthew, Jieming Mao, Seth Neel, and Aaron Roth. "The role of interactivity in local differential privacy." In 2019 IEEE 60th Annual Symposium on Foundations of Computer Science (FOCS), pp. 94-105. IEEE, 2019.
[2] Wang, D., Gaboardi, M., Smith, A., & Xu, J. (2020). Empirical Risk Minimization in the Non-interactive Local Model of Differential Privacy. Journal of machine learning research, 21(200).
[3] Wang, Ji, Jianguo Zhang, Weidong Bao, Xiaomin Zhu, Bokai Cao, and Philip S. Yu. "Not just privacy: Improving performance of private deep learning in mobile cloud." In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2407-2416. 2018.
summary_of_the_review: This paper proposed a good analysis for task-aware local DP.  However, my main concern is the setting of local DP.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a task aware local-DP approach to improve the performance on multi-dimensional data with same level of privacy. This approach is based on encoder-decoder framework that perturbs only the task relevant encoding instead of the raw user data that generally causes less noise addition and improves the task accuracy. Besides, the paper provide a heuristic learning algorithm for more general settings. The proposed approach is compared with task agnostic and privacy agnostic approaches on different datasets and the results show the proposed method outperformes them on overall task loss under different privacy budgets.
main_review: I think this paper addresses an important problem which is improving the privacy-utility trade-off in LDP where there are multiple users' data.
The proposed approaches solves the problem in an efficient way. The motivation is clear, it is well-structured and well-written. In my opinion, there are three things that can improve this paper. i) The paper presents a heuristic learning algorithm, but the analysis of the task-aware privacy preservation problem for approximate LDP would be more interesting. ii) Comparison with the state of the art approaches both in text and experiments could be improved. In the current version, the difference between the previously proposed methods is not so clear. iii) The results on larger datasets and the result in terms of accuracy. 


summary_of_the_review: This paper considers and important problem and brings an efficient solution. It is well-written. The experiments shows the performance improvement of the proposed method. However, the experiements could be extended on larger datasets and they can be compared with SOTA.
review_title: null

Meta-review: This work considers the problem of how to predict on sensitive user points while preserving their privacy. It proposes a fairly straightforward way to create a local randomizer that optimizes loss for a given model subject to preserving LDP. The work also gives theoretical analysis of the randomizer for least squares linear regression. 
The problem formulation is different from the standard LDP framework where privacy of training data points needs to be preserved. The submission does not motivate this setting and I don't see a good motivation for this problem either. More importantly, it does not sufficiently emphasize that the problem is entirely different from prior work. Indeed all reviewers were confused about various aspects of comparison with previous work. Therefore, in my opinion, the submission is not sufficiently well motivated and clearly presented to be accepted.
Decision: Reject