Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper tackles with the hyper-parameter tuning problem in logistic regression on encrypted data. Assuming a fraction of public data and a fraction of public features, the authors show by experiments that similar or better accuracy can be achieved with ~3x speed.
main_review: I find the classification-to-regression idea and the computation trick quite interesting. The speed-up is significant and evident. I think this part is solid work and makes a good publication. However, I have to say that as a non-expert, I cannot judge the novelty of the idea and the trick. In particular, I feel that this classification-to-regression idea may generally be useful in domain adaptation / transfer learning. I cannot tell if it already appeared in that community. Secondly, I suppose the linear algebraic trick used in lines 150-176 was at least partially known. For example, I suppose Equation (4) should be well-known, but is it so well-known that no reference is needed/possible? Some notes would definitely help readers like me.

On the other hand, I have one major criticism: The theory in section 4 doesn't seem to make any contribution to the paper. It also doesn't make much sense to me. I recommend removing this part, together with the claim that the authors find a new bound. Otherwise, section 4 needs a significant improvement at the very least. For example, in Theorem 2, I don't know what is $\\tilde\{H\}\_0$. The definition in line 214 doesn't make much sense to me. According to line 209, $R_{D_T}$ relies on a ground truth $c_T$. What is it? Why are we interested in the divergence defined in line 214? How does $H$ and $\tilde{H}$ effect the divergence? Is it an original definition or is there a source? I couldn't find it in the book cited. In the proof of Theorem 2, the authors cited Theorem 13.2 of the book, but that deals with multi-class case. Isn't 13.4 a better fit for the binary problem in the paper? I'd like to see the response on this issue and am open to change my rating.

Finally, a comment on the presentation as a non-expert reader: It takes a bit effort for me to figure out who does what and who sees what and why the protocol is secure. Stating these facts with explicit reference to Figure 2 and/or Algorithm 1 may help.


limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative societal impact of their work
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces an interesting new approach for learning a classifier on homomorphically encrypted data. A few important assumptions are made: only very few features (one to three maybe) are privacy-protected and the modeler is able to build a regression model on a part of the dataset. Next, further training data including private features are used to convert the initial model into a Ridge regression model. The key innovation is that the encrypted computation step can be isolated to make it very simple when few features are encrypted.  
main_review: I think the idea is very interesting and hits many of the pain-points in learning on encrypted data. Realistically not all features are equally private, so it's great to see that this can be taken advantage of so well. Another huge advantage is getting rid of the sigmoid approximation, which is always problematic due to inability to control the input range.

This work seems totally novel and interesting to me, but I'm not enough of an expert to evaluate whether (from an ML point-of-view) the approach that is taken is actually sound. The encrypted computing part is rather simple and could be done with a number of techniques to achieve semi-honest security.
limitations_and_societal_impact: Training on encrypted data can raise all kinds of new concerns. For example, poisoning becomes really challenging to prevent, as input data can't be validated by the modeler. This may or may not matter, depending on the exact use-case. Maybe some paragraph discussing concrete scenarios, where the proposed system makes sense, could be added.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This study considers a situation that a part of unencrypted samples are available while the other samples are provided with encrypted by FHE (or LHE). First, the modeler trains a logistic regression model with unencrypted samples. By using this model, given encrypted samples,  the modeler obtains the encrypted logits. To absorb the difference of sample generating distributions of non-private samples and private samples, moment matching is applied on encrypted logits (I could not understand why this works).  Then, ridge regression is solved with the encrypted logits with labels, in that the solution can be obtained in an analytic form instead of iterative methods as logistic regression regularly requires.
**Due to the lack of a detailed description of the security model and global information flow of the protocol, this summary might be wrong. 

main_review: In my understanding, this paper does not contain a technically original contribution, while a practical solution for logistic regression model training over FHE is presented by combining known techniques in matrix algebra and cryptography. In this sense, this paper presented a meaningful construction of logistic regression over FHE.

The weak point of this manuscript is that the contents of the paper overly emphasize the detailed mathematical operations while necessary information to understand the security model considered in this study is lost. Due to the lack of this, the reviewer could not understand the information flow between data owner, modeler, and crypto-service provider. Also, the reviewer could not understand who wanted to protect what from whom by this protocol. For this reason, the reviewer thinks it is not appropriate this publish this manuscript in its current form.

Line 130 
...we should assume that M should possess unencrypted D1 that can be used for step 1

This paper does not give a specific security setting (who does what and what is private). Without a clear description of the security model, it is impossible to see if the presented scheme is secure or not.


Also, line 120
... The details of our protocol to achieve the security goal is in Appendix.

The protocol itself is the main contribution of this work so it should be contained in the main body.

Line142
... we suggest adding a regularization term β on l2 so that it can take into account the difference between two distributions

This does not make sense. Why regularization term can resolve the difference between two distributions? 

Line 148
...By encrypting a small private portion of the whole information, 

Does this mean that some portion (samples? variables?) is private and the other (samples? variables?) are not private?  Who decides which is private and which is not?

limitations_and_societal_impact: No.
I think this type of study is originally presented to enhance privacy, no need to support it.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper tackles the problem of performing logistic regression in a homomorphic encryption setting. The propose the following:

- Pretrain a regular logistic regression on some public data (completely plaintext)

- Run this network on the private data to produce soft targets (and shift the means of the soft targets to deal with datashift shift, with a hyper-parameter)

- Solve ridge regression (possibly with some non-private variables) to estimate these targets on the private data

The authors provide a fast solution for step 3 and show that their model can outperform LRHE in accuracy and speed on different datasets from the UCI repository. 

main_review: Originality:
The authors propose to tackle the problem of hyper-parameter selection in homomorphic encryption by leveraging a plaintext-text dataset at the start of the training process. This offers an interesting novel direction to explore. 

Quality / Significance:

The core claims of this paper are:

*Introducing a hyper-parameter free version of HE logistic regression

However, the paper introduces a hyper-parameter lamba for their ridge regression which still needs to be selected somehow. Perhaps I’m misunderstanding something but there seems to be a hyper-parameter here. 

*This method is faster and more accurate than prior work

I have a few questions here. Is the LRHE baseline also adapted to only use a few private variables? It seems that the number of private variables here is critical to runtime and so a head to head comparison where all variables are private is critical.  Moreover, how does LRHE use the plaintext data? Does it not use it all? If so, that isn't a fair comparison to the model here. 

Also, how does it compare to other mechanisms for using the plaintext data? For example, can the pretrained teacher be used as an initial guess of weights for both LRHE and this approach? Moreover, how does it perform by itself without any training on the private data (in table 1)?

As of now, it is difficult to assess the significance of these results. 

limitations_and_societal_impact: The paper notes that this framework cannot be used for nonlinear models easily (like NNs), however, it can be used with kernel methods.  The study does not note possible negative social impacts.
review_title: null

Meta-review: This paper presents an interesting solution to the problem of training models on private data. The progress is made by first noticing that not all features are equally sensitive and then using it to reduce the classification problem, which is hard to train using HE to a regression problem that is easier to solve.
Most reviewers agreed that the paper makes a nice contribution to the field and should be accepted. The presentation of the work is fine but could be improved. Adding to comments made by the reviewers note, for example, that on line 122 “knowledge distillation” is discussed as if it was already presented before. The line reads “…which mimics the first phase of the knowledge distillation in that …” but this is the first-time knowledge distillation is being discussed. In this case, removing the “the” before “knowledge distillation” and adding a reference to a relevant paper could help the reader. We encourage the authors to revisit the paper and try to improve it readability.

Decision: Accept (Poster)