Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes an interesting idea where they seek to distinguish between activation channels that are crucial for preserving information flow and approximating the rest with low degree polynomials. The authors show that this doesn't result in decrease in accuracy but leads to speedup in performance.

I think it is a neat trick to exploit the high-dimensional intermediate feature space given that the network only uses a low dimensional subspace of it. It is an interesting use of the idea which has been demonstrated in prior works - low dimensionality of the feature space or that the learning task can be done by randomly switching of a huge fraction of available channels.

However, I think the paper has two main issues which prevents me from providing a more favourable decision.


* Novelty and Comparison
   * The key novelty of this paper is to figure out which channels can be approximated with polynomials and which needs to be retained in its original capacity. This approximation leads to a speedup. While it is a neat modification, I think it is very incremental in its novelty and the performance is not very large either.  The time for one ResNet32 inference on CIFAR100 with SafeNet is 0.62x of Delphi, which is not a very large improvement.
   * It would also be helpful to compare with techniques other than gazelle and Delphi that modifies the neural networks prior to training (eg. binarizing the network, ternarizing the network, fixed point precision etc) so that the accuracy-latency of the trade-off can be put into perspective and compared easily.  When considering methods to speedup Encrypted Prediction as a Service, one should naturally talk about fixed precision networks and binary neural networks (BNNs). In the past, BNNs have shown remarkable speedups in computation without a major hit to accuracy (Sanyal et. al. 2018), which is in fact a major point of this paper. Similarly, the paper should also discuss and compare with techniques that uses modification of encryption schemes specifically meant to optimize activations in NNs (eg. Lou et. al. 2019)

There has been a large body of work in the past few works that claim to improve latency of encrypted prediction. It is getting hard to say whether a technique really provides a speedup unless a more comprehensive experimental survey is done with a series of papers using benchmark datasets and networks. i would encourage the authors to have more techniques in their experimental comparison section.

* Clarity -  I have found the paper very hard to read in general and some major gaps when considering baselines and background discussion.
    * I think the paper would be much easier to read if the citations were modified with something akin to \citep and \citet. 
    *  The figures (Fig 1, Fig 3, Fig 4) are very far from being self-contained. I think it is okay to omit some details from the figures But there should be proper explanations in the captions. It is very hard to understand to understand what the notations mean in Fig 1A and what 0L, 4L, and the percentages mean. Similarly Fig 4a, is very congested with overlaps between arrows and figures and it is very hard to read and understand what the different notations mean.

I think, overall, the paper needs an overhaul to first make sure that the various components used in the pipeline are explained properly, make sure the figures are clear, the notations are clear especially in the earlier sections. I think further advantages can be gaining incites from papers that talk about importance of various layers (Zhang et. al. 2019), using low rank layers or representations etc and doing comparisons with other works that claim to provide an improvement in encrypted prediction latency.

Zhang, C., Bengio, S., & Singer, Y. (2019). Are all layers created equal?. arXiv preprint arXiv:1902.01996.

Lou, Qian, and Lei Jiang. "SHE: A Fast and Accurate Deep Neural Network for Encrypted Data." Advances in Neural Information Processing Systems. 2019.

Sanyal, A., Kusner, M. J., Gascon, A., & Kanade, V. (2018). Tapas: Tricks to accelerate (encrypted) prediction as a service.  International Conference in Machine Learning, 2018.
review_title: Reviews of Reviewer 1

final_score: rating: 7: Good paper, accept
review_content: Summary:

The main contribution of this paper is a new heuristic for identifying "less useful" activation channels. The authors then propose using simple approximations for activation functions for these channels without compromising network accuracy. The main novelty in the approximation used by the authors is flexibility in the degree of the polynomial approximation. Additionally, the authors propose a new hyper-parameter search strategy (BTPBT) to efficiently search the hyper-parameter space for the optimal approximation parameters.

Score Rationale:
- The paper has three novel contributions:
  - Channel-wise approximation of activations
  - More flexibility in the form of the polynomial approximation
  - BTPBT hyper-parameter search
- As a result of these contributions the authors demonstrate a 40% reduction in total inference latency compared to state of art

Strengths:
- The authors present a thorough description of the experimental procedure that is sufficient to replicate their work
- An informative ablation study is presented to summarize the relative contribution of the various techniques proposed
- The baselines chosen are competitive and allow for a fair evaluation

Weaknesses:
- The main weakness of this work stems from the modest improvement in latency afforded by these techniques
- In particular while secure inference is still orders of magnitude less efficient compared to the corresponding plaintext computation these techniques can only account for a factor of 2 compared with a baseline that implements no secure inference specific network tuning (e.g. Gazelle)

Additional Comments/Questions:
- Is the accuracy of the three networks from Fig 4b comparable?
- Fig. 5 caption has a typo. The (a) and (b) sub-captions should indicate the network name
- What is the source of the ~10% improvement in the runtime of the linear layers when compared with Delphi?
review_title: Interesting technique for model optimization targeting secure neural network inference 

final_score: rating: 7: Good paper, accept
review_content: Summary:
The paper present a system for two-party deep learning inference. The main contribution is activation layers that are more expensive in two-party computation are replaced by approximations dynamically based on the training data. To this end, the authors use a divide-and-conquer approach to gauge the impact of replacing activation functions of some layers by a version more amenable to secure computation. Furthermore, the algorithm also considers various degrees for approximation (0, 2, and 3). Experiments show that this reduces the latency by up to two thirds while maintaining a similar accuracy.

Pros:
- The dynamic approach is more sophisticated.
- The improvement in efficiency is clear.

Cons:
- Deriving public meta-parameters from data might impact privacy, but no consideration is given to this aspect.
- The further exploration of when to replace ReLU with an approximation seems somewhat expectable despite the novelty of the exact approach.

Minor issues:
- 1: "huge" (unscholarly language)
- throughout: "Delphi Mishra et al." - maybe "Delphi by Mishra et al."?
- References: curly brackets in several references such as {USENIX}

Conclusion:
I recommend acceptance given the novelty.

review_title: Further exploration of a known space in two-party secure inference

final_score: rating: 5: Marginally below acceptance threshold
review_content: 


This paper proposed a new method called SAFENet.  It is able to support a secure, accurate and fast neural network inference service. The best result presented in Table 2, is 3x faster than previous. The best result presented in Table 2, is 2x faster than previous. 

Algorithm 1 is not very well written, it is hard to follow what is that algorithm trying to say.

In page 2, the 2nd line in Section 2.1, the sentence ``current state-of-the-art cryptographic inference, Delphi'' It should have a citation here. The section 2 is related work, but Section 2.1 doesn't cite any previous papers, this is a bit strange.

In the introduction, this paper discusses about cryptography but many important references are missing. For example [Yao] and [Gentry]

Andrew Yao. Protocols for secure computations. FOCS 1982.

Criag Gentry. Fully homomorphic encryption using ideal lattices. STOC 2009.


In related work, the following recent work should be discussed, since they had proposed a practical way to encrypt the image and text data. 

Yangsibo Huang, Zhao Song, Kai Li, Sanjeev Arora. InstaHide: instance-hiding schemes for private distributed learning. ICML 2020
Yangsibo Huang, Zhao Song, Danqi Chen, Kai Li, Sanjeev Arora. TextHide: Tackling Data Privacy in Language Understanding Tasks. EMNLP 2020

review_title: An ok submission, but writing quality requires more improvement

Meta-review: The authors did a nice job of responding to the concerns of reviewers during the discussion phase which increased reviewer scores. Because of this I will vote to accept. 

The authors should carefully edit the paper for typos, grammatical errors, and style errors. Some examples:
- Abstract: Make this one paragraph without a line break
- End of 1st paragraph in Intro: "So there is an urge" -> "So there is an urgent"
- Start of 3rd paragraph in Intro: "State-of-the-art cryptographic" -> "The state-of-the-art cryptographic"
- Last paragraph of 2.1: "To solve above" -> "To solve the above"
- End of 2.3: "Compared to the light-weight InstaHide and TextHide, MPC and HE are of advantages in the security guarantees so far." -> "Compared to the light-weight methods InstaHide and TextHide, MPC and HE provide much stronger security guarantees."

I also urge the authors to please double check the reviewer comments when preparing a newer version to ensure all concerns are taken into account.
Decision: Accept (Poster)