Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper seeks to improve the computational efficiency of convolution layers in FHE. Since homomorphic rotations are the primary computational bottleneck of convolutions in FHE, the paper seeks to reduce the number of rotations. This is achieved in two steps, 1) adopting group convolutions, which reduces out-level rotations, and 2) weight pruning, which reduces inner-level rotations.

The efficiency of the proposed convolution is evaluated on *shallow* CNNs designed for MNIST and CIFAR-10. The proposed approach shows appreciable speed-up over *naive* implementations of convolutions in FHE.
strength_and_weaknesses: Strengths:
- The paper rightly identifies the main bottleneck of *naive* implementations of convolution in FHE, namely rotations. As such, efforts to improve efficiency of convolution is necessary. Leveraging alternative convolutions, such as group convolutions or in the extreme depth-wise convolutions is interesting.
- Adopting pruning for sparsifying the convolution and optimizing the sparsity pattern for reducing number of homomorphic rotations.

Weaknesses:
- The main drawback of the paper is the lack of comparisons to prior work that improve efficiency of convolutions. These include multiplexed convolutions [1], mobile networks explored HEMET [2].
- The main premise of the paper is that rotations in convolutional layers are the main computational bottleneck of networks in FHE. So the paper considers shallow networks only, which are not likely to be practically useful. For instance 85% accuracy on CIFAR-10 is quite poor by the standards of the best plaintext models which achieve ~99% accuracy.
- As networks become deeper, the main accuracy bottleneck is low-degree polynomial approximations of non-linear functions like ReLU and the main computational bottleneck is the bootstrapping operations required for evaluating high-multiplicative depth circuits. So improving efficiency of convolutional layers does not benefit deeper networks since convolution is not the main bottleneck for such networks.

Other Clarification Questions:
- The paper does not mention how pooling operations or strided convolutions are handled. Strided convolutions result in wasted slots. How  does that affect the proposed convolutions?

[1] HEMET: A Homomorphic-Encryption-Friendly Privacy-Preserving Mobile Neural Network Architecture, ICML 2021
[2] Low-Complexity Deep Convolutional Neural Networks on Fully Homomorphic Encryption Using Multiplexed Parallel Convolutions, ICML 2022

clarity,_quality,_novelty_and_reproducibility: - The paper is clear for the most part. The figures are a bit challenging to understand, but becomes clear with the description in the text.

- The quality of the paper is good for the most part, however there are important baselines that are missing. Such methods have not been cited, discussed, or compared against.

- The proposed method is fairly novel. Most existing CNN implementations in FHE using standard dense convolutional layers. This paper proposes to use group convolutions which are more HE friendly. Sparsity pattern is also optimized for minimizing rotations as opposed to other criterion used in standard networks.

- The proposed approach is not reproducible based on the descriptions in the paper. There is missing information, hyper-parameters etc. And the paper does not provide code, nor do the authors promise to release code publicly later on.
summary_of_the_review: The paper proposed to use group convolutions and a weight pruning to mitigate the computational bottlenecks of convolutional layers, namely homomorphic rotations. The paper, however, does not compare to or discuss existing attempts toward HE-friendly CNNs. Furthermore, the experiments are conducted on shallow networks. The proposed approach will not provide much computational benefit for deeper networks since convolution is not the main bottleneck for such networks. Reproducibility is also limited.

Overall, the paper has good ideas, but an evaluation, comparison, and discussion of the broader utility of the proposed approach are missing.

**Update After Rebuttal:** The author's rebuttal does not adequately address the comments from the initial review. In theory, the method may have promise in achieving the claims in the rebuttal, but I do not believe it is straightforward and needs to be demonstrated. I will maintain the original rating.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a method for privacy-preserving neural network inference using homomorphic encryption. The main contributation is optimizing the convoluational layer to fit the SIMD structure of homomorphic encryption, including reducing the number of non-zero weights and re-training.

strength_and_weaknesses: The core idea is interesting and novel to me, but I see issues with the presentation.

clarity,_quality,_novelty_and_reproducibility: The technical contribution is described solely in figures and examples. Neither the appendix nor the supplemental material provide further insight. Similarly, I don't understand Algorithm 1 because it's so terse.

I'm missing a description of HEFNet.

Minor issues:
p2: missing whitespace after footnote 1
p3: "A client could upload his/her (...) data (...) for obtaining an online inference service." - Maybe "obtain the inference result"?
p3: "semi-honest (e.g. honest but curious)" - The two terms are synonyms and not an example.
p4: "pruning(Han", "pruning(Wen" - missing whitespace before brackets
p5: suggest G|M instead of "G%M=0"
p6: "algorithm 1" - Algorithm 1
p7: "state-of-the-art method. y We adopt"
p9: missing and misplaced whitespace in the caption of Table 4
p9: "There shall exist potential optimizations" - maybe "We expect potential optimizations"?

summary_of_the_review: Interesting idea marred by editorial issues
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents two techniques for speeding up convolutional neural network inference in homomorphic encryption settings.
Conceptually, the underlying RNS-CKKS homomorphic encryption provides the functionality to securely evaluate functions over a vector of encrypted data in a SIMD manner. In this specific setting, this paper proposes a novel data encoding scheme and a HE-specific pruning technique (sub-block pruning) that substantially reduce the number of homomorphic operations.
strength_and_weaknesses: Strength:
They considered the problem very common and important in building homomorphic encrypted neural network applications, especially the data encoding one. Also, their solutions are simple and very effective. 

Weakness:
- In the experiment setup, they didn't explicitly mention how they deal with level-zero ciphertexts. I assume that they didn't use bootstrapping and do the re-encryption.
- Same as above, usually in applications, bootstrapping will dominate in the computational cost. Which will make this work less important than it is justified to be. However, these are two independent directions of improvement.
- Except for bootstrapping, the choice of activation function is also questionable. Beyond quadratic polynomials, other works already adopt high-precision approximation of the non-linear ReLU function for higher accuracy (e.g. https://arxiv.org/abs/2105.10879 ). Though, it's also independent of the proposed techniques. But it's better to check in experiments if they could be combined.
clarity,_quality,_novelty_and_reproducibility: The result of this paper looks convincing to me, and their group-interleaved data encoding scheme is very interesting.

For clarity, I think the data encoding scheme deserves a better ( and more formal ) explanation. It's hard to comprehend from just a figure and a short paragraph.
summary_of_the_review: Overall, I think the problem they considered is very important, and the proposed solutions are simple and effective and have a very wide application in secure neural network applications. But this paper's contribution is also limited by the lack of consideration of other aspects like bootstrapping and approximation of non-linear activation function.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents way to make computational speedups for FHE in convolutional layers. There are specific aspects of FHE that are extremely slow and the paper makes progress towards reducing costs by some pruning and group convolutions.

Some experiments are presented with shallow CNNs for MNIST and CIFAR. However in terms of accuracy the drop is significant. More comparison prior work and more empirical evaluations are needed. There are some interesting observations in the responses to the reviews, but in my opinion they need to be incorporated properly in the paper and  the paper resubmitted at a suitable venue.



justification_for_why_not_higher_score: Not good enough.
justification_for_why_not_lower_score: NA
Decision: Reject