Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a new HCNN implementation to perform private inference (PI) called HyPHEN. It proposes a replication-based convolution method (RAConv), which is innovatively alternated with the channel-aligned convolution method (CAConv). It also proposes a hybrid packing method, which is a combination of two existing packing methods: duplicate and multiplex packing. Both these methods reduce the overall number of homomorphic rotations and lead to 3-4 times lower latency compared to the baseline, with comparable accuracy based on ResNet architectures with RNS-CKKS implementation on the CIFAR-10 dataset. 
strength_and_weaknesses: Strengths: 
-	Figures explaining convolutional operations
-	Well-described problem statement

Limitations:

1) Main contributions of the paper, namely, RAConv and hybrid packing method have limited novelty. RAConv is not entirely different from CAConv and the hybrid packing method is the mixture of two existing packing methods. The resulting HyPHEN implementation can be described as a minor/incremental improvement over existing approaches. 

2) Only the SISO case is considered. There is no mention about other scenarios (e.g., MIMO) and whether the proposed techniques can also be applied to other scenarios.

3) Results on larger datasets with higher resolution images (say 224 x 224 x 3) must be included. This is critical to determine if the proposed methods can make real-world private inference scenarios feasible.  
clarity,_quality,_novelty_and_reproducibility: The paper is clear and concise. The quality of the work is reasonable. Experiments are minimal but clear. Novelty is limited. Key training details and code are not available, which makes it difficult to assess reproducibility.

summary_of_the_review: The structure of the paper is good and easy to understand. Moreover, obtained results (e.g., speedup in throughput) are better than the state-of-the-art. However,  the novelty is limited and including experiments for complex settings (e.g., larger datasets) would enhance the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new packing scheme for improving the efficiency of convolution layers over homomorphically encrypted data. The paper first identifies that homomorphic rotations are the computational main bottleneck. Then a new data packing scheme is proposed to minimize the number of required rotations. The network architecture is based off of AESPA and evaluation is performed on CIFAR-10 using the HEAAN library. The proposed approach is able to reduce the computational complexity of ResNet models compared to multiplexed convolutions, the current state-of-the-art.
strength_and_weaknesses: Strengths:

1. The paper identifies a bottleneck in implementing convolutional layers in CKKS, namely the large number of rotation operations. The paper proposes a new packing method that reduces the amount of expensive rotations.

2. The proposed convolution when coupled with low-degree polynomial approximation in AESPA leads to high-performance networks with better latency in ciphertext compared to multiplexed convolution. Although the source of the latency gains are probably elsewhere as described below.

Weaknesses:

1. The comparison to multiplexed convolution [2] (Lee et.al. 2022a) is not entirely fair. [2] seeks to approximate ReLU functions for directly employing them in pre-trained models, as opposed to training the models from scratch as is the case for HyPHEN. As such, the goals and operational settings of the two papers are not the same. HyPHEN needs models to be trained from scratch which is not always possible. This is an important distinction and that is ignored by the paper.

2. Encrypted inference in CNNs is bottlenecked in three respects,
 - (1) depth consumption of the circuit
 - (2) expense of bootstrapping operations
 - (3) cost of convolution operations.

Using low-degree polynomial approximations from AESPA mitigates (1). And, because of using low-degree approximations from AESPA, HyPHEN can use one less bootstrapping layer per residual block compared to [2], which helps mitigate (2). And since bootstrapping and polynomial approximations of ReLU consume the most levels and are the slowest parts, a majority of the latency gains in HyPHEN are probably from these two aspects as opposed to the proposed packing scheme.

Rebuttal Requests:
1. As far as I understand, the main difference from AESPA [1] is a convolution layer with fewer rotations. Yet, the performance of the ResNet models considered in this paper have improved from the corresponding ones in AESPA. Can you comment on what brought about this change, since changing the implementation of convolution should not affect the performance of the model itself?
2. In order to really understand the source of improved overall latency, can you provide a breakdown of the cost of each operation like Table 3 in [2]? Please report this for the proposed approach but with two versions of convolution, (a) multiplexed convolution, and (b) the proposed HP-CAConv+HP-RAConv? Or alternatively add another version of Table 4 where the convolution is multiplexed convolution as opposed to HP-CAConv+HP-RAConv. This should help delineate the source of latency improvements.
4. Reporting latency and claiming latency improvements is not adequate. Latency is affected by multiple system level factors, including number of cores, type and speed of each processor, throttling, other processing that maybe running etc. Measurements for the baselines and the proposed methods have to be performed under the exact same settings, which is hard to do, and must be repeated multiple times for reliability. Are the reported latency values for HyPHEN and the baseline measured under the same settings? Can you report mean and standard deviation over multiple runs?
5. Can you comment on the memory consumption for inference on encrypted data?

[1] AESPA: Accuracy Preserving Low-degree Polynomial Activation for Fast Private Inference, arXiv:2201.06699

[2] Low-Complexity Deep Convolutional Neural Networks on Fully Homomorphic Encryption Using Multiplexed Parallel Convolutions, ICML 2022
clarity,_quality,_novelty_and_reproducibility: - The writing in the paper is relatively clear for the most part, expect for some experimental details that are unclear. 
- The quality of the paper is high for the most part. The experiments are thorough, though there are a few relevant results that are missing.
- The proposed method and results cannot be reproduced without source code. And the paper does not comment on open sourcing the code for the research community. As such, there is limited scope for reproducing the results.
summary_of_the_review: Overall, I am moderately positive about the paper; the new convolution layer certainly reduces the number of required rotations, which are computationally expensive. However, as pointed out above, there is missing information in the paper that does not allow the reader to understand the actual source of improvements, both from a performance and latency perspective.

**Update after Rebuttal:** I saw the rebuttal and the updates to the paper. The comparisons to the convolution in Lee 2022a are favorable from a latency perspective but at the price of higher memory requirements. So it is not clear if the proposed convolution is strictly better. The latency improvements over AESPA seem marginal. I am discarding the accuracy improvements over Lee 2022a since they are essentially a function of better plaintext training, which is equally applicable to Lee 2022a. Given the above, I am inclined to maintain my original rating of the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a hybrid packing method that can speed up the Fully Homomorphic Encryption (FHE) based neural network. Experiments are performed on speeding up the CKKS implementation of ResNet on CIFAR-10 image classification. The motivation is that, in the CKKS FHE computation of convolutions, the HE computation can be speed up by a dedicated design of repacking the values such that homomorphic rotation can be avoided, thus speed up the overall pipeline. Specifically, two new conv algorithms named RAConv and CAConv are proposed. The authors further extend the design of stacking RAConv and CAConv, in building a new scheme of ResNet basic block by  combining CAConv + RAConv. And with multiplex packing, the efficiency of the FHE CKKS pipeline is further improved. The experiments show that RAConv saves more time and the number of rotations over CAConv, and the combination of CAConv + RAConv is more efficient than only using CAConv.
strength_and_weaknesses: Strengths

- The proposed packing method using CAConv and RAConv is novel in improving the performance of the heavy computation of CKKS FHE. 

- RAConv has advantages over CAConv. The combination of CAConv+RAConv has advantages over only using CAConv.


Weaknesses

- Please report the information about the number of ciphertexts in your hybrid packing experiments. Also please report the amount of memory used for the comparison of pros and cons.

- This paper is hard to read in the explanation of the proposed RAConv and CAConv (Fig. 2) and the hybrid packing (Fig. 4). I have to read multiple passes of the section to understand.

- The details of "lazy-SISO" is not provided. It is mentioned multiple times in the paper but never explained how it works. What part of the computation is postponed and calculated when needed? 


3.* Clarity, Quality, Novelty And Reproducibility Can you provide an evaluation of the quality, clarity and originality of the work?
clarity,_quality,_novelty_and_reproducibility: The RAConv and CAConv+RAConv ResNet basic block scheme seems novel and helpful for HE-DNN related research.
But the explanation needs to be made clear. Consider adding supplementary material to provide details. Please also provide sufficient explanations of the lazy-SISO and IR (Image Realign) and the experimental model structure of ResNet20/32/44/18.
summary_of_the_review: The proposed RAConv and CAConv+RAConv Basic Block represent a contribution to the secure DNN inference research.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a new HCNN called python. The methods of alternating convolutions (RA and CA) and the hybrid packing method reduces the overall number of rotations and reduces latency. It is felt that while the problem is interesting and important, the progress this represents over the prior work is relatively marginal.

justification_for_why_not_higher_score: The reviews are very confident and thorough, and I agree with the reviewers that the paper is just below the borderline.
justification_for_why_not_lower_score: NA
Decision: Reject