Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new secure cryptographic protocol for standard components in Transformers, such as matrix multiplications, Softmax, GeLU, and LayerNorm. The authors' method is the first implementation of secure inference on Transformer for 2PC setup. Furthermore, their new customized homomorphic encryptions allow faster and more efficient matrix multiplications over existing state-of-the-art methods. 
strengths_and_weaknesses: Strengths
1. Iron is the first private inference framework for the Transformer in secure 2-party computation technique.
2. Iron's custom homomorphic encryptions allow fast linear and non-linear operations by a significant margin compared to the existing state-of-the-art methods such as Cheetah and SIRNN.
3. Provides the protocols for essential components in the Transformer (e.g., Softmax, GeLU) instead of replacing them with easier operations like ReLU. 

Weaknesses
1. The authors' submission does not include the official implementation.
questions: 1. In line 198, what are these symbols mean? Typo?
2. Compared to the garbled circuit used in the ReLU protocol from CNN secured inference, is secure GELU protocol par regarding run-time and communication cost to the garbled circuit on ReLU? 
limitations: The authors provide detailed comments on their limitations and possible future directions further to accelerate the private inference on the transformer models.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a solution based on LHE+Secret Sharing for private inference of transformer models. The key challenge is in adapting different transformer layers, namely large matrix-matrix multiplication, Softmax, layer normalization, and GELU non-linear function. 

The main contribution of this paper is an encoding scheme that is designed to make matrix-matrix multiplication in LHE more efficient than existing schemes, and a protocol for performing secure matrix multiplication. There are minor contributions in improving the efficiency of evaluating Softmax, layer normalization, and GELU.

Experiments are conducted on 4 NLP models, BERT-Tiny, BERTMedium, BERT-Base, and BERT-Large. Comparisons are performed against Cheetah, a baseline method in terms of runtime (latency) and communication efficiency. 
strengths_and_weaknesses: **Strengths:**
+ The paper is very well written in terms of providing the right amount of background, and clarity of descriptions. I enjoyed reading the paper.

+ Extending the concept of private inference for transformer-based models. Given the growing application of transformers, it is critical to design private inference-friendly versions of these models.

+  The speedup over SIRNN is notable.

+ The data encoding scheme and associated secure matrix multiplication protocol is effective.

**Weaknesses:**
- The primary contribution of the paper is a new data encoding scheme that is tailored for improving the efficiency of matrix-matrix multiplications. However, the paper does not compare (qualitatively or quantitatively) a key existing solution [1] for the same. The current paper only compares against the method in Cheetah, but it is not clear to this reviewer how it compares to [1]. I believe there is no single encoding scheme that is optimal for all matrix sizes.

- The paper appears to be an incremental improvement over CHEETAH. The matrix multiplication is certainly better, but other protocols are minor adaptations of existing practice (e.g., subtracting max before computing softmax is very common practice even in applications that focus only on plaintext) and those implemented in CHEETAH.

[1] Secure Outsourced Matrix Computation And Application to Neural Networks.

This is just a comment to make the authors aware of concurrent work [2] along the direction of adapting transformers for private inference with homomorphic encryption.
[2] THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption, ACL 2022
questions: Overall I am positive about the paper, and will revise the score based on the author’s rebuttal.

- How does the proposed matrix-matrix multiplication scheme compare quantitatively and quantitatively to [1] above. And how do they all scale with matrix sizes since this is a key aspect of variation in transformers?

- Line 323-324 states, "For linear operations, we can generate in advance Beaver’s triple in the matrix form [21, 36] using our matrix multiplication protocol." Did you mean non-linear operations, unless I am missing something I am unsure why Beaver triples are needed for matrix multiplication?
limitations: There is no explicit discussion on the limitations of the current work. There is a separate section for discussing some orthogonal directions to augment the current work, in terms of mixed-bit widths and applying model optimizations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes to use homomorphic encryption for Transformer model inference.  Moreover, the proposed method improves the speed of matrix operations. The experiments show that the inference results are numerically comparable.
strengths_and_weaknesses: Strengths:

- Speedup the inference speed of homomorphic encryption for Transformer models.

- Because the Transformer network is hot now across research areas, how to host the models using private inference is worth exploring.

- The submission will also release the code?

==========

Weaknesses:

- The abstract claims "the first Transformer-based private inference framework". However, there was one paper https://aclanthology.org/2022.findings-acl.277/ that was published in ACL-22. The ACL-22 paper works on the same problem and also uses homomorphic encryption as the solution. The claim can be removed to make the claim more precise.

- (L267) Does the proposed method only support CPUs? There is no experiment conducted on GPUs. It makes the work less practical.

- (abstract) "which preserve the model accuracy of plaintext" The claim is not well supported by experiments. I did not find exact numbers of accuracy in the main content of the submission.

- Only NLP tasks with short input texts are evaluated. More applications can be demonstrated, especially for longer texts. 

- The speed is not directly compared with standard Transformer inference. It seems that the inference is quite slow.
questions: - (L267) Does the proposed method only support CPUs? There is no experiment conducted on GPUs.

- I did not find exact numbers of accuracy in the main content of the submission.

- Only NLP tasks with short input texts are evaluated. How about the performance and speed for longer inputs?

- How about the results on other tasks, such as vision problems?

- The speed is not directly compared with standard Transformer inference.

- How about the memory cost?

- What are the costs of private inference?
limitations: The paper can explicitly include discussions about the limitations.
review_title: null

Meta-review: metareview: The paper studies private inference on transformer-based models. It provides methods to securely perform matrix multiplication and certain other non-linear function computation. The evaluations find that their methods are much more efficient than the state of the art.

All of the reviewers are positive about the paper and I recommend acceptance subject to the authors following up on the promised changes (including "We have submitted our source code in the supplementary material in the revision, and will open source it in the near future").
confidence: Certain
Decision: Accept