EncryptedLLM: Privacy-Preserving Large Language Model Inference via GPU-Accelerated Fully Homomorphic Encryption
Authors: Leo de Castro
Daniel Escudero
Adya Agrawal
Antigoni Polychroniadou
Manuela Veloso
Venue: ICML 2025 poster (2025)
Keywords: Fully homomorphic encryption
encrypted inference
approximate activation functions
Primary area: social_aspects->privacy
TL;DR: We develop a GPU-accelerated implementation of FHE and use this implementation to evaluate GPT-2.

Abstract: As large language models (LLMs) become more powerful, the computation required to run these models is increasingly outsourced to a third-party cloud. While this saves clients' computation, it risks leaking the clients' LLM queries to the cloud provider. Fully homomorphic encryption (FHE) presents a natural solution to this problem: simply encrypt the query and evaluate the LLM homomorphically on the cloud machine. The result remains encrypted and can only be learned by the client who holds the secret key. In this work, we present a GPU-accelerated implementation of FHE and use this implementation to benchmark an encrypted GPT-2 forward pass, with runtimes over $200\times$ faster than the CPU baseline. We also present novel and extensive experimental analysis of approximations of LLM activation functions to maintain accuracy while achieving this performance.

EncryptedLLM: Privacy-Preserving Large Language Model Inference via GPU-Accelerated Fully Homomorphic Encryption
Leo de Castro Daniel Escudero
Adya Agrawal? Antigoni Polychroniadou Manuela Veloso
Abstract
resources to be trained, and are often not publicly available as this constitutes the provider's 's intellectual property. This leads to a "inference-as-a-servve send their queries to external providers who locally run an LLM to return result to the client. Furthermore, even open source LLMs such as Llama 2 are very expensive to run in commodity hardware and still require in most cases delegating inference to third party provider.
As large language models (LLMs) become more powerful, the computation required to run these models is increasingly outsourced to a third- party cloud. While this saves clients' compu- tation, it risks leaking the clients' LLM queries to the cloud provider. Fully homomorphic en- cryption (FHE) presents natural solution to this problem: simply enerypt the query and evalu- ate the LLM homomorphically on the cloud ma- chine. The result remains encrypted and can only be learned by the client who holds the secret key. In this work, we propose a GPU-accelerated FHE scheme and leverage it to benchmark an encrypted GPT-2 forward pass. Our approach achieves runtimes that are over x faster than the CPU baseline. We also present novel and ex- tensive experimental analysis of approximations of LLM activation functions to maintain accu- racy while achieving this performance.
scenario, where clients
Unfortunately, delegating inference is undesirable in many settings where the client wants to preserve the privacy of their input. Furthermore, as mentioned above, there are multiple contexts which the model owner also wants to retain privacy of the model itself, for example when the model involves massive monetary resources be trained, or when it incorporates sensitive data (e.g. a bank servic- ing a credit score model trained on internal data). This is particularly relevant as LLMs become more pervasive and find more use-cases that permeate all areas of society. This tension between privacy and utility heavily limits the appli- cability of LLMs, rendering them useless contexts where data cannot be outsourced due to privacy constraints.
Towards resolving this tension, fully homomorphic encryp- tion (FHE) is a promising tool that enables computing on data without revealing it, only outputting the final result (cf (Marcolla et al., 2022) for a survey). Using FHE, a client can encrypt their query to the server, who can locally apply their model this encrypted data, making use of the ho- momorphic properties of the scheme obtain encrypted result, which is sent back to the client for decryption. See Fig. 1a for pictorial representation of this interaction pat- tern. Advances in the last decade on all fronts including algorithms, software, and hardware, have made FHE prac- tical for several tasks that were not within reach before. However, LLMs are in an entirely different regime: their computation is already very expensive in the clear, up the point in which specialized software such high-end GPUs, coupled with several architectural optimizations, are needed in order to provide a reasonable inference latency. Any computation that is ran under FHE becomes much slower, which is going to be a major blocker when porting LLMs to FHE. However, the question remains:
1. Introduction
Large language models (LLMs) have proven to be ground- breaking artificial intelligence tools that are set to change the way humans interact with software. By training on mas- sive amounts of data and using an incredibly large amount of trainable parameters, LLMs are able to provide unprece- dented inference results. The tasks that LLMs excel at include natural language generation, question-answeringg summarization, translation, code generation, among sev- eral others. Models like GPT-3' can produce coherent and contextually appropriate text on a wide range of top- ies. However, these models require massive amounts of
J.PP. Morgan Chase Cybersecurity & Technology Controls, New York, New York, USA FJ.P. Morgan Al Research & Algo- CRYPT CoE, New York, New York, USA. Correspondence to: Leo de Castro dee..decatto@@e @reastrr@@jmcchaee.comm jpmchase.com>.
Proceedings of the 42nd International Conference on Machine Learning, Vancouver, Canada. PMLR 267,, 2025. Copyright 2025 by the author(s). https: :/tpppaa///ppaiiii com
Phttps: //ttps: meta. com/ /lmma//

---

EncryptedLILM: Privacy-Preserving LLM Inference via GPU-Accelerated FHE
How practical is FHE-based privacy-preserving LLM evaluation?
1.1. Our contributions
We approach the problem of improving the efficiency of FHE-based privacy-preserve LLM inference, by provid- ing a GPU-based implementation of the transformer ar- chitecture using CKKS. Prior work (Jung et al., 2021) has shown GPUs to help in improving the efficiency of CKKS. However, to the best of our knowledge, there is cur- rently no available implementation of such works to deploy and test these ideas. In contrast, there are popular open- source CPU-based frameworks that aim at making FHE techniques more accessible by providing high level pro- gramming interfaces, and access to multiple FHE schemes, like CKKS. One such framework is OpenFHE (AI Badawi et al., 2022), which has gained traction as one of the most comprehensive and widely used FHE implementa- tions available. Unfortunately, OpenFHE is limited to CPUs, and hence its performance in tasks such as LLM inference would be rather poor.
Approximating LLM Activation Functions. Fully ho- momorphic encryption can, in principle, evaluate any fune- tion over encrypted data. One approach to implementing an LLM in FHE is to simply evaluate the exact Boolean circuit the describes the LLM over the encrypted input. However, the vast majority of operations in an LLM are the additions and multiplications the comprise the linear layers. Converting arithmetic operations Boolean oper- ations results in significant overhead. This is particularly true in modern FHE schemes, which naturally support the arithmetic operations of addition and multiplication over encrypted vectors. Despite comprising the majority of the work in plaintext LLM evaluation, the linear operations in enerypted LLM evaluation are quite cheap. Instead, the majority of our runtime comes from evaluating the LLM activation functions. As is the case when representing arith- metic operations with Boolean gates, representing the com- plete activation functions with arithmetic gates would re- sult in substantial overhead. Instead, we opt for evaluating only approximations of the LLM activation functions. In- tuitively, this works because modern I.LMs perform well even with low precision, which has been demonstrated in the success of quantized LLM models. We provide exten- sive accuracy benchmarks of our LLM with approximate activation functions in section 4.
In this work we extend the capabilities of OpenFHE by en- abling a GPU-based workflow, which leads to direct effi- ciency improvements across many FHE applications that build on this framework-not only LLMs. This involves combining several optimizations (de Castro et al.. 2021; Kim et al., 2022) simultaneously achieve high perfor- mance and high accuracy in this implementation. We have open-sourced the code of our OpenFHE+GPU extension ?, which we believe will be of independent interest. To our knowledge, this is the fastest open-sourced implementation of CKKS when running on a GPU.
To address this question, good starting point is the CKKS scheme by (Cheon et al., 2017), which enables approximate additions and multiplications over real (in fact, complex) numbers. We provide general introduction to FHE in sec- tion 2.2. The literature in improving the efficiency of this scheme is vast and fruitful (Han & Ki, 2020; Bossuat et al., 2021: Jung et al., 2021), and this has enabled several appli- cations in contexts such as logistic regression (Chen et al., 20) and secure password search (Chen et al., 2018b).
With our GPU-based implementation in place, we set out to benchmark the performance of large language models un- der FHE. We focus specifically on the GPT-2 architecture by OpenAI, which is fully open-source and shares com- mon features with many of the more powerful industry- grade models. One first obstacle we face is that FHE tech- niques do not support all operations available to a common CPU/GPU and instead only supports additions and mul- tiplications. As usual in the FHE literature, we use off- the-shelf polynomial approximations to replicate as faith- fully as possible the transformer architecture, while adapt- ing for FHE use. Notably, we incorporate the recent Soft- Max optimization suggestion of (Cho et al., 2024). which substantially improves the runtime of the approximation by removing the expensive computation of the max cir- cuit and replacing this circuit with just table lookup. This required computing the table of max lookup values, which we based on the extensive tests of the approximate model. Our approximations are discussed in Section 3. Note that these modifications have the potential of nega- tively affecting the accuracy of the model, which is far from
Only the recent work of (Zhang et al., 2024) has explored large language model inference via CKKS, reporting im- plementation of the transformer architecture in C++, using the SEAL library for FHE (https: / / githhu.... com/M icroooft /SEAL), Their experiments report minor accu- racy degradation due to polynomial approximations needed in FHE, and performance in Intel CPUs seems promising, as it is accelerated via HEXL (Boemer et al., 2021). We discuss this work further in section 1.2. Although promis- ing given the massive overheads involved in both LLMs and FHE, this is still far from practical for real-world us- age, even for applications that are not latency sensitive such as text summarization or content generation (in contrast to chatbots or Q/A tasks, which are more demanding in terms of responsiveness).
Pnttpa: // thub http:////ee/////eeeeeeeeeeeeeeee ublic
2

---

EneryptedLILM Privacy-Preserve LLM Inference via GPU-Accclerated FHE
Server
Client
Inlo
Tayerferr di Okdl
dix G Tue
Safmaan
1010 1010
@
Sabwart
apm Norts
Saltes
Trgeos
Emerype
credit repprrs"
Evalusto adapted modell entryoled
"Name: Juhn Smith sttt 4538873328
GLL
Argear
The
Deorgtt
(a) Interaction pattern in outsourced privacy- preserving I.M evaluation. Here, client (right side) encrypts (say) financial reports to the server (left side), who receives an unintelligihle cipher- text. The server evaluates the adapted model- incorporating approximations and working with ci- phertexts instead of cleartext values- and sends back an encryption to the client. Finally, the elient, who is the only party who knows the secret-key. can decrypt this message and Iearn the result.
(h) Overview of the GPT-2 inference flow. The diagram only shows the blocks that are expensive to run in FHE, ignoring simpler operations such as linear or affine layers. The input is the sentence "hello how are", and the completion is "you". To get the next token, only the path associated to the lastest token "you" needs be computed, which in this example leads the token "today". The vertical lines between the leftmost softmax boxes illustrates that each new softmax is somehow dependent on the inputs of previous oncs. Every block is labeled with the dimension of the input it takes. For softmax, the number in parentheses represents how many such calls are made.
Figure 1. On the left: communication pattern between the two parties. On the right: GPT-2 architecture, which corresponds to the local computation by the server.
ideal. To address this, we modify the GPT-2 implemen- tation from HuggingFace's transformers library (https: //gi thub. com hugg i ng fa /t tanns forerss) so that it includes these FHE-friendly modifications, and thoroughly benchmark the resulting accuracy using the LM evaluation harness library (cttps: / //github. com/E E poutherAI/Im-evee weweeaaaaaaaaeaeeeeeee oa-oonnaaaee haress)) on a selec- tion of tasks. This allows us to select optimal parameters for the approximations that strike the right balance between efficient FHE runtimes and model accuracy. Furthermore, for reproducibility we also open source our modified Hug- gingFace GPT-2 implementation.
FHE- LLM inference. FHE-based secure infer- ence has the notable advantage that it preserves the same communication pattern of non-private inference: the client sends the query to the server, who performs certain (pre- sumably heavy) computation and sends back the result. This is applicable to real-world settings where client and servers may not be well connected, and the server is con- siderably more powerful than the client. In this context, the most relevant work in secure LLM inference with FHE is (Zhang et al., 2024). This work makes use of several polynomial approximations from the literature, some of which we borrow as well (see Section 3). Importantly, their implementation is limited to CPU, which caps their performance substantially. Rather than comparing to this work, we instead compare directly to the out-of-the--bo OpenFHE CPU implementations of the FHE functions. This allows us to account for variations in the approxima- tions and the placement of the bootstrapping functions.
Our results given in section 4 show that GPU-accelerated FHE implementation provides a roughly x00 speedup in the GPT-2 forward pass, reducing the time from several hours to just few minutes. This brings the forward pass time down to a range where non-real-time applications be- come more practical, such as document summarization and fine-tuning models on private data.
The work of (Zimerman et al., 2023) studies HE-friendly approximation of the transformer architecture, but it is not applicable to our case since this require re-training. Primer (Zheng ct al., 2023) and THE-X (Chen et al., 2022) also employ FHE for LLM evaluation (Primer in fact mixes FHE and MPC), but these works also make substantial modifications to the underlying model. THE-X even veals intermediate values of the computation.
1.2. Related Work
There is long line of works studying secure inference for protecting the privacy of both client owning a query, and a server holding a trained model. At high level, we can divide these techniques into two groups: highly interac- tive approaches based on MPC, and less communication- demanding but more computationally-heavy paradigms based on FHE.
Privacy-preserving ML for other models. Finally, we mention that there are several other works that have stud- ied FHE-based inference of other machine learning models,
3

---

EncryptedLLMM: Privacy-Preserving LLM Inference via GPU-Accclerated FHE
such convolutional neural networks (ef. (Ao & Boddeti, 2024; Juvekar al., 2018; Gilad-Bachrach et al., 2016; Boemer et al., 2019)). These are not applicable to trans- formers directly as they do not support all of the operations involved in this architecture, and additionally the scale of the models they consider is much more reduced.
and indexing the i-th entry is denoted by ali). Given positive integer n, we let [n] denote the set (1,...,n).
2.1. Large Language Models
A large language model (LLM) is a type of machine learn- ing (ML) model that is characterized by its ability to predict language, with the "large" term emphasizing their com- paratively gigantic sizes and computational demands. The work of (Vaswani et al., 2017) introduced the transformer architecture, which is the basis for several LLMs that came right after. Among LLMs, an interesting and relevant fam- ily are generative pretrained transformers (GPTs). which are used in natural language processing contexts. This fam- ily, developed by OpenAl, has been widely influential and has spawned a series of follow-ups. In this work we focus specifically on the GPT-2 model, which is trained on Web- Text: 40 GB of text, 8 million documents, from 45 million webpages upvoted on Reddit. We chose this model as (1) it is fully open source, (2) it follows the transformer architec- ture shared by other more powerful LLMs, and (3) this is already challenging in terms of efficiency for current FHE approaches. We note however that our findings carry out to several other LLMs that follow this paradigm, such as the larger models like GPT-3 or GPT-4 or other transformer- based LLMs like Llama and Llama In what follows, we describe the GPT-2 architecture in detail. There are four variants of GPT-2 which vary in size and performance: S, M. L and XL, and we discuss below the points where these differ.
LLMs and privacy. The works of (Mireshghallah et al., 2024: Shao et al., 2024) expose critical privacy risk in LLMs, showing they often disclose sensitive informa- tion in inappropriate contexts. In particular, the authors of (Mireshghallah et al., 2024) introduce CONFAIDE, a benchmark assessing LLMs' privacy reasoning, revealing that even GPT-4 and ChatGPT fail to uphold contextual pri- vacy 39% and 57% of the time. This highlights the urgent approaches, such as
need for stronger performing LLM computations on encrypted data without decryption and ensuring that only authorized parties can decrypt and access the final outputs.
1.3. Setting and Threat Model
We consider a client who holds as input a text sequence, and a server who holds a large language model. The goal for the client to learn the evaluation of their query on the model without leaking the input to the server, and while protecting the privacy of the model towards the client. See Fig. la for a pictorial representation of the task and the communication flow. The server does not learn any infor- mation about the client's input, but we provide no correct- ness guarantees regarding the result the server returns to the client-a corrupt server can return an incorrect answer, or no answer at all. This is consistent with prior works, and it is strictly better than the guarantees provided by MPC- based solutions, which may leak information towards cor- rupt server that deviates from the protocol specification.
LLMs use deep learning to analyze and generate human- like text. The transformer architecture by (Vaswani et al., 2017) receives as input a piece of text, which is split into numerical representations referred to as tokens. Transform- ers are comprised of an enconder and decoder section, which are very similar in structure. However, generative LLMs such as GPT are decoder-only, and so for the sake of this work we will focus on the decoder component of the transformer architecture; we note that encoders follow a similar structure and our findings apply to encoder-decoder or encoder-only architectures as well.
We assume the client has access to the tokenizer of the model so that the client can locally transform their text into a sequence of real-valued vectors, which are then en- crypted towards the server. We do not provide any guar- antees on the plaintexts underlying the ciphertexts that the client sends. In particular, corrupt client may send se- quence of vectors that does not correspond to valid token embeddings, and will be able to learn the LLM evalua- tion on this input. This is in par with previous privacy- preserving ML works based on FHE.
The model is trained to predict the best next word given sequence of words. For example, it may receive as input "Today is good", and then predict "day" as the next word. The resulting concatenated sentence "today is good day" can be fed into the model again to obtain as the next word, perhaps, "for". This way a sequence like "today is a good day for running outside" can be generated.
2. Preliminaries
An overview of the GPT-2 architecture, highlighting the blocks that are most relevant for FHE, is given in fig. 1b./
In what follows we provide background on large language models and fully homomorphic encryption.
Some general notation we will use throghout the paper is the following. Vectors are denoted by bold letters, like x.
4

---

EncryptedIILM: Privacy-Preserving LLM Inference via GPU-Accclerated FHE
2.2. Fully Homomorphic Eneryption
Below point the depth of the resulting computa- tion, which is what dictates the bottleneck when instanti- ated with FHE. Note that degree-D polynomial can be evaluated with depth log, (D).
A fully homomorphic encryption (FHE) scheme (Rivest et al., 1978). (Gentry, 2009) is encryption scheme that allows computations to be performed over the data while the data remains encrypted. More formally, an FHE scheme is defined by the following tuple of algorithms.
3.1. Approximation of Comparison
We approximate the output of the sign function
(sk, pk. evk) < KeyGen(1*). This is the key generation algorithm. The input is the security parameter À and the output is three keys. The secret key sk is used for decryp- tion, the public key pk is used for eneryption, and the eval- uation key evk is used to homomorphically compute over encrypted data.
1 x < 0
(ig) =
0 x= 0
x > 0
Arbitrary comparisons between x and y can be constructed by computing sign(x y)).
ct <- Encrypt(pk, m). This is the encryption algorithm. It takes in a message m and public key pk and outputs ciphertext ct.
We use the techniques from (Cheon et al., 2020). There, the approximation is given by h(x) f( gi (2). where fx( and 9m( are carefully chosen polynomials of degree 2n + 1 and 2m + 1 respectively. Note that the composition requires depth d log(2n 1) dg (2m + 1). We will make use of the ʃ and polynomials with degree 9 (so n = m = 4), and we will typically set dj dg =2
(da)
m' + Decrypt(sk, ct'). This is the decryption algorithm. takes in a ciphertext et' and a secret key sk and outputs message m'.
ct2 + Eval(evk, ct, f)) This is the homomorphic evalu- ation algorithm. It takes in input evaluation key evk, ciphertext ct, and a function f. Let m be the message encrypted by ct (i.e. m <- Dscrypt(sk, ct)). The output of Eval is the ciphertext y that encrypts fm).
3.2. Approximation of GeLU
We use the GeLU function (Hendrycks & Gimpel, 2016) defined as
GeLU(a) = 0.5z (1 tanh (V2/m (x 0.044715x²))
FHE must satisfy the same security level as a regular en- cryption scheme, which dictates that party without access to the secret key cannot distinguish between encryptions of any two messages, even if the messages adversarially chosen.
As in (Zhang et al., 2024), we make use of the GeLU ap- proximation from (Dong et.,. 2023), which consists of the following:
0,
X H< <4-4 -4
3. Approximate Activation Functions
Fo(x),
-4 x < -195
GeLU(()=
(1)
Fi(x),
-1.95 < x $3
Since the CKKS scheme is designed to handle arithmetic operations, polynomial evaluation is easily supported. In contrast, functions like exp(-) or tanh(-) cannot be sup- ported in a straightforward way. Following prior work on integer-only evaluation of deep learning models (Dong et al., 2023; Zhang et al., 2024), we approximate all func- tions required in the LLM evaluation with low-degree poly- nomials. Keeping the degree low is important as this mini- mizes the levels consumed in the polynomial evaluation, re- sulting in fewer bootstrapping calls. However, if the degree too low then the approximation may not provide good accuracy. Below we discuss the approximations of differ- ent functions we use all throughout our work. These ap- proximations are typically parameterized by different val- ues that determine the degree and hence the respective ac- curacy. We discuss in Section 4 how we instantiate these parameters concretely.
2,
>>3
where we use the sign(z) approximation from above per- form the comparison. The polynomial Fo has degree 3 and the Fi polynomial has degree 6.
3.3. Approximation of Layer Normalization
Recall that the LayerNorm operation, for E Rdd is de- fined
LayerNorm(y) : V02 +c
3..
Here 7, 3, E I are constants, / = Σi x[j] and = ( - n)?2. The value e a fixed small
constant to avoid division by zero. where μ 1 E a;
n
5

---

EncryptedLLM: Privacy-Preserving LLM Inference via GPU-Accclerated FHE
and Zi (a: n1) + c, where y and | are learned parameters and is small constant. The core non- polynomial operation is given by + 1/VE, for which we can use the inverse square root uses the techniques from (Qu & Xu, 2023). We discuss these velow.
that this optimization does not affect accuracy without any experimental evidence, in work verify this is the case indeed for GPT-2 small (see Table 1). This allows us to avoid the circuit for computing max while maintaining comparable accuracy to the cleartext model.
Division by Square Root. ton's s iterative method. Once starting approximation yo of 1/v ž is chosen, iterate the following for i = I,...,
The authors make use of New-
Division. Division uses Goldschmidt algorithm, which works as follows. To divide A/B, start with an approxima- tion Fo of L/B, and set No- = A and Do B. Then iterate F. +
D;-1, N; + N; - F and D; < Di-1 F,, for i = 1, .,..., d. The output of the division is Na 2 A/B. The depth of this approximation is d, since each iteration consumes one level.
9/-1 (3 zyp-1)
2
with the final approximation being yn- This has depth 3n.
In (Even et al., 2005), it is shown that, if < Fo < 2/B. then the algorithm converges. We set Fg = 10 as the initial estimate, which works well in our experiments.
For choosing the initial point yo, the authors first run a less accurate yet more efficient method. For this they propose two options: Taylor expansion, which is suitable for 1, and using Remez rational approximation, which is better for values that are close to O. The work of (Zimerman et al., 2023) has found empirically that the variance (which is es- sentially the input to the square root) is large, so we use the Taylor expansion for the initial value.
4. Experimental Results
In this section, we present the full LLM runtimes under FHE These evaluations are run entirely on the server, and at no point can the server view the underlying query or any intermediate value. Furthermore, the output of the LLM forward pass can be fed directly back into the model to compute the next token without any interaction with the client. This powerful technique allows arbitrary number of forward passes to be executed on the client's 's encrypted query. This method extends to other operations that require the forward-pan pass as a subroutine, such as fine-tuning on pri- vate data.
For an approximation in the interval [a, b). we choose an odd order Taylor expansion around 2u (a + b)/2 + 1 as the approximate initial value of 1/v2. As suggested in (Qu & Xu, 2023), we take degree 3 (which requires depth 2), so concretely this Taylor approximation looks like:
1
20
3(2 20)2
5(2 20) )a
V
2
8V 20
16
As we mentioned in the introduction, we focus on GPT-2 due to its accessibility as well as the similarity in the ar- chitecture of larger GPT models. We consider the small variant of GPT-2. Our runtimes can be extended to models with many more parameters by linearly scaling the trans- former architecture
3.4. Approximation of MoftMax
For x E RRd, SoftMax is defined as
exp(x [ max)
y m- == exp(xr]i) max
Σ
where Tmax max(x)) (z). The division by e is done in
4.1. Accuracy of the Approximate Model
order to avoid large numerators and denominators.
In order to make our LLM compatible with FHE, we re- place each non-linear function with the corresponding ap- proximation described in section 2. We evaluate the three variants of GPT-2 on standard accuracy benchmarks to ensure that these approximations do not compromise the model's performance. We achieve this by forking the GPT- 2 implementation in the HuggingFace trans forerss li- brary (https: : //gi///// thub. com/ wessggingfacco / rt ns f formers s), and making the following modifications in order to reflect the changes that FHE imposes:
Exponentiation.
The approximation of exp is done via
Taylor series, as in (Lu et al., 2023):
exp(x) R
+
2
x 05
where r, which corresponds to the resulting depth, is pa- rameter of choice.
Max. Although we could compute max using an arithmetic-friendly circuit as in (Cheon et al., 2020), we proceed instead as in (Cho et al., 2024) by empirically approximating this max value per layer using different datasets. While the authors (Cho et al., 2024) claimed
The GeLU activation is replaced by the approximation from Section 3.2. We use degree 2 for the f and g polyno- mials in the comparison from Section 3.1, and we compose
6

---

EncryptedIIIM: Privacy-Preserving LLM Inference via GPU-Accclerated FHE
them? times each.
deviations, highly exploited in the quantization literature (ef (Zhu et al., 2023)), and is crucial for enabling privacy- preserving inference. Note that these approximations are also useful for MPC-based approaches.
LayerNorm is approximated as in Section 3.3. We use 16 or 18 Newton iterations depending on model size as shown in table 2.
SoftMax is approximated as in Section 3.4. For the ap- proximation of exp we use r = 7, and for Goldschmidt algorithm- used for the division we use 14/18/22 itera- tions based model size as shown in table 2.
4.2. Runtimes of LLM Inference in FHE
We now present the end-to-end runtime of a GPT-2 forward pass using our GPU-accelerated FHE. Note that, as illus- trated in Fig. 1b, the complexity of a GPT forward pass is dependent on the position of the token being generated in the output, given that the dimension for the softmax in each decoder block depends on the token position. Furthermore, all tokens of the input sequence have to be processed once by the decoder blocks before any new token can be gener- ated, Throughout this section, we benchmark generating a token at position 128, assuming that the previous input to- kens have been processed. The cost of processing the input is amortized away as more tokens are produced, which is also consistent with prior works.
Performing these modifications is intricate as the transformers library is not intended to support changes such as replacing the SoftMax, for instance, which is rather uncommon in machine learning contexts. Once our modified model is loaded in HuggingFace" "format", we are able to leverage the Language Model Evaluation Harness library (https: : / //gttub..... c om/E wwweetherrimmmeeeeeeeeeeer
on- on-harness) which includes multiple benchmarks to evaluate LLM performance.
We note few important optimizations that are incorpo- rated into this benchmark:
Our accuracy benchmarks appear in table I, where we mea- sure the performance of our modifications with respect to the baseline GPT2 - Small, GPT2 Medium and GPT2 Large models. We run eight diverse tasks: HellaSwag, ARC (Easy), PIQA, Social IQa, MNLI, SST-2, ANLI, and WiC.
Input & Output Sizes. We give the depth of each approx- imation in table 3. Recall from the high-level GPT archi- tecture that Soft and GeLU are run once per block and LayerNorm run twice per block. The GPT-2 model con- sists of 12 blocks, and the final ArgMax function is run at the end of the forward pass. The dimension of one to- ken embedding is 768, and the inputs and outputs of both LayerNorm operations is 128 x 768. The GeLU input con- sists of 24 channels of the typical 128 x 768, resulting in a total input of 3072 x 768. By contrast, the SoftMax in- put is the result of many inner-product operations with the context embeddings, resulting in an input and output size of 128 128. With 216 slots in each ciphertext, this gives the values in the second row of Table 3.
HellaSwag tests an LLM'' ability to perform commonsense reasoning about situations described in natural language. ARC (A12 Reasoning Challenge) is a dataset created by the Allen Institute for Artificial Intelligence (A12) to evalu- ate question answering systems' ability to perform multi- step reasoning. PIQA is the Physical Interaction Ques- tion Answering tasks to test physical commonsense rea- soning. Social IQA measures social and emotional intel- ligence through questions about social interactions. MNLI (Multi-Genre Natural Language Inference) and ANLI (Ad- versarial Natural Language Inference) benchmark a models understanding of entailment, contradiction, and neutrality across genres and adversarially selected examples. SST-2 (Stanford Sentiment Treebank) evaluates binary sentiment classification of movie reviews. WIC (Word-in-Contexx) challenges models on contextual word sense disambigua- tion.
Batched Evaluation. When a function is evaluated over an input that does not use all available slots in a cipher- text, additional performance can be gained by evaluating another input to that function and using the additional un- used slots. This batched evaluation maximizes the avail- able parallelism in the CKKS scheme. For example, the LayerNorm function only requires 1.5 ciphertexts to store the input and output. If only one LayerNorm function is being evaluated, then we must perform the operation over two ciphertexts even though the second is half empty. How- ever, if we have the option of running a second LayerNorm function over an independent input, we can evaluate both LayerNorm functions using only three ciphertexts, which doubles our throughput with only a 50% increase in la- tency. This is an important optimization for tasks such as training or fine-tuning, where the model is evaluated on batches of samples from the training set. We also present
We also provide the different parameters we used in table 2. Together, these benchmarks provide a well-rounded evalu- ation of language understanding, covering reasoning, sen- timent, social context, and word meaning. We refer the reader to the evaluation harness library for details on these tasks.
Overall, we observe that our modifications incur in little ac- curacy degradation with respect to the baseline model. This reflects the robustness of large language model to slight
7

---

EncryptedLLLM: Privacy-Preserving LLM Inference via GPU-Accclerated FHE
Model
Benchmark
ARC (Easy)
Hella- Swag
PIQA
Social IQA
MNLI
SST2
ANLI
Wic
Baseline
0.438
0.289
0.629
0.366
0.337
0.551
0.349
0.492
GPT-2 small
This Work
0.429
0.281
0.626
0.374
0.332
0.556
0.348
0.511
Baseline
0.490
0.333
0.676
0.391
0.351
0.613
0.349
0.5
GPT-2 medium
This Work
0,482
0.319
0.671
0.390
0.353
0.579
0.341
0.491
Baseline
0.531
0.363
0.703
0.395
0.359
0.501
0.333
0.496
GPT-2 large
This Work
0.507
0.321
0.708
0.391
0.356
0.5
0.333
0,481
Table 1. Performance of various sizes of GPT-2 with our different approximations the unaltered baseline. While we only benehmark GPT-2 small in FHE, these plaintext accuracy benchmarks demonstrate the overall scalability of the approximations. The softmax lookup tables were computed seperately each model.
the "unbatched" single-input evaluation for comparison.
Newton
exp
Goldschmidt
Benchmarks. We present our benchmarks in Figure 2 and Figure 3. Both figures display the forward pass time of our encrypted GPT-2 small at position 128. All individual layer benchmarks include the internal bootstrapping time, which is interleaved within the function as needed. This machine has an Intel Xeon chip running at 2.4 GHz and TB of RAM as well an NVIDIA A100 80GB PCIe.
GPT-2 small
16
7
14
GPT-2 medium
18
7
GPT-2 large
18
7
22
Table 2. Approximation parameters for three model sizes. We use polynomials of degree 4 each composed twice for the com- parison approximations (see Section 3.1). We present values of Newton iterations for the inverse square root (see Section 3.3). For Softmax,
In Figure 2, we demonstrate the speedup of our GPU- accelerated FHE library when applied to the task of a GPT- 2 small forward pass. This figure measures our GPU imple- mentation against the out-of-the-box OpenFHE functions running on CPU.
the Goldschmidt algorithm for division, and r 7 for the exp approximation (see Section 3.4).
In the unbatched forward pass, the SoftMax function is one of the most expensive operations primarily due the low utilization of the ciphertext. When switching to batched evaluation, the overhead of the SoftMax drops significantly (4x) as well as the LayerNorm function discussed above. The GeLU function has full utilization the ciphertexts, so the overhead with batching remains the same. The batching speedups translate into the benchmarks for the full model. Recall that the full forward pass consists of 12 blocks and ArgMax. We do not batch the ArgMax evaluation since only a small portion of the ciphertext is left unused.
SoftMax
LayerNorm
GeLU
Argmax
depth
22
13
17
272
# of cts
0.25
1.5
6
Table 3. Depths of our approximate activation functions in GPT- 2 small and the number of ciphertexts required to hold the input for the t token. The approximations (described in section 3) have the same parameters as the plaintext cireuits benchmarked in table 1. The number of slots in each ciphertext is n = 2 Non- integer ciphertexts indicate that not all slots are filled and batched evaluation is available in this layer.
We provide benchmarks at two different security levels de- pending on the application requirements. Setting the secu- rity parameter 128 standard for encryption schemes, although many applications allow slightly weaker 80. Concretely, setting À 128 gives us bootstrapping routine that refreshes 20 ciphertext levels in roughly 550 milliseconds, while relaxing to À = 80 allows a bootstrap- ping routine that refreshes 45 levels in under second. This increase in the bootstrapping throughput is the main source of speedup.
8

---

EncryptedLLLM. Privacy-Preserving LLM Inference via GPU-Accelerated FHE
GPT-2 Encrypted Forward
Forward Pass of Block
Runtimes (In seconds)
Runtimes (In seconds)
4
10a
3
2
SoftMax
10a
CPU GPU
LayerNorm
GeLU
h. == 128
= B0
Security Level
128
À 80
Security Level
Figure This figure presents benchmarks our GPT-2 small for- ward pass running under FHE The polynomial approximations for the activation functions as well as the high-level bootstrap- ping algorithm identical in both henchmarks. The CPU bar uses the out-of-the-box OpenFHE functions, while the GPU bar uses GPU-acceleaated implementation. Both benchmarks for a single (unbatched) evaluation. The speedup when switching the GPU is about x00x.
GPT-2 Forward Pass
70
Runtimes (in seconds)
60
30
20
Blocks
ArgMax
4.3. Limitations
128
À 80
Security Level
We briefly discuss the limitations of our results. Our bench- marks are based on accuracy of the GPT-2 model with the activation functions replaced with polynomial approx- imations. The degree of these polynomials has major impact the performance of the encrypted forward pass, since higher degree directly translates into deeper circuits that require more bootstrapping operations. While many LLM models seems to remain accurate with low precision, many other Al models such as image recognition models require higher precision during evaluation maintain ac- curacy. If model requires a higher precision than GPT-2, the polynomial upproximations would need to be increased. When the required precision increases beyond roughly 16 bits, the complexity of the bootstrapping itself must be in- creased, since internal to the bootstrapping is an approxi- mation modular reduction function. The relatively low precision required by these transformer models crucial to our results.
Figure 3. This figure presents GPU-accelerated encrypted GPT-2 forward pass runtimes for generating token at position The hatched bars indicate the batched evaluation times, where unused eiphertext slots filled with independent evalu- ations (also, see description in paragraph "batched evaluation", p.7). The savings maximized with four independent evalua- tions, allowing the SoftMax to be fully utilized. The full forward pass consists of 12 blacks followed hy ArgMax.
cialized LLM for narrow task, such an assistant for technical role. The additional training data for this spe- cialized task could easily be proprietary, and the resulting model can then be decrypted by the data owner or remain encrypted on the cloud for evaluation. These applications present numerous directions for future work.
A Acknowledgments
5. Future Work
This paper prepared in part for information purposes by the Al Research Group, the AlgoCRYPT Center of Ex- cellence, and Cybersecurity & Technology Controls group of JPMorgan Chase & Co and its affiliates (JP Morgan), and is not product of the Research Department JP Morgan. JP Morgan makes no representation and warranty whatso- ever and disclaims all liability, for the completeness, curacy reliability of the information contained herein. This document is not intended investment research or in- vestment advice, or recommendation. offer or solicitation for the purchase or sale of any security, financial instru- ment, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction,
While real-time chatbots under FHE remains out of reach, these benchmarks suggest that many applications are now practical run in secure way. This includes tasks that require real-time results, such document summariz- ing or drafting (e.g. "Please write a speech for our CEO.") translate to
In addition, this performance improvement tasks that require the forward pass as subroutine, such as fine-tuning a public model on private data. This train- ing task computationally expensive and often requires outsourcing, which can be safely enabled by this library. More concretely, a company may wish to train more spe-

---

EneryptedIILM: Privacy-Preserving LLM Inference via GPU-Accclerated FHE
and shall not constitute solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or to such person would be unlawful.
Chen, H., Gilad-Bachrach, R., Han, K., Huang, Z., Jalali, A., Laine, K., and Lauter, K. Logistic regression over crypted data from fully homomorphic encryption. Cryp- tology ePrint Archive, Report 2018/462, 2018a. URL https: /tppss//////t..../// facr. www.iiaarrr..........o.. 201 8/ 42.
Impact Statement
Chen, H., Huang. Z., Laine, K., and Rindal, P. Labeled psi from fully homomorphic encryption with malicious security. In ACM SIGSAC Conference on Computer and Communications Security, pp. 1223-1237. ACM, Octo- ber 2018b. URL http::pppwwwwww...ooooooot.eeee. m hy/eanss//eeesacccc//plalliiiila -psi- oogal-frrom-ffllll---mmmmorppppp n- aa-wthh-ann -t--ii--mmliccoussseeeiii -
In this work, we present a new implementation of GPU- accelerated FHE and use this implementation to evaluate an enerypted GPT-2 forward pass. This work is an important step towards practical encrypted LLM deployments, which could have extensive applications in the medical and finan- cial industries. These industries require strict security for the sensitive data they handle, ranging from patients' med- ical records to corporate financial details. Enabling this data to be securely processed with state-of-the-art LLMs would significantly improve the efficiency of these indus- tries. Furthermore, FHE is broadly applicable technology. and accelerating FHE with commodity hardware is an ac- tive area of research. In addition to the secure LLM appli- cation discussed in this work, we view our implementation as a general advancement in the practicality of FHE.
Chen, T., Bao, H., Huang. S., Dong, L., Jiao, B., Jiang, D., Zhou, H., Li, J., and Wei, The-x: Privacy-preserving transformer inference with homomorphic encryption. In Findings of the Association for Computational Linguis- tics: ACL 2022, pp. 3510-3520, 2022.
Cheon, J. H., Kim, A., Kim, M., and Song, Y. S. Homomor- phic encryption for arithmetic of approximate numbers. pp. 40944044444 2017 doi: 10. 10.1007/978.33311--009443 5.
References
Al Badawi, A., Bates, J., Bergamaschi, F., Cousins, D. B.,
Cheon, J. H., Kim, D., and Kim, D. Efficient homomor- phic comparison methods with optimal complexity. pp. 221-256, 2020. doi: 10000/978-3000006488888
Erabelli, S., Genise, N., Halevi, S., Hunt, H., Kim, A., Lee, Y., Liu. Z., Micciancio, D., Quah, I., Polyakov, Y., R.V., S., Rohloff, K., Saylor, J., Suponitsky, D., Triplett, M., Vaikuntanathan, V., and Zucca, V. Openfhe: Open- source fully homomorphic encryption library. In Pro- ceedings of the 10th Workshop on Encrypted Compul- ing & Applied Homomorphic Cryptography, WAHC'22, pp. 5363, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450398770. doi: 10.1 10.1145/3560227.353333 org /1/:/10.20 /1w...1111//3111...10
Cho. W., Hanrot, G.. Kim, T., Park, M., and Stehlé, D. Fast and accurate homomorphic softmax evaluation. pp. 4391-4404, 2024. doi: 10. 10.1145/36686443300066
de Castro, L., Agrawal, R., Yazicigil, R., Chandrakasan, A., Vaikuntanathan, V., Juvekar, C., and Joshi, A. Does fully homomorphic encryption need compute accelera- tion? Cryptology ePrint Archive, Paper 2021/1636, 2021. URL https: /epprin//./....// wwappriittticnr......oiiimiimiet org/ /1636.
URL https: /oi.
Ao. W. and Boddeti, V. N. Autofhe: Automated adaption of cnns for efficient evaluation over fhe. USENIX, 2024.
Dong. Y., Lu, W.-j., Zheng, Y., Wu, H., Zhao, D., Tan, J., Huang, 7., Hong, C., Wei, T., and Cheng, W. Puma: Secure inference of Ilama-7b in five minutes. arXiv preprint arXiv:2307.125533 2023.
Boemer, F., Costache, A., Cammarota, R., and Wierzynski, C. ngraph-he2: A high-throughput framework for neural network inference encrypted data. In Proceedings of the 7th ACM workshop on encrypted computing & ap- plied homomorphic cryptography, pp. 45 15-56,77 2019.
Even, G., Seidel, P.-M., and Ferguson, W. E. A parametric error analysis of goldschmidt's 's division algorithm. Jour- nal of Computer and System Sciences, 1893:118-139, 2005.
Boemer, F. Kim, S., Seifu, G., DM de Souza, F., and
Gopal, V. Intel hexl: accelerating homomorphic encryp- tion with intel avx512-ifma52. In Proceedings of the 9th on Workshop on Encrypted Computing & Applied Ho- momorphic Cryptography, pp. 57-62, 2021.
Gentry, C. Fully homomorphic encryption using ideal lat- tices. pp. 169-178, 2009. doi: 10.1 10.11445/334144 440.
Bossuat, J.-P., Mouchet, C., Hubaux, J.-P. Efficient bootstrapping for approximate homomorphic encryption with non-sparse keys. pp. 587-617, 2021. doi:
J. R., and
Gilad Bachrach, R., Dowlin, N., Laine, K., Lauter, K., Nachrig, M., and Wernsing, J. Cryptonets: Applying neural networks to encrypted data with high throughput
10

---

EncryptedLII: Privacy-Preservinn LLM Inference via GPU-Accclerated FHE
and accuracy. In International conference on machine learning, pp. 201-210. PMLR, 2016.
Shao, Y., Li, Shi, W., Liu, Y., and Yang, D. Privacylens: Evaluating privacy norm awareness of language models in action. CoRR, abs/2409 00000,,,, 2024, doi: 10.48550 /ARXIV DEXXIV.2409.0111 URL https: /cii...... org/10 48550/5/// a 2409. 30.
Han, K. and Ki, D. Better bootstrapping for approximate homomorphic encryption. pp. 364-390, 2020. doi: 10.1 007/978-3-030-401 4000000-41-111161-1100-----3 86-3_16. 16.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, t.., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). arXiv: Learning, 2016. URL https: /api. 3.
Zhang, J., Liu, J., Yang, X., Wang, Y., Chen, K., Hou, X., Ren, K., and Yang. X. Secure transformer inference made non-interactive. Cryptology ePrint Archive, 2024.
Jung, W., Kim, S., Ahn, J. H., Cheon, J. H., and Lee, Y. Over 100x faster bootstrapping in fully homomorphic encryption through memory-centric optimization with GPUs. 2021(4):114-488 2021. doi: 10.46586/tches w2021..4.114-114. URL httpt:ttt//thhheeee iacr.o http:/ppppppptt///pacccccccceeee
Zheng, M., Lou, Q., and Jiang, L. Primer: Fast private transformer inference on encrypted data. In 2023 60th ACM/IEEE Design Automation Conference (DAC), pp. 1-6. IEEE, 2023.
Juvekar, C., Vaikuntanathan, V., and Chandrakasan, A. (GAZELLE]: A low latency framework for secure neu- ral network inference. In 27th USENIX security sympo- sium ( (USENIX security 18), pp. 1651-1669, 2018.
Zhu, X., Li, J., Liu, Y., Ma, C., and Wang, W. A survey on model compression for large language models. arXiv preprint arXiv:2308.07633 2023.
Zimerman, I., Baruch, M., Drucker, N., Ezov, G., Soceanu, O., and Wolf, L. Converting transformers to polynomial form for secure inference over homomorphic encryption. arXiv preprint arXiv:231 1,00611,, 2023.
Kim, A., Papadimitriou, A., and Polyakov, Y. Approxi- mate homomorphic encryption with reduced approxima- tion error. In Galbraith, S. D. (ed.), Topics in Cryptology CT-RSA 2022, pp. 120-144, Cham, 2022. Springer In- ternational Publishing. ISBN 978-3-030-95312-6.
A. GPU Implementation of the CKKS FHE Scheme
Lu, W.jj.. Huang, Z., Gu, Z., Li, J., Liu, J., Ren, K., Hong. C., Wei, T., and Chen, W. Bumblebee: Secure two-party inference framework for large transformers. Cryptology ePrint Archive, 2023.
In this section, we present our implementation of the CKKS FHE scheme. This implementation extends the popular and feature-rich OpenFHE library (Al Badawi et al., 2022) to use GPU accelerate the homomorphic operations. While we use this library to implement an LLM forward pass, this is the first open-soureed implementation of GPU-accelerated CKKS scheme, which is of significant in- dependent interest. The audience for this section is some- one more familiar with the CKKS FHE scheme; this sec- tion can be safely skipped by those who are only interested in the LLM benchmarks. However, as a brief motivation for the focus on this function, the bootstrapping operation is at least 50% of the runtime in all layers and typically closer to 80-90% of the total time.
Marcolla, C., Sucasas, V., Manzano. M., Bassoli, R., Fitzek, F. H., and Aaraj, N. Survey on fully homomor- phic encryption, theory, and applications. Proceedings of the IEEE, 110(10)1572--1699 2022.
Mireshghallah, N., Kim, H., Zhou, X., Tsvetkov, Y., Sap, M., Shokri, R., and Choi, Y. Can Ilms keep a secret? test- ing privacy implications of language models via contex- tual integrity theory. In The Twelfth International Con- ference on Learning Representations, ICLR 2024, Vi- enna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL ht hww.t::tt///....... // http://ppnreviieeeee net forum? d= gmg7t8b450.
Our starting point for this implementation is the work of Jung et al. (Jung et al., 2021), which focuses on acceler- ating the bootstrapping implementation. The public por- tion of this code* is limited to the individual operations accelerated in their work, including the expensive number- theoretic transform (NTT) and RNS basis-change opera-
Qu, H. and Xu, G. Improvements of homomorphic secure evaluation of inverse square root. pp. 110-127, 2023. doi: 10. 10.1000/978-000 981-9987356-999
Rivest, R. L., Adleman, L., Dertouzos, M. L., et al. On data banks and privacy homomorphisms. Foundations of secure computation, 4(11):169-188 1978.
htttpa://gi com/ scallcalllee weeeeennu/ckeee sntleessnncckkk------ ore

---

EncryptedLLM: Privacy-Peeserving LLM Inference via GPU-Accelerated FHE
tions, rather than end-to-end bootstrapping implemen- tation. We incorporate these kernels into the OpenFHE CKKS bootstrapping code and implement further opera- tions to connect these core functions and avoid any data movement off of the GPU. Our code includes an end- to-end bootstrapping implementation integrated into the OpenHHE API as well ment the LLM layers described above. This implementa- tion inherits the improved accuracy from the careful track- ing of the CKKS scaling factor in OpenFHE.
functions required to imple-
As prior works demonstrate (de Castro et al., 2021), the bottleneck of CKKS bootstrapping quickly becomes the memory transfer if the compute accelerates faster than the local storage capacity. This is due to the size of the evalu- ation keys, which for bootstrapping can reach bytes. For our benchmarks,
of giga- GPU with 80 GB of RAM, which allows us to cache all of the evaluation keys needed for bootstrapping and the subsequent LLM layers.
We present the benchmarks of our bootstrapping imple- mentation in fig. The CPU benchmarks were run on a machine with an Intel Xeon chip running at 2.4 GHz and 2 TB of RAM. The GPU benchmarks were run on the same machine and used NVIDIA A100 80GB PCIcAll bench- marks were run within OpenFHE, which runs depth 13 approximation of the CKKS modular reduction function. All bootstrapping hyperparameters were the same in all benchmarks. The level budget for the homomorphic en- coding and decoding was set to 4 resulting in a total boot- strapping depth of 21. The number of decomposition dig- its was to 3. The security level is least 128 bits for the 10 and 20 output levels and 80 bits for 30 and 40 out- put levels. This is to accommodate the maximum modulus without growing the ring dimension.
Bootstrapping 16 Slots
Runtimes milliseconds)
10
10a
CPU
GPU
20 of Output Levels
Figure 4. This figure presents comparison between CPU im- plementation of bootstrapping and GPU implementation of bootstrapping for
2 slots with various output levels. Ob- serve the log scale on the y-axis. The CPU implementation re- quires roughly 4-6 seconds per output
while GPU imple-
mentation only requires 22-27 ms per output level, representing speedup of 180-220x. These benchmarks are highly consistent with less than 5% change over 10 iterations.
12


Reviews:
[{"review_id": "GGtB3kglu5", "reviewer": "Reviewer_gAoN", "summary": "This paper presents a GPU-accelerated implementation of CKKS-based fully homomorphic encryption (FHE) for non-interactive private LLM inference. Specifically, it focuses on enabling privacy-preserving (for users' sensitive data) access to proprietary LLMs (e.g., ChatGPT) for latency-tolerant tasks such as document summarization. \n\nTo make the model compatible with HE-only inference, and also to reduce the computational burden, the authors employ off-the-shelf approximation techniques for nonlinear operations such as Softmax, GELU, and LayerNorm. The authors have shown a remarkable speedup—approximately **200×** over a standard CPU-based implementation (openFHE)—on the GPT-2 small model (12 layers, 12 heads, 768 embedding dimensions). This acceleration is primarily achieved by optimizing bootstrapping operations, which constitute the dominant source of latency in CKKS-based FHE.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n\nThe threat model here is semi-honest. \n\nWe wrote this paper with a broad audience in mind, including cryptographers who may not be familiar with the LLM circuit. The paper describes the details of the LLM circuit as it is necessary to completely implement the LLM as a homomorphic encryption circuit, so in a sense this is a description of the homomorphic computation that is benchmarked in our work. \n\nWe agree that the results in Appendix E are very important, and our implementation of GPU-accelerated homomorphic encryption is a core contribution of this work. We placed these results in the appendix since the main audience for this work is machine learning researchers who may not be familiar with the FHE bootstrapping operation. We are happy to move them to the main body. \n\nQ1: Yes, the final Layernorm time is included in the final argmax time. We will clarify this in the next version. \nQ2: Yes, all FHE operations are performed on CUDA device vectors without transferring back to the CPU. This includes all low-level polynomial operations like the NTT as well as higher-order FHE algorithms like key switching, automorphisms, and residue number system decomposition. \n\n\nLarger models: \n\nSince the submission, we have run additional experiments on larger models with nearly identical approximation parameters.\n\nAll accuracy benchmarks are reported with the standard, unmodified benchmark first followed by the model run with polynomial approximations. \n\nGPT-2 Small (additional benchmarks)\n\n- Social IQA: 0.366, 0.374\n- MNLI: 0.337, 0.331\n- SST2: 0.550, 0.556\n- OpenBook QA: 0.164, 0.186\n- ANLI-R1: 0.341, 0.341\n- ANLI-R2: 0.339, 0.329\n- ANLI-R3: 0.349, 0.348\n- Wic: 0.492, 0.511\n\n\nGPT-2 Medium\n\n- Arc Easy: 0.491, 0.489\n- PIQA: 0.676, 0.675\n- Social IQA: 0.391, 0.393\n- MNLI: 0.352, 0.354\n- SST2: 0.614, 0.638\n\n\nGPT-2 Large\n\n- Arc Easy: 0.532, 0.533 \n- PIQA: 0.703, 0.707\n- Social IQA: 0.396, 0.393\n- MNLI: 0.359, 0.357\n- SST2: 0.5, 0.5", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "There are no theoretical claims made in the paper.", "experimental_designs_or_analyses": "Yes. The experimental evaluation (presented in Table 1) is quite limited, and it could have been more comprehensive. There are two key limitations: 1) the authors have used only the GPT-2 small model, and do not show the implication of performance degradation incurred from the approximation of nonlinear operations for deeper and wider models. For example, what happens (to the efficiency gain and performance degradation) when we increase the number of layers (from 12 to 18 or 24) and/or context size (from 128 to 256); and 2) since the authors have used the pre-trained model and evaluated the performance degradation on downstream tasks, it should have been included a more diverse set of downstream tasks (from the llm-evaluation-harness library). \n\nIdeally, the implications of approximations should also be shown for training from scratch. How much does it increase the perplexity?", "relation_to_broader_scientific_literature": "It could have been much better when it comes to the connection with the prior related findings/results. For instance, the paper does not mention the some of the prior work on GPU acceleration (HE and MPC), and also the methods for improving the bootstrapping performance.", "essential_references_not_discussed": "[1] Watson et al., Piranha: A GPU Platform for Secure Computation, USENIX Security 2022\n\nWhile this paper does not accelerate FHE on GPUs but rather focuses on MPC-based acceleration for nonlinear operations, the authors should have included a discussion on whether their approach could be extended to accelerate LLM nonlinearities such as Softmax, GELU, and LayerNorm. \n\n[2] Kim et al., Cheddar: A Swift Fully Homomorphic Encryption Library for CUDA GPUs, 2024\n\nThis paper directly focuses on GPU acceleration for CKKS-based FHE. Could the NTT acceleration presented in [2] be integrated with the authors' bootstrapping acceleration for further improvement? If so, discussing the feasibility, potential challenges, and expected benefits of such a combination would strengthen the paper and provide valuable insights for future work. \n\n[3]  Jha et al., AERO: Softmax-Only LLMs for Efficient Private Inference, 2024\n\nThis paper removes LayerNorm and GELU activations to enable faster private inference in hybrid protocol settings (HE + MPC). Given this design choice, would the authors' GPU-accelerated approach be even more beneficial for LLMs with fewer nonlinear components? A discussion on how the acceleration scales with different levels of nonlinearity would strengthen the paper and provide insights into the broader applicability of their method.", "other_strengths_and_weaknesses": "### Strength \n\n$\\bullet$ The authors have provided the code implementation and promised to open-source it, which could be beneficial for the researcher working in this field. \n\n$\\bullet$ Writing is coherent and the paper is easy to follow.", "other_comments_or_suggestions": "$\\bullet$ The threat model presented in Section 1.3 could be more clearly articulated. It is not explicitly stated whether the setting assumes a semi-honest or malicious client/server, which is crucial for understanding the security guarantees of the proposed approach. \n\n$\\bullet$ There is an excessive emphasis on the basics of LLM architecture and cryptographic protocols in the main text, which could have been more concise. Meanwhile, some key experimental results have been relegated to the appendix. In particular, the results in **Appendix E** are crucial to the paper’s core contributions and should be included in the main text for better visibility and impact.\n\n\n**Line #385** Substituting BatchNorm with LayerNorm in LLM does not work. See [3]\n\n[3] Wang et al., Understanding the Failure of Batch Normalization for Transformers in NLP, NeurIPS 2022.", "questions_for_authors": "$\\bullet$ Did you include the final LayerNorm layer (in the LM-head) for end-to-end latency? A GPT-2 model with the 12 layers has 2*12 +1 LayerNorm layers. \n\n$\\bullet$ Does the GPU acceleration improve the NTT kernel?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZH2iUdQxoL", "reviewer": "Reviewer_BXJU", "summary": "The paper presents a novel approach to privacy-preserving inference for large language models (LLMs) using GPU-accelerated fully homomorphic encryption (FHE), specifically targeting the GPT-2 architecture. It addresses significant privacy concerns associated with LLMs, particularly when deployed on third-party cloud services, by allowing clients to encrypt their queries and perform secure computations without disclosing sensitive information. The authors introduce a GPU-based implementation of the CKKS scheme, which enhances the performance of homomorphic operations, achieving speedups of over 200 times compared to traditional CPU methods. Key contributions include efficient approximations for layer normalization and SoftMax operations, which maintain model accuracy while optimizing computational efficiency. The paper also discusses the architecture of the neural network decoder block and the use of polynomial approximations for activation functions to facilitate secure evaluations under FHE. Experimental results demonstrate the feasibility and practicality of their approach for applications such as document summarization and fine-tuning on private data. Overall, the research significantly advances the field of secure LLM inference, making it more accessible for sensitive applications in areas like healthcare and finance.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. \n\nWe assure the reviewer that implementing GPU-accelerated FHE is a highly non-trivial task, requiring the synthesis of dozens of algorithms & optimizations from prior works. We additionally implemented several optimizations derived from the design of custom ASIC & FPGA implementations of CKKS. This includes the fusing of operations into CUDA larger kernels to maximize throughput. Overall, we added roughly 10k lines of code to the core OpenFHE library. We are happy to provide more details of our implementation in a comparison to the work [1] of Zhang et al. We had previously excluded [1] from our comparisons in an earlier draft of this work, since prior versions of [1] had serious errors in the reported benchmarks. The current version of [1] is the latest in a series of revisions correcting these major issues. We have not had a chance to confirm their latest results, although based on the benchmarks reported in the latest version of [1] our bootstrapping is at least 10x faster. We are confident we can outperform this work when using the same resources & parameters.", "claims_and_evidence": "1. I think the technical contribution is very weak for this work. I think author should better clarify the challenge for using GPU to perform this large-scale task, and what is the main difference from CPU.\n\n2, Missing important citation [1], this work also using GPU to do large-scale machine learning model inference in FHE. How is the difference between this work and [1]?\n\nReference:\n[1] Zhang, J., Yang, X., He, L., Chen, K., Lu, W. J., Wang, Y., ... & Yang, X. (2024). Secure transformer inference made non-interactive. Cryptology ePrint Archive.", "methods_and_evaluation_criteria": "see above", "theoretical_claims": "none", "experimental_designs_or_analyses": "none", "relation_to_broader_scientific_literature": "none", "essential_references_not_discussed": "see above", "other_strengths_and_weaknesses": "none", "other_comments_or_suggestions": "none", "questions_for_authors": "none", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "01jVc54Ef2", "reviewer": "Reviewer_LCvT", "summary": "This paper addresses the practical challenge of performing LLM evaluation, where clients preserve the privacy of their inputs and model owners retain privacy of the model. They propose using FHE to achieve the goal: encrypt the client inputs, perform evaluation homomorphically over encrypted data, and the results can only be decrypted by the clients who own the decryption key.\nSpecifically, the authors contribute a practical implementation of GPU-accelerated FHE scheme and use it to realize and evaluate an encrypted GPT-2 forward pass.\nThe results show that they achieve 200x faster than the CPU-based FHE implementation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. \n\nQ1: We set the parameters of our approximation so that accuracy is essentially unchanged. To see the evidence of how well our approximations scale, please see the rebuttal for reviewer gAoN. \n\nQ2: We added roughly 10k lines of code to the core OpenFHE library as well as an additional 5k lines for the LLM layer benchmarks & associated tests. This was a complete reimplementation of the core CKKS algorithms optimized for the high-throughput parallelism of a GPU.\n\nQ3: We chose three standard benchmarks to test our approximations, and we selected the three values in the submission as varying sufficiently from one another. In the rebuttal for gAoN, we give benchmarks on additional datasets. \n\nQ4: The CKKS homomorphic operations are highly parallel, so the runtimes of the activation function layers will mostly scale with the size of the input. The runtime of the linear layers will also grow, although proportionally this will still be a small additional overhead. Aside from minor changes due to the circuit layout, the model runtime can be computed by summing the runtimes of the individual layers.", "claims_and_evidence": "The claim that GPU-accelerated FHE-based LLM inference makes non-real-time applications practical is supported by their evaluation results that end-to-end forward pass takes several seconds using GPU acceleration.", "methods_and_evaluation_criteria": "The reason why they choose FHE is well-justified, as it maintains the same client-server communication pattern as in non-private inference.\nThe authors' rationale for evaluating the GPT-2 model is sound. GPT-2 is open-sourced and is a representative of the transformer model.", "theoretical_claims": "This is a practical paper, no theoretical claim made.", "experimental_designs_or_analyses": "The finding that their modifications incur in little accuracy degradation with respect to the baseline model. The results are congruent with findings in the quantization literature. This alignment across different research fields reinforces the reliability of the experimental design.\n\n\npage 6 col 1 line 312-313: we benchmark generating a token at position 128. \nPlease clarify why 128 is chosen.\n\n\npage 7 col 1 line 330-333: the comparison is confusing as the larger security parameter takes less time than the smaller security parameter. Please clarify this.", "relation_to_broader_scientific_literature": "The method can be extended to other operations that require the forward-pass as a subroutine, such as fine-tuning on private data.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\nThis paper is well-written. I really enjoy reading the paper. The clear explanations and structuring makes the methodology and findings accessible to readers.\n\n\nWeaknesses:\n* While engineering aspects are well-executed, the paper lacks substantial theoretical contributions. Most of the techniques used in this paper are quite standard, e.g., use polynomial approximations to improve the performance.\n* Although the authors discuss the differences with other works in the related work section, including numerical data comparisons could significantly strengthen this work.", "other_comments_or_suggestions": "Typos:\n* page 2 col 2 line 98-100: This required computing the table of max lookup values, which we based on the extensive tests of the approximate model. (remove “we”?)\n* page 7 col 2 line 372: could easily be proprietary, and The resulting model can (remove “The”?)\n\n\nComments:\n* page 6 col 1 line 317-318: We note a few important optimizations that are incorporated into this benchmark\nI was confused the first time I read the paragraph below cuz there’s only one optimization (batch evaluation), while “Input & Output Sizes” is a prerequisite for the optimization. And also this optimization is not FHE specific, but more like a general optimization approach.", "questions_for_authors": "* Please clarify how do you balance between privacy (security parameter) and utility\n* Is there any engineering efforts that worth mentioned? Like how much lines have been added to the original framework?\n* “LM evaluation harness library to select optimal parameters for the tradeoffs between efficiency and accuracy.” \n Could you please clarify how you decide the criteria to choose those parameters?\n* page 5 col 1 line 272-274: Our runtimes can be extended to models with many more parameters by linearly scaling the transformer architecture.\nCould you please explain explicitly how linear scaling would work?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B09INrZUyG", "reviewer": "Reviewer_VjCM", "summary": "This privacy preservation for cloud-deployed LLMs is considered. This work proposes a GPU-accelerated Fully Homomorphic Encryption(FHE) for LLMs. Evaluations are made on a GPT-2 LLM.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. \n\nWe refer to the rebuttal of reviewer gAoN for additional benchmarks with larger models. We will include these in the next version.", "claims_and_evidence": "See Strengths And Weaknesses below", "methods_and_evaluation_criteria": "See Strengths And Weaknesses below", "theoretical_claims": "See Strengths And Weaknesses below", "experimental_designs_or_analyses": "See Strengths And Weaknesses below", "relation_to_broader_scientific_literature": "See Strengths And Weaknesses below", "essential_references_not_discussed": "See Strengths And Weaknesses below", "other_strengths_and_weaknesses": "strengthens:\n\n1.\tPromising performance (200 times faster than the CPU baseline).\n\n2.\tSource code of the implementation provided as supplemental material.\n\nWeaknesses:\n\n1.\tThe empirical evaluation is only done for GPT-2 small (124M), which is somewhat like a toy model compared to today’s mainstream open-sourced LLMs like LLAMA2 (7B/13B). it remains questionable whether the conclusion made on the former still hold for the latter.\n\n2.\tThe effect of FHE approximation on the utility performance. I am afraid an evaluation on only three benchmarks can hardly capture the performance of an LLM.\n\n3.\tLack of ablation study on the choices of implementation", "other_comments_or_suggestions": "1.\tL804: citation error.\n2.\tThe paper is written like a tech report for a software. I found it hard to grasp many useful insights, neither theoretical nor technical.\n3.\tSome more works (eg [A]) on homomorphic encryption for transformers can be discussed.\n\n \n[A] HETAL: efficient privacy-preserving transfer learning with homomorphic encryption. ICML’23.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a method to performing FHE with GPUs that achieves significant speedup compared to existing CPU solutions. This required both significant engineering and systems design, and some novel applications of cryptographic techniques to achieve the superior performance. However, some reviewers were concerned with the lack of thorough experiments which was partially addressed in the rebuttal. Further, though this is a substantial improvement, it is likely still not the case that FHE can be deployed (even with GPU) for large frontier models.
Decision: Accept (poster)