# Enhancing Privacy in Large Language Model with Homomorphic Encryption and Sparse Attention
## 2 Related Work  
### Fully Homomorphic Encryption  
The fundamental principle of FHE enables arbitrary computations to be performed on ciphertexts, with the result of such computations, when decrypted, remaining correct [23]. The essence of this principle is to process and analyze data without exposing the original data content, thus accomplishing complex data processing tasks while protecting privacy [24].  
Mathematically, assume a plaintext message \(m\) is encrypted using a public key \(pk\) to produce a ciphertext \(c\), as shown in Figure 1:  
\[c=\text{Enc}_{pk}(m) \tag{1}\]  
FHE supports two basic operations: homomorphic addition and multiplication. Given two ciphertexts \(c_{1}=\text{Enc}_{pk}(m_{1})\) and \(c_{2}=\text{Enc}_{pk}(m_{2})\), homomorphic addition and multiplication can be represented as:  
\[c_{\text{add}}=c_{1}\oplus c_{2}=\text{Enc}_{pk}(m_{1}+m_{2}) \tag{2}\]  
\[c_{\text{mul}}=c_{1}\otimes c_{2}=\text{Enc}_{pk}(m_{1}\cdot m_{2}) \tag{3}\]  
Here, \(\oplus\) and \(\otimes\) denote the homomorphic addition and multiplication operations, respectively. In the context of large-scale dialogue models, for example, the intention might be to compute the language model scores of a user without revealing the user's input.  
Figure 1: Flowchart of FHE.  
If the user's input is \(x\) and the language model parameters are \(\theta\), the desired score to compute is \(f(x;\theta)\). With FHE, the input \(x\) can be encrypted on the user's device to obtain \(c_{x}=\text{Enc}_{pk}(x)\), then the computation under encryption is performed on the server to obtain the encrypted score \(c_{f}=\text{Enc}_{pk}(f(x;\theta))\), and finally, \(c_{f}\) is sent back to the user's device for decryption. The application of FHE in natural language processing (NLP) tasks can extend to various machine learning models, such as sentiment analysis and text classification. For instance, in sentiment analysis, the model might need to encode each word in a sentence and compute sentiment predisposition. For each word, \(w_{i}\), encoded as \(v_{i}\), the sentiment predisposition calculation can be viewed as the dot product of a weight matrix \(W\) and the encodings:  
\[s=\sum_{i}W_{i}\cdot v_{i} \tag{4}\]  
If each \(v_{i}\) is encrypted, then the aforementioned computation needs to be completed under homomorphic encryption:  
\[c_{s}=\bigoplus_{i}(W_{i}\otimes c_{v_{i}}) \tag{5}\]  
In the task of this paper, a privacy-protected dialogue large model framework based on FHE and attention mechanisms, the computation of attention weights can also be implemented using homomorphic encryption. Normally, a simple attention weight computation can be expressed as:  
\[\alpha_{ij}=\frac{\exp(\text{score}(h_{i},h_{j}))}{\sum_{k}\exp(\text{score}( h_{i},h_{k}))} \tag{6}\]  
where _score_ is a scoring function, and \(h_{i}\) and \(h_{j}\) are different states in a sequence. Under the framework of homomorphic encryption, this computation can be transformed into:  
\[c_{a_{ij}}=\frac{\exp(\text{Enc}_{pk}(\text{score}(h_{i},h_{j})))}{\bigoplus_ {k}\exp(\text{Enc}_{pk}(\text{score}(h_{i},h_{k})))} \tag{7}\]  
However, as exponential operations and divisions are not directly supported in homomorphic operations, as shown in Figure 1, approximate algorithms or special encoding methods need to be designed to implement these operations. In summary, FHE technology offers significant potential in protecting privacy within large dialogue models. Although FHE currently faces challenges in computational efficiency, with advancements in algorithms and hardware, it is expected to be widely applied in the near future. In this paper, we will explore in detail how to apply FHE to large dialogue models, particularly, how to ensure the correctness and efficiency of computations in natural language tasks by designing effective homomorphic algorithms. Through such research, it is hoped to advance the application of privacy protection technologies in dialogue systems, providing a stronger technical guarantee for the security of user data.