# Enhancing Privacy in Large Language Model with Homomorphic Encryption and Sparse Attention
## 3 Materials and Methods
### Proposed Method
#### 3.3.3 Privacy-Preserving Transformer Framework  
A novel transformer framework has been designed to provide privacy-preserving capabilities, primarily through the integration of FHE technology, as shown in Figure 5. The core idea of this framework is to ensure the security of data throughout the processing chain, from input to model processing, and finally to output, with every step completed in an encrypted state. This section details the main differences between our privacy-preserving transformer framework and the original transformer model, as well as the specific details of the implementation.  
**Key Differences in Model Architecture:** The original transformer model is based on the self-attention mechanism, capturing internal dependencies by computing attention scores across all pairs of words in the input sequence. This process involves extensive real-number computations, which are infeasible in the context of FHE. Consequently, in our privacy-preserving transformer framework, the original computational flow has been adjusted to only use operations supported by homomorphic encryption algorithms.  
1. Adjustment of Self-Attention Mechanism: Within the homomorphic encryption environment, it is not possible to directly compute the softmax function, a key component of the self-attention mechanism in the original transformer model. Therefore, a method for calculating attention scores based on FHE has been designed, utilizing algorithms compatible with homomorphic encryption to approximate the softmax function.
2. Encryption of Weights and Biases: In the privacy-preserving transformer framework, all model weights and biases are pre-encrypted. Hence, the model does not need to access any plaintext information before performing any calculations.
3. Protection of Intermediate States: At every layer of the model, all intermediate states exist in encrypted form. This ensures that data privacy is protected, even within a multi-layer network structure.  
**Input, Output, and Processing Flow:** Within our privacy-preserving transformer framework, a user's input is first encrypted using a homomorphic encryption algorithm and then fed into the model in an encrypted state. The model performs encrypted linear transformations, encrypted activation functions, and encrypted attention mechanisms on the encrypted input and produces an encrypted output. Finally, the encrypted output is returned to the user, who can decrypt it locally with a private key to obtain the final plaintext result.  
Figure 5: The privacy-preserving transformer framework proposed in this paper.  
Mathematically, suppose the user's original input is a vector \(x\), the model's encrypted weight matrix is \(\text{Enc}(W)\), the encrypted bias is \(\text{Enc}(b)\), and the encrypted output is \(\text{Enc}(y)\), then the processing flow can be expressed as:  
\[\text{Enc}(y)=\text{Model Processing}(\text{Enc}(W),\text{Enc}(x),\text{Enc}(b)) \tag{19}\]  
where \"Model Processing\" includes encrypted linear transformations, encrypted activation functions, and encrypted attention mechanisms.