# Enhancing Privacy in Large Language Model with Homomorphic Encryption and Sparse Attention
## 2 Related Work
### Attention Mechanism  
The attention mechanism is a significant innovation in the field of deep learning in recent years, simulating the mechanism of human visual attention: rather than passively receiving all visual information when observing the surrounding world, attention is selectively focused on certain parts according to the current task [4; 25; 26]. In computer vision and NLP tasks, the attention mechanism allows the model to dynamically focus on important parts of the input data, significantly improving the efficiency and effectiveness of information processing. Mathematically, the attention mechanism can be viewed as a mapping function that maps a set of queries, keys, and values to an output. Queries are usually related to the current task or target state, while keys and values correspond to different parts of the input data. The attention function computes the similarity between each key and the query,generating a weight distribution that is used to weight the corresponding values to form the output. The mathematical formula can be expressed as:  
\[\text{Attention}(Q,K,V)=\text{softmax}\bigg{(}\frac{QK^{T}}{\sqrt{d_{k}}}\bigg{)}V \tag{8}\]  
where \(Q\), \(K\), and \(V\) are the matrix representations of queries, keys, and values, respectively; \(d_{k}\) is the dimension of the key vectors, used to scale the dot product to prevent it from becoming too large. In computer vision, the attention mechanism can be used for image classification, object detection, and image segmentation tasks. For example, in image classification, attention can enable the network to focus on the most informative regions of the image for the classification task. For a convolutional neural network (CNN), attention weights \(\alpha\) can be applied to the feature maps to enhance or suppress features in certain areas, as follows:  
\[F^{\prime}=\alpha\odot F \tag{9}\]  
Here, \(F\) is the original feature map, \(\odot\) represents element-wise multiplication, and \(F^{\prime}\) is the feature map after attention weighting. In NLP, the attention mechanism is widely used in machine translation, text summarization, question-answering systems, and other tasks. In machine translation, sequence-to-sequence (Seq2 Seq) models often use an encoder-decoder architecture, where the attention mechanism helps the decoder focus on the parts of the encoder that are most relevant to the word currently being generated. If \(h_{t}\) represents the hidden state of the decoder at time step \(t\), and \(s_{i}\) represents the hidden state sequence of the encoder, attention weights can be calculated as:  
\[\alpha_{t,i}=\frac{\text{exp}(\text{score}(h_{t},s_{i}))}{\sum_{j}\text{exp}( \text{score}(h_{t},s_{j}))} \tag{10}\]  
Then, the current hidden state of the decoder can be updated using the weighted sum of the encoder's hidden states:  
\[h_{t}^{\prime}=\sum_{i}\alpha_{t,i}s_{i} \tag{11}\]  
In the task of this paper, namely a privacy-protected dialogue large model framework based on FHE and attention mechanisms, the attention mechanism can help the model focus computational resources on the most critical information for the current task when processing encrypted data. If \(c_{h_{t}}\) and \(c_{h_{j}}\) are the homomorphically encrypted hidden states, it is necessary to calculate their homomorphic encrypted attention weights \(c_{a_{ij}}\), which can be achieved through the homomorphic encryption version of the dot product and normalization functions. In a homomorphic environment, this computation process requires special design because the traditional softmax function involves exponentiation and division operations, which are usually not directly feasible on homomorphic encryption.