# Enhancing Privacy in Large Language Model with Homomorphic Encryption and Sparse Attention
## 3 Materials and Methods
### Dataset Preprocessing  
Data preprocessing represents a crucial step within the machine learning workflow, involving the transformation of raw data into a format comprehensible by models. In this study, preprocessing encompasses the following steps: (1) Data cleaning involves the removal of noise and irrelevant information from the dataset, such as eliminating HTML tags, URLs, user tags, and extraneous punctuation marks. (2) Text normalization converts the text into a uniform format, including transforming all letters to lowercase, expanding abbreviations to their full forms, and replacing special characters. (3) Tokenization processes the text by breaking down continuous strings into meaningful units (words, punctuation). (4) Vocabulary construction entails building a vocabulary from the tokenized results, where each word is assigned a unique index. (5) Text encoding transforms the text into a numerical form that can be processed by the model, usually by replacing each word with its corresponding index or word vector. (6) Sequence padding or truncation ensures that all sequences are of consistent length to enable the model to process inputs of varying lengths. (7) Construction of homomorphic encryption-compatible formats: given the study's involvement with homomorphic encryption, the numerically formatted text must be further transformed into a format suitable for homomorphic encryption operations. The mathematical principles underlying preprocessing are primarily concerned with text encoding and sequence handling. During text encoding, word embeddings transform discrete words into points in a continuous vector space, capturing semantic relationships between words. Mathematically, word embeddings can be realized through an embedding matrix \(E\), where each row of \(E\) corresponds to a word in the vocabulary. For a word \(w\), its encoded vector is obtained by looking up the embedding matrix:  
\[v_{w}=E_{\text{index}(w)} \tag{14}\]  
Sequence padding or truncation involves adjusting all text sequences to the same length \(L\). For sequences shorter than \(L\), padding tokens (such as 0) are added to the end to reach length \(L\); for sequences longer than \(L\), the excess is truncated. Let \(S\) be the original sequence and \(S^{\prime}\) the processed sequence, then this is mathematically expressed as:  
\[S^{\prime}= \begin{cases}S\oplus[0]_{L-|S|},&\text{if }|S|<L\\ S[:L],&\text{if }|S|\geq L\end{cases}  \tag{15}\]where \(\oplus\) represents the sequence concatenation operation, and \([0]_{L-|S|}\) denotes a padding sequence of length \(L-|S|\). Preprocessing is essential for the tasks addressed in this study, aimed at enhancing model performance and generalizability. The importance of preprocessing is elucidated as follows:  
1. Data Quality: High-quality data are pivotal to the performance of models. By cleaning and normalizing text, noise that could confuse the model is removed, thus improving data quality. 2. Computational Efficiency: By padding and truncating sequences, a uniform shape for all data during computation is ensured, which is beneficial for parallel processing by hardware such as GPUs and simplifies model design. 3. Model Training: Proper preprocessing can enhance the efficiency and stability of model training. For instance, uniform text encoding and length can accelerate the convergence of gradient descent. 4. Homomorphic Encryption Compatibility: In the application of homomorphic encryption, preprocessing data into a compatible format is a prerequisite for encrypted computations. Due to the constraints typically associated with homomorphic encryption operations, preprocessing steps must be specially designed to accommodate these constraints, thereby ensuring the model functions correctly on encrypted data. In summary, data preprocessing not only impacts the training efficiency and final performance of the model but also plays a key role in protecting user privacy.