## 3 Method Design
### Instruction Tuning
\[W_{new}=W_{0}+A\cdot B \tag{2}\]where \(W_{0}\) is the original model weight, and \(A\) and \(B\) are the learned low-rank matrices used for fine-tuning. Step 3: The client performs training based on the local LLM, calculates the anomaly detection regularization term and loss function related to the global LLM, evaluates the usefulness and safety of the response, and performs backpropagation to adjust the parameters of the local model, especially updating the parameters of the LoRA part, as shown in Algorithm 1. During the backpropagation process, the gradient of the loss function concerning the new weights is calculated as follows:  
\[\
abla WL=\
abla W_{new}L\cdot(I+\frac{\partial(A\cdot B)}{\partial W}) \tag{3}\]  
where \(\
abla W_{new}L\) is the gradient of the loss function \(L\) concerning the new weights, \(I\) is the identity matrix, and \(\partial(A\cdot B)/\partial W\) is the derivative of the product of low-rank matrices concerning the original weights. ```
0: Global model parameters \(w_{glo}\), Local Dataset \(D_{i}\)
0: Local model update \(\triangle\ w_{i}\)
1: While Server returns update do
2: Receive global model parameters from Server \(w_{glo}\). 3: For number of iterations k = 1,..., K do:
4: Train model using local data D\({}_{i}\). 5: Train local model update \(\triangle\ w_{i}\) using LORA. 6: end for
7:\(Enc(pk,\ w_{i})\rightarrow\triangle\ w_{i}\). 8://Encrypt the local updates \(w_{i}\), where \(pk\) is the encryption operator. 9: Send encrypted updates \(\triangle\ w_{i}\) to Server. 10: End While
```  
**Algorithm 1** Federated Learning-based LoRA Client Training Process  
Step 4: The server aggregates the sparse adjustable layers (i.e., the unfrozen parts of the LLM) from all clients to obtain an updated global LLM, and sends the unfrozen parts of the global LLM to all clients, as shown in Algorithm 2. ```
0:\(Pre-trained\ model\ parameters\ w_{pro},number\ of\ clients\ N\)
0:Global model parameters \(w_{glo}\)
1:\(w_{glo}\gets w_{pre}\)
2:While not converged do
3: For each client \(i\) in 1 to N do
4: Send the global model parameters \(w_{glo}\) to client \(C_{i}\)
5: End For
6: For each client \(i\) in 1 to N do
7: Receive the encrypted update \(\triangle\ w_{i}\) from client \(C_{i}\)
8: if anomalydetect ( \(\triangle\ w_{i}\)) is normal then
9:\(Dec(sk,\triangle\ w_{i})\to w_{i}\). 10:\(w_{glo}=(w_{1}+w_{2}+\cdots+w_{n}\ )/\text{N}\)
11:endif
12:end for
13: CheckConvergence(\(w_{glo}\)). 14:End While
```  
**Algorithm 2** The parameter aggregation process on the LORA server based on FL  
The global loss function combines the loss from each client and the regularization term for anomaly detection:  
\[L_{global}(w_{glo})=\sum_{i=1}^{N}w_{i}L_{i}(w_{glo};D_{i})+\lambda\sum_{i=1}^{N}R (\vartriangle w_{i}) \tag{4}\]  
where \(L_{i}(w_{glo};D_{i})\) is the loss function of client \(C_{i}\), and \(R(\vartriangle w_{i})\) is the regularization term for anomaly detection. Secure aggregation is shown in Algorithm 3. ```
0:{\(\vartriangle w_{1},\vartriangle w_{2},\cdots,\vartriangle w_{n}\)}
0:Global model parameters \(w_{glo}\)
1:\(w_{sum}\gets 0\)
2:\(\vartriangle w_{glo}\gets 0\)
3: For client \(C_{i}\) to N do
4: If TrustScore(i) > threshold then
5:\(\vartriangle w_{glo}\leftarrow\vartriangle w_{glo}+\vartriangle w_{i}\). 6:\(w_{sum}\gets w_{sum}+1\)
7: end if
8: End For
9: If \(w_{sum}\) > 0 then
10:\(\vartriangle w_{glo}\leftarrow\vartriangle w_{glo}/w_{sum}\)
11: End if
12: return \(w_{glo}\)
```  
**Algorithm 3** Security Aggregation Process for LoRA Servers Based on FL  
The global loss function plays a crucial guiding role in both federated learning and the LoRA architecture, determining the direction of model parameter updates. Both the server side and the client side aim to minimize the global loss function for model optimization. Step 5: Steps 3 and 4 are repeated until the convergence criteria for federated learning are met.