## 3 Methodology
### Hybrid Data Fine-Tuning  
By leveraging structured pruning to initialize the student model and subsequently transfer cross-lingual knowledge from the teacher model through knowledge distillation, LightChatGLM achieves a streamlined model with cross-lingual capabilities. To enhance the student model's understanding of computer English, LightChatGLM continues with cross-language knowledge distillation on downstream tasks to further refine the model's performance in computer languages. Specifically, LightChatGLM aims to refine the teacher model through fine-tuning to enhance its cross-linguistic competence. Then, this competence is transferred to the student model through cross-language knowledge distillation on downstream tasks. This section introduces a hybrid data-based fine-tuning method wherein annotated data from both the source and target languages are randomly mixed for fine-tuning the teacher model. This approach enables the teacher model to autonomously learn the connections and similarities between languages during training, thereby improving its performance in the target language with the aid of rich source language knowledge. We prepared the labeled datasets for the source and target languages as \(D_{s}\) and \(D_{d}\), respectively. These datasets contain training samples for the target task along with their corresponding labels. We randomly mixed the labeled data from \(D_{s}\) and \(D_{d}\) to create a hybrid dataset. In the mixed-data fine-tuning process, a balancing parameter, \(\alpha\), is required to appropriately weigh the contributions of the source and target languages. The value of \(\alpha\) can be determined based on the relative sizes of the source and target language datasets, as well as their significance to the target task. This relationship can be expressed by the following formula:  
\[\alpha=\frac{w_{s}\cdot n_{s}}{w_{s}\cdot n_{s}+w_{d}\cdot n_{d}}, \tag{11}\]  
where \(w_{s}\) and \(w_{d}\) represent the weighting coefficients for the source and target languages, which are usually based on task-relevant performance metrics, such as translation accuracy or cross-language comprehension. The terms \(n_{s}\) and \(n_{d}\) denote the number of samples in the source and target language datasets. Furthermore, the teacher model is adapted to the characteristics of the target task, and its performance is improved by performing backpropagation and parameter updating on the hybrid data. The performance of the teacher model on the mixed data can be measured using the cross-entropy loss function. Specifically, for each sample (\(D_{s}(i),D_{d}(j)\)), the final objective function can be expressed as  
\[L_{hybrid}=-\frac{1}{N}\sum_{i=1,j=1}^{N}(\alpha\cdot log(f_{ teacher}(D_{s}(i)))+(1-\alpha)\cdot log(f_{teacher}(D_{d}(j)))), \tag{12}\]  
where \(N\) is the total number of samples in the hybrid dataset, \(\alpha\) is a balancing parameter between the source and target language samples, and \(f_{teacher}\) represents the prediction function of the teacher model. During the fine-tuning process with random mixing, LightChatGLM continues to perform knowledge distillation to transfer the cross-linguistic competence of the teacher model to the student model, thereby enhancing the performance of the student model on the target language. The loss function for knowledge distillation on the downstream task is then formulated as follows:  
\[L_{distill}=-\frac{1}{M}\sum_{j=1}^{M}(\lambda\cdot log(f_{teacher}(D_{d}(j)))+ (1-\lambda)\cdot log(f_{student}(D_{d}(j)))), \tag{13}\]  
where \(M\) is the total number of samples in the target language dataset, and \(f_{student}\) represents the prediction function of the student model. The balance parameter \(\lambda\) is used to control the weighting between the teacher model predictions and the student model predictions, and typically, the value of \(\lambda\) can be adjusted between 0 and 1.