## 4 Experimental Evaluation
### Experiment Analysis  
Figure 3 illustrates the variation in training time per epoch for the transformer base model across different sample block sizes and a comparison of various parallelization strategies. As the sample block size increases, the training speed also increases, with the optimal speed achieved when the sample block size is set to 256. Additionally, the use of different parallelization strategies can significantly reduce the training time per epoch. LightChatGLM leverages pipelined parallelism to facilitate efficient distributed training across multiple GPUs, resulting in a minimum training time of 180 s per epoch. However, the higher communication overhead associated with data and model parallelism slightly extends the training duration. It is also important to consider that memory usage escalates substantially as the sample block size increases, necessitating the careful design of the sample block size to achieve optimal training acceleration. This training speedup effect is observed in several training sets, including WMT 2020, UM-Corpus, and Ai Challenger. Experiments on these datasets show that the training speedup is, indeed, the result of the combined effect of pipeline parallelization and reasonable sample block division. Figure 4 illustrates the validation loss of the mixed compression method compared with random pruning over an equal number of training rounds. We trained the base LightChatGLM model on the UM-Corpus dataset using the full dataset and parameter set to establish the initial validation loss. Subsequently, the model was subjected to mixed compression and random pruning to derive the corresponding student models, with the validation loss computed on the validation set. As shown in Figure 4, the mixed compression approach, which integrates structured pruning and knowledge distillation techniques, \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Name**} & **Chinese** & **English** & **Training Set** & **Development** & **Test Set** \\  & **Words/Million** & **Words/Million** & **Size** & **Set Size** & **Size** \\ \hline WMT 2020 Chinese-English [65] & 130 & \(110\) & 90\% & 5\% & 5\% \\ UM-Corpus [66] & 50 & 48 & 80\% & 10\% & 10\% \\ Ai Challenge [67] & 250 & 200 & 80\% & 10\% & 10\% \\ IWSLT 17 [68] & 0.8 & 1 & 85\% & 5\% & 10\% \\ XGLUE 20 [69] & 0.9 & \(1.1\) & 80\% & 10\% & 10\% \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Datasets. achieves higher compression efficiency while preserving model performance. The validation loss curves for the mixed compression method typically exhibit faster convergence and stabilize at lower loss values. In contrast, random pruning methods, which achieve compression by randomly removing certain weights from the model, may cause greater fluctuations in model performance. As a result, the validation loss curves for random pruning tend to converge more slowly and stabilize at relatively higher values. The experimental data demonstrate that the mixed compression method outperforms random pruning in both convergence speed and final validation loss, effectively maintaining model performance. Table 2 offers a comprehensive comparison of the accuracy achieved in computerized English-Chinese translation without the application of target data for fine-tuning and knowledge distillation. A noteworthy observation from the table is that the absence of utilizing rich source-language annotated data for cross-language knowledge migration results in the diminished generalization ability of the compressed model. Among the various methods evaluated, the DistilmBERT approach yields the most favorable outcomes,  
Figure 4: Validation loss over training steps for mixed compression and random pruning. Figure 3: Effect of different parallel strategies on training time with different batch sizes. with 46.3% in Chinese-English bilingual translation.