## 4 Experimental Evaluation  
### Experiment Setup  
Hardware environment: The server we used is equipped with an Intel(R) Xeon(R) Silver 4210R CPU at 2.40 GHz and NVIDIA Tesla T4 GPUs with 16 GB of RAM each, interconnected via PCIe-III. The server runs on a 64-bit Ubuntu 20.04 system with CUDA toolkit version 10.2 and PyTorch 1.10.2.  
Hyperparameter setting: LightChatGLM utilized Wikipedia pages containing computer-specific English and Chinese nouns as the training corpus. These corpora were randomly mixed together. The hyperparameters were set as follows: a batch size of 256, a maximum sequence length of 128, a dropout rate of 0.1, a parameter decay of 0.05, and 400,000 steps for model parameter updates. The learning rate was initially set to 0.9 and decayed after the first 10% of the update steps. Based on these initial parameters, we performed a hyperparameter search to further optimize the model's performance. Hyperparameter search is usually performed using grid search or random search, combined with cross-validation, to select the optimal combination of parameters to enhance model performance for a specific task [49; 63; 64].  
For the downstream multilingual task, the datasets we used are shown in Table 1. The WMT 2020 Chinese-English comprises both Chinese and English segments from the Chinese-English translation tasks of the WMT competition, serving as a benchmark dataset for bilingual machine translation tasks. The UM-Corpus, developed by the University of Macau, is primarily used for high-quality research on Chinese-English parallel corpora, making it suitable for machine translation and semantic understanding tasks. The Ai Challenger encompasses Chinese-English parallel corpora for various tasks and is an essential resource for research in natural language processing and machine translation. IWSLT 17 is mainly used for multilingual spoken translation tasks, especially Chinese-English translation tasks. XGLUE is an evaluation benchmark for cross-language natural language understanding launched by Microsoft, which includes multiple tasks, including translation, question answering, and text classification.  
During fine-tuning or the distillation of the student model for this task, the hyperparameters were adjusted accordingly. Specifically, the maximum sequence length was set to 128, the batch size to 32, and the balance factors \(\alpha\) and \(\lambda\) were both set to 0.65. Additionally, the smoothed logit temperature value was set to 1.  
Comparison models: we evaluate the performance of LightChatGLM by comparing it with the following typical schemes:  
* mBERT_drop [70]: mBERT is a multilingual BERT model pretrained for 104 languages, featuring the same structure as the original BERT model. Meanwhile, mBERT_drop represents a compression technique specifically designed for mBERT, involving the direct pruning of the top transformer layer of the mBERT model.
* DistilmBERT [71]: The multilingual version of DistilBERT is a pretraining model that employs knowledge distillation techniques to decrease the size and enhance the speed of the BERT model. The concept behind its design is straightforward: construct a smaller model, referred to as DistilBERT, as the student model and utilize the original BERT model as the teacher model. The goal is for the student model to learn from the teacher model as much as possible, thereby retaining the reasoning capabilities of the teacher model to the fullest extent possible.
* ChatGLM-6B [72,73]: ChatGLM-6B is an open-source, bilingual conversation language model built on the GLM architecture. By leveraging model quantization technology, it demands as little as 6 GB of video memory when operating at the INT4 quantization level.