## 3 Methodology
### Mixed Compression
The dashed lines specifically denote the pruned dimensions within the attention heads and the feed-forward network sublayers, thereby simplifying the model's structure. Additionally, a subset of the transformer layer is selected from the ChatGLMmodel to undergo structural pruning in the depth direction. By employing pruning in both the depth and dimension directions, LightChatGLM can obtain a student model with reduced layers and smaller dimensions, initialized by the teacher model parameters. After deriving the student model through structured pruning, LightChatGLM proceeds with knowledge distillation to facilitate the migration of bilingual knowledge from the ChatGLM model to the student model. This step aims to help the student model recover the information lost due to structured pruning. Drawing from the knowledge distillation approach employed in the task-independent phase of the TinyBERT framework [60], LightChatGLM adopts a distinct layer mapping strategy. Referencing the existing literature reveals [61; 62] that the last layer strategy proves more beneficial for the student model compared with the average strategy utilized in the original TinyBERT framework during the task-independent knowledge distillation phase of the pretraining process. Consequently, LightChatGLM opts to employ the last layer for distillation instead of distilling all hidden layer knowledge from the teacher model. Assuming that the student model and the teacher model consist of \(M_{s}\) and \(N_{t}\) layers of the transformer, respectively, and the last hidden state of the teacher model is denoted as \(H_{t}^{N_{t}}\), we utilize \(H_{t}^{N_{t}}\) as a soft label to guide the training of the student model. For each training sample, the input data are forwarded through the first \(M_{s}\) layers of the student model to obtain the last hidden state \(H_{s}^{M_{s}}\) of the student model. Then, the squared loss function is employed to measure the loss of the distillation process as follows:  
\[L_{KD}=\frac{1}{2}\sum_{i,j}(H_{t}^{N_{t}}(i,j)-H_{s}^{M_{s}}(i,j))^{2}, \tag{10}\]  
where \(H_{t}^{N_{t}}(i,j)\) and \(H_{s}^{M_{s}}(i,j)\) denote the \(j\)-th feature of the \(i\)-th sample of the hidden state in the last layer of the teacher model and the student model. The parameters of the student model are optimized by minimizing the loss function \(L_{KD}\) so that the student model can closely approximate the output of the teacher model. By using the last layer of the knowledge distillation strategy, LightChatGLM successfully transfers bilingual knowledge from the teacher model to the student model, enabling the student model to recover the information lost due to structured pruning. This strategy is particularly effective in task-independent knowledge distillation scenarios during the pretraining phase, where the final layer strategy proves more beneficial to the student model compared with distilling all the hidden layers of the teacher model. The whole process is shown in Algorithm 2. The trained teacher model undergoes structural pruning in both the depth and gradient directions, as illustrated in lines 1-6 of Algorithm 2. Subsequently, knowledge distillation is carried out to  
Figure 2: Structured pruning strategy. further transfer bilingual knowledge from the teacher model to the student model, enabling the student model to recover information lost during structural pruning, as outlined in lines 7-14 of Algorithm 2. Ultimately, the compressed student model is obtained.