## 3 Methodology
### Mixed Compression  
The computations in the transformer structure are first reorganized by the attention head, and the proposed structured pruning method is given based on this; then, the knowledge distillation process for the PLM is introduced. A typical transformer layer comprises a multihead attention (MHA) sublayer and a feed-forward network (FFN) sublayer. By segregating the computation of MHA and FFN into independent parallel tasks, computational resources can be utilized more efficiently. This restructuring of computations within the transformer layer facilitates accelerated computation, thereby enhancing the efficiency of both model training and inference processes. In the standard transformer layer, assuming that each transformer layer has \(N_{a}\) attention heads, the MHA obtains the parameters associated with the attention heads as query, \(Q\), key, \(K\), and value, \(V\), matrices by transforming the input sequence \(X\) through a linear transformation as follows:  
\[Q_{i}=XW_{i}^{Q},K=XW_{i}^{K},V=XW_{i}^{V}, \tag{6}\]  
where \(W_{i}^{Q}\), \(W_{i}^{K}\), and \(W_{i}^{V}\) are the weight matrices of the linear transformation. Then, the attention score is calculated, the Softmax function is applied to obtain the attention weights, and finally, the output for each position is obtained by weighted summation as follows:  
\[A_{i}(Q,K,V)=Softmax(\frac{Q_{i}K_{i}^{T}}{\sqrt{d_{k}}}), \tag{7}\]  
where \(d_{k}\) represents the dimension of the query or key. Then, the output of each attention head is concatenated and linearly transformed to obtain the final attention head output, i.e., the attention matrix \(A\) is weighted and summed against the value matrix \(V\) by  
\[Output_{i}=A_{i}\times V_{i}. \tag{8}\]  
Finally, the output matrix, \(Output\), of each attention header is spliced and linearly transformed to obtain the final attention header output by  
\[Output=Concat(Output_{1},Output_{2},...,Output_{N_{a}})W^{O}, \tag{9}\]  
where \(W^{O}\) is the weight matrix of the output linear transformation, and \(Concat\) denotes the splicing operation. The concept of reorganization involves treating the computation of each attention head as an individual task to be executed in parallel. Consequently, the computation of each head in the multihead attention mechanism can be decomposed into independent computing units that operate without interference, thereby achieving parallel computation. The FFN performs a linear transformation of the inputs at each position and then generates the final output by means of an activation function (e.g., ReLU) and another linear transformation. This process is applied independently to each position in the input sequence, meaning that each vector at each position undergoes the same FFN transformation. This independence allows the model to capture position-specific features, which are crucial for tasks such as language modeling and machine translation. As depicted in Figure 2, by leveraging the aforementioned formulas, if we maintain the number of attention heads, \(N_{a}\), as being unchanged while altering the dimension of the attention heads from \(d_{a}\) to \(d_{a}^{\prime}\), the model dimension will consequently become \(d^{\prime}=d_{a}^{\prime}\times N_{a}\). Similarly, if the dimension of the FFN intermediate layer, \(d_{ffn}\), is adjusted to \(d_{ffn}^{\prime}=\frac{d_{a}}{d_{a}}\times d_{ffn}\), LightChatGLM can preserve the original transformer structure parameter scale. The dashed lines in the diagram indicate the dimensions of the channels, attention heads, and feed-forward networks that are pruned in the model. As shown in Equations (6)-(9), the model determines the relative importance of each weight, neuron, convolutional kernel, and attention head during the training or validation process. Based on this importance evaluation, dimensions that contribute less to the overall performance are pruned.