###### Abstract
Examples include error feedback, prompts, and assistance in computer operations. Additionally, a significant portion of technical details in computer-related data is articulated in English. Consequently, difficulty in accurately interpreting such content may hinder internet access and impede work efficiency. To effectively handle domain-specific tasks  
Figure 1: LLMs development process. such as computer English translation, PLMs must undergo training tailored to specific domains. Customizing models for various scenarios within the domain facilitates the development of comprehensive models tailored to specific fields. This approach ensures that PLMs possess the requisite expertise to address domain-specific challenges effectively. Extensive research has been conducted on PLMs, exploring various algorithms such as quantization, pruning, distillation, dynamic networks, data parallelism, model parallelism, and fine-tuning [10; 11; 12; 13; 14]. However, two prominent challenges persist regarding acceleration, compression, and fine-tuning techniques for large language models:  
* Many compression algorithms necessitate fine-tuning or even retraining the model post-compression. Notably, the significant challenge associated with large models lies in the considerable cost incurred by model fine-tuning or training. Consequently, several algorithms for large models are delving into approaches that circumvent the need for tuning, such as quantization and pruning. * Large models prioritize generality and generalization capabilities over performance in singular tasks. However, targeting specific domains requires a concentrated effort to align with downstream tasks more effectively. Hence, there's a growing emphasis on developing strategies that enhance alignment with specific domains while leveraging the inherent generality of large models. To address these challenges, we propose LightChatGLM, based on the ChatGLM-6B architecture, specifically tailored for training PLMs optimized for computerized English tasks. ChatGLM-6B is a bidirectional autoregressive generative model built on the Transformer architecture. It integrates the advantages of self-attention mechanisms with feed-forward neural networks. Unlike traditional unidirectional generative models, ChatGLM-6B employs a bidirectional encoder-decoder framework, enabling it to comprehend language and manage longer contexts in generation tasks efficiently. The encoding process is handled by a multilayer Transformer module, which leverages a multihead attention mechanism to weight the input sequences dynamically. With 6 billion parameters, ChatGLM-6B offers fine-grained capabilities in both language comprehension and generation [15]. However, its large scale introduces significant challenges regarding computational resources and storage requirements. To address these concerns, we introduce a hybrid compression strategy in LightChatGLM aimed at optimizing model efficiency without compromising performance. LightChatGLM integrates structured pruning and knowledge distillation with advanced techniques, such as pipeline parallelism, to create a lightweight model. This approach optimizes efficiency while maintaining excellent cross-linguistic capabilities, making it particularly effective for tasks involving computerized English. The main contributions of this paper are summarized as follows. * The introduction of a pipeline parallelism-based training method that optimally utilizes computational resources, thereby enhancing training efficiency for transformer-based basic models. * The proposal of a mixed compression method for pretrained language models that amalgamates multiple compression methodologies (each with distinct principles of operation) and capitalizes on their respective strengths. Initially, the teacher model is pruned structurally to obtain the student model. Subsequently, the student model undergoes knowledge distillation to recover lost information due to structured pruning, thereby refining its performance.