## 2 Related Work
### Compression Techniques  
PLMs are known for their emphasis on generality, generalization capabilities, and even emergence across a wide array of tasks and unseen data. Consequently, compressed PLMs must undergo careful verification to ensure their retention of generality and generalization capabilities. In response to these challenges, various compression methods have been proposed specifically tailored for PLMs. Common compression techniques include quantization, pruning, knowledge distillation, and dynamic networks. Quantization refers to the process of converting a model's parameters and/or computations from high-precision representation to lower-precision formats. This technique is commonly used to reduce the computational resources required for running models and improve their efficiency, especially in deployment scenarios where hardware constraints are a concern [26; 27; 28; 29; 30; 31]. These include methods that do not necessitate retraining, such as post-training quantization (PTQ), and methods that do require retraining, such as quantization-aware training (QAT). PTQ methods circumvent the need for an expensive retraining process, making them a more viable direction for most researchers. Pruning is a technique used to reduce the size and complexity of a neural network by removing certain elements, such as weights, neurons, or entire layers. This helps in making the model more efficient without significantly sacrificing performance. However, its effectiveness can be undermined during the fine-tuning phase, which is often costly, particularly for models with a large number of parameters. Despite these challenges, pruning remains a critical technique for model compression, deserving further investigation to improve its application in PLMs. Notable unstructured pruning methods, such as SparseGPT [32] and Wanda [33], have set benchmarks for subsequent approaches. Wanda, for instance, introduces a unique pruning metric that accounts for both weight magnitude and activation values, while another method, relative importance and activation (RIA), addresses channel corruption by clipping entire rows and columns from the weight matrix. To further enhance pruning effectiveness in PLMs, various auxiliary techniques have been developed, including region-specific sparsity rates [34; 35], post-pruning fine-tuning strategies [36; 37; 38], and hardware optimizations [39; 40]. Despite advancements, significant challenges remain, particularly in integrating pruning with other techniques and addressing the high cost of fine-tuning. Knowledge distillation (KD) is a technique in machine learning where a smaller, more efficient model (often referred to as the \"student\") is trained to replicate the performance of a larger, more complex model (the \"teacher\") [41]. The goal is to transfer the knowledge from the teacher model to the student model, enabling the student to achieve similar performance while being more computationally efficient and faster to deploy [42]. To improve the effectiveness of distillation, various enhancement strategies have been proposed, such as multitask learning and tailored dataset distillation. For instance, DISCO leveraged a pretrained language model (PLM) to generate counterfactual data, which were subsequently filtered using a large-scale teacher model for natural language inference tasks [43]. Similarly, PubMedBERT is tailored to handle the nuances and specialized terminology of biomedical text, which makes it particularly useful for tasks such as biomedical information extraction, literature mining, and question-answering within the medical domain [44]. PromptMix is a technique designed to enhance the performance of language models by combining multiple prompts in a structured way. It leverages the strengths of different prompts to provide more comprehensive and contextually relevant responses [45].