# MIRA-ChatGLM: A Fine-Tuned Large Language Model for Intelligent Risk Assessment in Coal Mining
## 3 Methods
### Parameter-Efficient Fine-Tuning (PEFT)  
With the rapid development of large language models (LLMs), the scale of model parameters has been increasing significantly. However, due to limitations in computing resources and costs, it has become difficult for regular researchers to perform full-parameter fine-tuning on consumer-grade hardware. Furthermore, since the size of the fine-tuned models remains the same as the original model, storing and deploying separate fine-tuned models for each task has become prohibitively expensive. To address these issues, the parameter-efficient fine-tuning (PEFT) method was proposed [25]. This method significantly reduces computation and storage costs by fine-tuning only a small set of additional parameters while keeping the majority of pre-trained model parameters frozen. Currently, \begin{table}
\begin{tabular}{l} \hline \hline
**Example** \\ \hline Instruction: You are an expert in the field of coal mine gas risk analysis and are familiar with \\ relevant coal mine safety regulations. Please professionally and correctly answer questions about \\ coal mine safety that users want to ask. \\ Input: The name of the coal mine is xx, and the installation position is carbon monoxide in the \\ general return air lanes. The measurement type at this point is for carbon monoxide, and the \\ maximum value of carbon monoxide measurement is 511.0 ppm. What happens? What is the \\ response? \\ Output: The alarm type is calibration, the alarm cause is sensor calibration, and the treatment \\ measure is the cause: sensor calibration. \\ \hline \hline \end{tabular}
\end{table} 
Table 2: An example of instruction data. advanced PEFT techniques such as adapter-tuning, prefix-tuning, P-tuning, and LoRA have demonstrated performance comparable to full parameter fine-tuning. Adapter-tuning [26] works by integrating smaller neural network modules, known as adapters, into each layer of the pre-trained model. During the fine-tuning phase, only the parameters of these adapters are updated, while the original model's parameters remain unchanged. Prefix-tuning [27], on the other hand, introduces additional trainable prefix tokens to the model's input or hidden layers, with only these prefix tokens being fine-tuned. P-tuning [28] follows a similar approach to prefix-tuning, but instead utilizes a limited set of continuous embedding parameters as prompts to boost the performance of generative pre-trained transformers (GPT) in natural language understanding (NLU) tasks. The main distinction between the two methods is that prefix-tuning is designed for natural language generation (NLG) tasks, while P-tuning specifically targets embedding layers, without modifying other layers as prefix-tuning does. LoRA (low-rank adaptation) [29] is an efficient method for fine-tuning large-scale pre-trained models, based on the idea of introducing low-rank matrices to restrict parameter updates during fine-tuning to a low-rank subspace, thereby reducing the number of parameters that need to be updated for efficient fine-tuning. Traditional fine-tuning methods require updating all parameters of the model, whereas LoRA reduces the number of parameters to be updated during fine-tuning by introducing a low-rank matrix, thus lowering computational and storage overhead. During training, only the parameters of the low-rank matrix are optimized. For a linear layer \(h=Wx\), the forward propagation is replaced by the following equation:  
\[h=Wx+BAx, \tag{1}\]  
where \(W\in R^{d\times d},A\in R^{d\times r},B\in R^{r\times d}\), with rank \(r\ll d\). Matrix A is initialized using a random Gaussian distribution, whereas matrix B starts with all zeros, ensuring that only the main path is active during the initial phase. The data flow in the forward propagation of LoRA is shown in Figure 3.  
QLoRA [30] builds upon LoRA with further optimizations, employing quantization techniques to reduce model memory usage and computational complexity (Figure 4).