# MIRA-ChatGLM: A Fine-Tuned Large Language Model for Intelligent Risk Assessment in Coal Mining
## 3 Methods
### Evaluation Metrics  
This study employs metrics such as bilingual evaluation understudy (BLEU) [31] and recall-oriented understudy for gisting evaluation (ROUGE) [32] to assess the degree of match between candidate texts and reference texts. We use metrics like Predict_runtime, Predict_samples_per_second, and Predict_steps_per_second to evaluate the inference speed of the fine-tuned models. These metrics enable a comprehensive assessment of the model's performance in terms of accuracy, fluency, and information completeness, thereby guiding model improvement and optimization.  
BLEU is a metric used to assess the similarity between two texts. The formula for calculating BLEU is as follows:  
\[BLEU=BP\cdot exp(\sum_{n=1}^{N}w_{n}\log p_{n}), \tag{2}\]  
where BP is the length of the penalty term, called the brevity penalty, defined as follows:  
\[BP=\begin{cases}1,&if\ c>r\\ exp\big{(}1-\frac{r}{c}\big{)}&if\ c\leq r\end{cases}, \tag{3}\]  
In Equation (3), c is the length of the generated text and r is the length of the reference text.  
\[P_{n}=\frac{\sum_{i}^{E}\sum_{k}^{K}\text{min}\bigg{(}h_{k}(c_{i}),\underset {k\in M}{\text{min}}h_{k}(s_{i};j)\bigg{)}}{\sum_{i}^{E}\sum_{k}^{K}\text{min} (h_{k}(c_{i}))}, \tag{4}\]  
where E denotes the total number of candidate texts and K denotes the total number of word groups. \(h_{k}(c_{i})\) denotes the frequency of occurrence of the kth word group in the candidate text \(c_{i}\). \(s_{i}\) denotes the reference, where j \(\in\) M, and M denotes the number of reference answers. \(h_{k}(s_{i};j)\) denotes the frequency of occurrence of the kth word group in the standard answer \(s_{i,j}\).  
Figure 4: Different tuning methods have varying memory requirements. QLoRA improves upon LoRA by quantizing transformer models to 4-bit precision and employing a paging optimizer to manage memory peaks.  
In Equation (2), \(w_{n}\) is the weight, usually using equal weight for each n-gram, \(w_{n}=\frac{1}{N}\), where N is the order of the largest n-gram (e.g., N = 4 represents 1-g to 4-g).  
ROUGE (recall-oriented understudy for gisting evaluation) is a metric used to evaluate the quality of automatic text summarization, machine translation, and text generation. It primarily assesses quality by comparing the overlap between system-generated texts and reference texts. Unlike BLEU, ROUGE focuses more on recall, which is the proportion of words or n-grams in the generated text that match those in the reference text. The calculation formula for ROUGE is as follows:  
\[ROUGE-N=\frac{\sum_{\mathcal{S}\in\{Reference\ Summaries\}}\sum_{\mathit{ gram}_{n}}Count_{match}(gram_{n})}{\sum_{\mathcal{S}\in\{Reference\ Summaries\}}\sum_{\mathit{gram}_{n}}Count(gram_{n})}, \tag{5}\]  
where N denotes the length of N-grams, and \(Count_{match}(gram_{n})\) represents the maximum number of times the n-gram occurs in both the candidate text and the reference text. ROUGE-1 evaluates single-character matches, ROUGE-2 focuses on two-character matches, ROUGE-L identifies the longest common subsequence, and so on.  
Predict_runtime represents the total time taken by the model to generate a batch of samples; Predict_samples_per_second indicates the number of samples the model can generate per second, commonly used to evaluate the model's inference speed; and Predict_steps_per_second denotes the number of steps the model can execute per second.