## 3 Results
### Expanded Lexicon  
Fine tuning pre-trained models for specific domains is crucial to maximizing their effectiveness. Pre-trained models acquire substantial basic semantic knowledge from large volumes of unlabeled data, and fine tuning with domain-specific data enables more accurate extraction of semantic information relevant to a particular domain. Figure 6 illustrates the comprehensive process employed in this study, from pre-training to fine tuning, and finally to application, using the BERT model. During the pre-training phase, the model initially undergoes language modeling through self-supervised learning. Specifically, the \"Whole Word Masking\" (WWM) strategy is applied to mask Chinese text, ensuring that entire words, rather than individual characters, are masked simultaneously. Upon completing the pre-training, the model is fine tuned for specific downstream tasks, with its parameters being adjusted to enhance performance on these tasks. In our research, we fine tuned the Chinese-RoBERTa-WWM-Ext model [49]. This model is configured with key parameters, including a hidden layer size of 768, 12 hidden layers, and 12 attention heads, equipping it with the capacity to discern nuanced linguistic relationships. It addresses nonlinear complexities using the GEL  
Figure 6: Chinese-RoBERTa-WWM-Ext model architecture and fine-tuning steps. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Candidate Words** & **TF-IDF Values** & **Candidate Words** & **TF-IDF Values** & **Candidate Words** & **TF-IDF Values** \\ \hline Construction & 0.87825 & Valve & 0.60116 & Rescue & 0.55138 \\ \hline Operation & 0.77903 & Engineering & 0.58870 & Organization & 0.55049 \\ \hline Liquefied & & & & & \\ Petroleum Gas & 0.77732 & Excavation & 0.58689 & Department & 0.54887 \\ \hline Pipeline & 0.76862 & Excavator & 0.58360 & Plan & 0.53793 \\ \hline Production & 0.75127 & Emergency & 0.58144 & Construction Site & 0.53704 \\ \hline Natural Gas & 0.74487 & Inspection & 0.57918 & Safety & 0.53486 \\ \hline Gas Pipe & 0.72926 & Deflagration & 0.56927 & Disposal & 0.53117 \\ \hline Pipeline & 0.71646 & Management & 0.56493 & Connection & 0.49896 \\ \hline Installation & 0.61668 & Explosion & 0.55686 & Pressure & 0.45071 \\ \hline Underground & 0.60599 & Construction & 0.55518 & Gas & 0.42280 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Examples of candidate words and corresponding TF-IDF values. implemented a dropout rate of 0.1 to reduce overfitting and set the maximum positional embeddings at 512, allowing for the processing of extended text sequences. The model employs absolute positional embeddings to enhance its comprehension of sequence positioning. The minimum setting for layer normalization is established to ensure numerical stability. Throughout the training and inference phases, features such as outputting historical states and caching capabilities were activated to augment training efficiency and accelerate inference rates. These carefully calibrated parameters facilitate a deeper understanding and generation of language, significantly bolstering support for natural language processing tasks. The model completed all 82,524 training steps by the 6 th cycle, with each cycle lasting approximately 6 h, 20 min, and 12 s. Figure 7 shows the model's training loss and trend line, illustrating a pronounced downward trend throughout the fine-tuning process. This pattern signifies the model's adaptation to the training data, as evidenced by the continual decline in loss values. Initially, the training loss was 1.4 but quickly dropped below 1.0 after a sharp early decrease. The greatest loss reduction occurred within the first 10,000 steps of early training. During this period, the model rapidly adjusts to the training dataset from its initial random weight configurations. As training progresses, the reduction in loss slows, suggesting the model is approaching its performance limits after learning simpler patterns. Beyond approximately 20,000 steps, the loss curve stabilizes between 0.8 and 1.0.