# Research on Large Language Model for Coal Mine Equipment Maintenance Based on Multi-Source Text
## 3 Methodology
### Large Language Model Construction Based on Triple-LoRA Fine-Tuning and DPO
#### 3.3.2 Triple-LoRA Fine-Tuning Mechanisms  
To ensure the specialization and task-oriented function of a coal mine equipment maintenance LLM, a multi-expert fine-tuning strategy was proposed. Low rank adaptive (LoRA), a mechanism for effective parameter fine-tuning [32], is used to train individual components of the model on a specialized dataset. They can operate autonomously without interfering with each other. LoRA applies adaptation to the pre-trained language model by introducing a low-rank matrix in its weight matrix. The pivotal steps in the LoRA adaptation process are as follows  
1. Initialization: Start with a pre-trained language model and add a low-rank adaptation layer to its weight matrix array. This layer is represented by a low-rank matrix that is initialized to random values. 2. Fine-tuning: Train the model on a new task or domain, updating only the low-rank adaptation layer while keeping the weights of the pre-trained model fixed. This allows the model to learn task-specific information in a targeted way without changing its general knowledge. 3. Prediction: Use the fine-tuned model to make predictions about new, unseen data from the target task or domain. LoRA indirectly trains some of the dense layers in a neural network. Afterward, LoRa employs trainable rank-decomposition matrices at each layer of the transformer architecture, thereby significantly reducing the required number of training parameters for downstream tasks. For the pre-trained weight matrix, LoRA uses low-rank decomposition, indicating the latter to constrain its updates. \[W_{0}+\Delta W=W_{0}+BA \tag{2}\]  
where \(A\in\mathbb{R}^{d\times k}\), \(B\in\mathbb{R}^{d\times k}\) and \(r\ll\min(d,k)\). During training, \(W_{0}\) is frozen and does not receive gradient updates, but \(A\) and \(B\) contain trainable parameters. Note that both \(W\) and \(\Delta W=BA\) both multiply with the identical inputs, and their corresponding output vectors are added in the coordinate direction. The original forward propagation is \(h=W_{0}x\) The modified forward propagation is:  
\[h=W_{0}x+\Delta Wx=W_{0}x+BAx \tag{3}\]  
Figure 5: Autoregressive blank infilling objective and self-attention masking. **(a)** The original text is sampled; **(b)** replace the sampled spans with \([M]\) in Part A and shuffle the spans in Part B; **(c)** autoregressively generates Part B; **(d)** self-attention mask. The gray areas are obscured. Part A tokens can focus on themselves (blue frame), but not on Part B. Part B marking can focus on A and the antecedent in B (yellow and green frames correspond to two spans). \([M]\) stands for [MASK], \([S]\) stands for [START], \([E]\) stands for [END]. Ultimately, the inference speed is increased and the inference delay is reduced. Specifically, we trained the three LoRA modules with fine-tuning of the base model for the corresponding part of the instruction. Switching between different functions during deployment only requires replacing the LoRA parameters loaded on the current base model. This eliminates the need to retrain the entire model. In addition, this approach improves the efficiency of the model for each mission. In this paper, three modules were set: LoRA for coal mine equipment dialog consultation, LoRA for coal mine professional advisory service, and LoRA for maintenance program analysis and decision. The triple LoRA mechanism is illustrated in Figure 6.  
Coal mine equipment dialog consulting: The first LoRA model was designed to address the problems and consulting dialog challenges in the context of coal mine equipment including the basic knowledge of coal mine equipment such as the development of coal mine equipment and the introduction of fully mechanized coal mining equipment like related structures and components. It mainly provides basic dialog and consultation services.