# Large Language Model Based Intelligent Fault Information Retrieval System for New Energy Vehicles
## 4 Results
### New Energy Vehicles Fault Model Training
#### 4.2.2 New Energy Vehicle Fault Classification Model Training  
Since annotated data is sufficient in the field of fault classification, and BERT models often achieve better results in NLP tasks, considering the resource consumption of model deployment, we conduct a fault text classification experiment using Long Short-Term Memory (LSTM) and BERT. Additionally, TF-IDF + Naive Bayes and TF-IDF + Ridge Regression classifiers are used as baseline comparison groups. Figure 9: BLEU-4 and ROUGE-L scores for LoRA and P-Tuning V2 models during evaluation: (**a**) BLEU-4 score of the LoRA model; (**b**) ROUGE-L score of the LoRA model; (**c**) BLEU-4 score of the P-Tuning V2 model; (**d**) ROUGE-L score of the P-Tuning V2 model. The study employs a cross-validation method with training and test sets, dividing the NEVs fault corpus into 10 parts, with 8 parts used for training and 2 parts for testing. The dataset is not shuffled. The deep-learning Transformers library provided by Hugging Face is used, and the pre-trained Chinese hfl/chinese-roberta-wwm-ext model from iFlytek is loaded for training. In the BERT-based text classification experiment, the tokenization and preprocessing step sets the maximum sequence length to 512 tokens. The experimental model includes a DistilBertModel as the pre-trained model, followed by a fully connected layer (768 to 768), a ReLU activation function, a Dropout of 0.2, and a final fully connected layer (768 to 12) for classification. The classifier layer parameters of the pre-trained model are used to initialize the final fully connected layer. The optimizer is set to AdamW with a learning rate of \(2\times 10^{-5}\). Dynamic padding is used to ensure uniform data length within each batch. During training and testing, data shuffling is enabled for training, and the last incomplete batch is discarded. In the LSTM classification experiment, the dataset was tokenized and word2 vec was used to train word embeddings, converting each question into a word vector matrix. The experiment was implemented using TensorFlow, with an optimized learning rate of 0.015, a batch size of 10, the Adam optimizer, and the Sigmoid function for final classification. The classifier's input was derived from the LSTM hidden layer output. When evaluating the performance of the trained models, this study uses accuracy, recall, and F1-score as the main evaluation metrics. The calculation formulas are as follows:  
\[P =\frac{TP}{TP+FP}\times 100\% \tag{1}\] \[R =\frac{TP}{TP+FN}\times 100\%\] (2) \[F_{1} =\frac{2\times P\times R}{P+R}\times 100\% \tag{3}\]  
TP (true positive) refers to the number of instances correctly predicted as positive, while FP (false positive) represents the number of instances incorrectly predicted as positive when they are actually negative. Similarly, FN (false negative) denotes the number of instances incorrectly predicted as negative when they are actually positive. These metrics are used to evaluate the classification performance of the model. P (precision) represents the accuracy of the model's predictions, R (recall) reflects the model's ability to retrieve relevant instances, and the F1-score, as the harmonic mean of precision and recall, provides a comprehensive assessment of the model's overall performance. To ensure the reliability of our experimental results, we calculated 95% confidence intervals for precision, recall, and F1 Score. The results indicate that the precision ranges between 72.02% and 84.59%, recall between 74.25% and 85.61%, and F1 Score between 73.13% and 85.13%. These confidence intervals confirm that the model's performance is stable and not due to random chance. The results of some fault classification algorithm experiments are shown in Table 3.  
From the analysis of the above experimental results, it can be concluded that Naive Bayes, due to its simplicity and efficiency, is usually suitable as a baseline model, while Ridge Regression performs better when dealing with fault data classification, especially in datasets with high feature correlation.