# LLM-Enhanced Framework for Building Domain-Specific Lexicon for Urban Power Grid Design
## 3 Results
### Analysis of Dictionary Effectiveness and Sensitivity  
To validate the effectiveness of the domain-specific lexicon built in this study, we evaluated the quality of the dictionary based on the accuracy of domain document recognition. The experimental dataset used in this section consists of 1000 corpus entries related to power design (independent of the corpus described in Section 2.1). The evaluation was conducted by comparing the performance of a Support Vector Machine (SVM) model with and without loading the domain dictionary into the custom dictionary of the Jieba tokenizer. The specific experimental steps are as follows:  
(1) The domain dictionary was loaded into the custom dictionary of Jieba for the experimental group, while the control group did not include the domain dictionary. Both groups underwent word segmentation and stop word removal processes, following the same procedures outlined in Section 2.1.  
(2) The dataset was split into a training set (700 entries) and a test set (300 entries) at a ratio of 7:3.  
(3) The TF-IDF algorithm was applied to construct a text feature matrix for the corpus, which was then input into the SVM model for training to generate a text classifier.  
(4) The test set was used to evaluate the performance of the classifier, with accuracy, recall, and F1-score serving as the evaluation metrics.  
The experimental results demonstrate that the performance of the text classifier significantly improved after loading the domain dictionary. The detailed results are presented in Table 5.  
It can be observed that after loading the domain-specific dictionary, the classifier's accuracy improved by 8.6%, recall by 9.8%, and F1-score by 9.2%. This confirms the effectiveness of the domain dictionary in enhancing the accuracy of text classification.  
Figure 5: Histogram of Semantic Cosine Similarity.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Lexicon** & **P/\%** & **R/\%** & **F1/\%** \\ \hline Loading & 91.0 & 95.3 & 93.1 \\ Unloading & 82.4 & 85.5 & 83.9 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Evaluation Results of Text Classifier.  
To further validate the rationality of the number of seed vocabulary, we adjusted the quantity of seed vocabulary and monitored changes in the F1-score. The specific experimental design is as follows:  
We adjusted the high-frequency terms W(t) \(\geq\) 0.85 \(\times\) max(W) obtained from the TF-IDF algorithm results described in Section 2.2. For the control group, we experimented with different thresholds: 0.8, 0.825, 0.875, and 0.9. These thresholds respectively resulted in seed vocabularies of sizes 4,567, 3,996, 2,854, and 2,284. At each level of seed term quantity, we repeated the aforementioned text classification experiments, recording Precision (P), Recall (R), and F1-score. The experimental results are illustrated in Figure 6.  
The F1-score peaked at 93.1% when the seed vocabulary's total was 3425. Increasing the frequency coefficient to reduce the number of seed vocabulary led to a significant decline in the F1-score (dropping to a minimum of 91.1%), indicating inadequate vocabulary coverage. Conversely, decreasing the frequency coefficient to increase the seed vocabulary's total resulted in the stabilization of the F1-score. It is evident that the current seed vocabulary total (3425) achieves an optimal balance in the F1-score, ensuring adequate dictionary coverage while avoiding excessive consumption of computational resources.  
Through text classification experiments and sensitivity analysis, we have validated the effectiveness of the domain-specific dictionary constructed in this study for improving the accuracy of document recognition in the field of power design. Additionally, the sensitivity analysis demonstrates that the current seed vocabulary total (3426) achieves an optimal balance in the F1-score, providing strong support for the practical application of the lexicon in engineering contexts.