# Candidate Term Boundary Conflict Reduction Method for Chinese Geological Text Segmentation
## 4 Experiments and Discussion
### Experiment and Results  
Before the start of word segmentation, the activity of single characters should be quantified to avoid the manual definition of stop words. The information entropy of single characters for different texts will be calculated and then different proportions of single characters are selected according to the descending order of information entropy. Then, the appropriate proportion of the single characters in the full text can be compared and concluded. Finally, the top 1% of all single characters is selected as the optimal proportion of stop words. The examples of some stop words for a text are extracted, such as \"De (of)\", \"Wei (for)\", \"He (and)\", \"Yu (with)\", \"Huo (or)\", \"Zai (on)\", \"Shi (is)\", and \"Yi(by)\". By verifying the performance of the HFCR algorithm in different fields and different texts, this study designed two experiments to compare the text segmentation effects of our algorithm and other tokenizers, respectively, such as Baidu LAC, Pkuseg, Hanlp, and Jieba. The first experiment is to compare the performance of the HFCR algorithm and other tokenizers in geological texts. The selected part of specific experimental results is shown in Table 2. It can be seen that the word segmentation results of the HFCR algorithm meet expectations and can be correctly segmented in terms of age, rock, place name, numerical value, and non-professional expression, while other tokenizers all have partial semantic segmentation errors. Experiments show that the average F1-value of the HFCR algorithm in geological text word segmentation can reach more than 95%, which is superior to four representative tokenizers. The specific accuracy comparison is shown in Table 3 below. In Table 3, the text length is the length of characters. Our initial observation is that the word separation accuracy improves with increasing text character length, as shown in Table 3. Therefore, we verified the impact of candidate term sets calculated for different percentages of text on the text segmentation results. The results are shown in Figure 5 below. For the single geological independent text, the accuracy gradually increases as the range of the candidate term set is calculated, reaching the highest level when calculating the candidate term set for the full text. This result is consistent with how we read specific texts in reality. When we read part of a specific text, we may not understand it; however, \begin{table}
\begin{tabular}{c c} \hline \hline
**Description** & **Text** \\ \hline \multirow{6}{*}{Result in Pinyin} & /[PERSON]/[PERSON]/[PERSON]/ [PERSON] \\  & [PERSON]/[PERSON]/[PERSON]/[PERSON]/: [PERSON]/[PERSON] \\  & [PERSON]/[PERSON]/[PERSON] /[PERSON]/[PERSON]/[PERSON] \\  & [PERSON]//[PERSON]/[PERSON]/:[PERSON] \\  & [PERSON]/50\%,/[PERSON]/44\% \\  & /Geological profile/measured in/Muqiququan/Zanhuang \\  & Rock Group//,/Rencun Town/,/Linzhou City/:The \\ Result in English & profile/starting coordinates/X...... /Gray-black/altered \\  & gabbro, main mineral \\  & composition:/plagioclase/50\%,/diopside/44\% \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Result display of the HCRF algorithm for geological text segmentation. when we read more, our understanding will be deeper and clearer. For three geological texts, the accuracy is higher for longer text and slightly lower for shorter text. The second experiment is to compare the performance of the HFCR algorithm and other tokenizers in general text. Datasets of general text segmentation experiment are PKU and MSR datasets, which are standard datasets for testing Chinese general word segmentation tools in academia. The selected part of the specific experimental results is shown in Table 4 below. It can be seen that the segmentation granularity of long terms such as \"[PERSON] (Sunnass Hospital)\" and \"[PERSON] (University of Oslo)\" are too short. The specific experimental accuracy is shown in Table 5 below.