# Candidate Term Boundary Conflict Reduction Method for Chinese Geological Text Segmentation
## 3 Candidate Term Boundary Conflict Reduction Method Based on Information Entropy  
### Technical Route of Word Segmentation Algorithm for Single Geological Text  
Analogously to the process of human reading, when a string appears frequently in a text with a fixed connection, it is probably a correct word. By observing the writing features and characteristics of geological texts, it can be found that proper nouns are expressed as fixed connections in terms of character relationships. All the Chinese words in this article are replaced by Pinyin and explained in English. For example, \"Shan Chang Yan (diorite)\" in Chinese becomes a word with complete basic meaning through the connection of three characters: \"Shan (flash)\", \"Chang (long)\", and \"Yan (stone)\". This word is correct and is allowed to appear freely in other positions of the text as a fixed whole. Therefore, there should be abundant possible items for the left and right connecting characters of this term, and the left and right degree of freedom of the correct term is relatively high. The tightness of internal character connection and richness of external character connection of one term are meaningful and regular. This research uses information entropy to quantify the left and right degrees of freedom of candidate terms.  
Although there are no natural space separators as there are in English, Chinese text has natural punctuation marks such as commas, periods, semicolons, exclamation marks, and question marks, which can be initially cut off through preprocessing. The term combination cases of a text are exhaustive, and candidate terms are all combinations of characters of indefinite length. According to the text reading habit, all character combinations are obtained by traversing the preprocessed text from left to right and calculating information entropy; terms with an information entropy of 0 (that is, the degree of freedom is 0) were deleted as preliminary filtering to obtain the candidate term set.  
In the actual word segmentation, some overly active characters such as \"De (of)\", \"Shi (is)\", and \"Le (the)\" in Chinese will freely appear in connection with other characters in the text. To a certain extent, these special combinations will affect the result of word segmentation. These active characters are called stop words in many word segmentation studies, which need to be skipped in advance. In order to further avoid manual definition, information entropy can be used to quantify the activity of a single character. After performing single-character experiments on numerous texts, the first part of the single character with the highest activity (that is, the highest information entropy) can be summarized and extracted as stop words.  
There will be certain conflicts between candidate terms such as inclusive conflicts between \"[PERSON] (Henan)\" and \"He Nan Sheng (Henan province)\" and intersecting conflicts between \"[PERSON] (gray)\" and \"[PERSON] (color middle)\". These are conflicts about the term boundary, and the information entropy of the determining term boundary is an index to quantify the internal tightness and external degree of freedom. A term with high information entropy is represented as a whole with close connections and a high degree of freedom in the text. For example, words such as \"Yan Shi (rock)\", \"Kuang Qu (mining area)\", \"Di Zhi (geology)\", and \"Zuan Kong (borehole)\" show extremely high information entropy. Compared with using a fixed threshold to filter candidate terms, it is more in line with the cognition of the meaning of information entropy to filter the value of information entropy on the conflict side. Thus, the technical route flow chart of Chinese geological text word segmentation based on information entropy can be obtained, as shown in Figure 1 below.