# The Application of Unsupervised Learning TF-IDF Algorithm in Word Segmentation of Ideological and Political Education

Authors: Silian Chen; Lina Wu; Jiexian Zhuo
Journal: Wireless Communications and Mobile Computing
 (2022)

---

## Abstract

Word segmentation is an essential part of Chinese learning. It often plays a decisive role in the quality of language processing results. Its domain vocabulary has the advantages of fast birth of new words and wide coverage. Large vocabulary, which has caused great difficulties in word segmentation and follow-up work in the field. This paper designs and implements a system. The system builds a corpus based on domain literature, trains a word-level language model based on statistical ideas, and uses the Viterbi algorithm to obtain preliminary Chinese word segmentation results. The word optimization algorithm optimizes the preliminary. The system provides users with functions such as keyword extraction, word frequency statistics, and word cloud drawing for the word segmentation results, so as to realize Chinese word segmentation and text analysis of field documents. The experimental results show that building system based on the corresponding ideas has improved the text analysis efficiency, related research on documents and text processing in the field have a certain promotion effect.


---

## Body

## 1. Introduction

Marxist theory and ideological and political education are usually called “two courses education” in the new educational system of universities. The establishment of this course provides a solution to students’ psychological contradictions. From the results of different two-course education, students need to use a large number of ideas to analyze cases and make in-depth summary in the learning process. Through the analysis of a large number of documents, this paper studies the related fields of the feasibility analysis scheme [1]. However, Chinese is significantly different from many Western Latin languages. In the written expressions of many Western languages, spaces have been used as fixed space symbols at the beginning of writing, Chinese is only words, sentences, and paragraphs, which can be simply delimited by obvious delimiters, but words do not have a formal delimiter, and words are relatively independent. However, in Chinese, the difference is that there is no fixed interval between the words, which also leads to the different segmentation of words in the same text sequence may directly lead to different meanings of the text sequence. Only by using the relevant knowledge of natural language processing can the text be further analyzed [2]. Therefore, which is often the basis and key natural language processing, and the result directly determines whether the subsequent results of natural language processing are ideal. At present, Chinese word segmentation can achieve high accuracy in the general field. However, in the face of word segmentation problems in many professional fields, general word segmentation methods often cannot achieve good word segmentation results. The characteristics of vocabulary and style have led to the following challenges in Chinese word segmentation based on supervised learning [3].First, it requires interpretation of a large number of professional documents. Second, due to the relative lack of prior knowledge, the labeling effect of domain literature may not be ideal. The third is that due to the characteristics of the early starting point and fast development speed, the research direction is wide, the professional is very large, and the general corpus is difficult to meet the needs of field document segmentation. Fourth, the update speed of research hotspots in the field of two-course education and the birth of new words are fast, and professional new words emerge in an endless stream, which brings certain challenges to corpora and manual annotation.From the above research, we can know that the fundamental need of the two-course education is to pay attention to the research accuracy and efficiency analysis in this field. Combined with the relevant knowledge of “two-course” education in various states, this paper further analyzes the relevant information in this educational field. By supervising the learning contents of different subjects, the key information in the field of ideological education is constructed, to solve the need for manual labeling of labor costs, according to the characteristics of literature in related fields, through the Chinese word segmentation optimization algorithm to further improve the accuracy and through the keyword extraction algorithm which completes the extraction and provides functions such as to meet the needs of text analysis and to help the research work in the field.


## 2. State of the Art

Word segmentation in natural language has always been the focus of research in academia [4]. The research on Chinese word segmentation can be traced back to the 1980s. As early as more than 30 years ago, Professor Liang Nanyuan of Beihang University has considered the word segmentation method of “checking the dictionary,” that is, the sentence in the text to be segmented. Scan the dictionary from the beginning to the end. When the words in the dictionary are scanned, they are marked. When encountering a string that does not exist in the dictionary, it is divided into single-character words. This theory is enough to complete a simple Chinese word segmentation task [5]. In the process of this theoretical development, Dr. Wang Xiaolong from Harbin Institute of Technology further theorized this word segmentation method through research and summarized it as a word segmentation theory for the minimum number of words, that is, the optimal result of the sentence to be segmented. Sentences should be segmented into the smallest number of word strings [6]. In Chinese word segmentation-related technologies, in the 1990s, Professor Huang Changning from Tsinghua University put forward four difficult problems in Chinese word segmentation in view of the vague standard of Chinese word segmentation at that time and the challenges due to Chinese characteristics, namely, the segmentation specification of Chinese word segmentation, the master-subordinate relationship of understanding, and the disambiguation of Chinese of unregistered words. As a guide for Chinese natural language processing, the four problems proposed by Professor Huang Changning provided directions for later Chinese language processing researchers [7]. With the increasing application of the method based on neural statistics in the analysis of language vocabulary, this paper analyzes the related contents in this field [8].The language model is used to calculate the probability of the occurrence of a sentence. The probability of the occurrence of words in each sentence is related to all the previous words. In order to improve efficiency, the starting length of the longest matching tends to be set in actual use. If you want to further improve the speed of word segmentation, you can divide the dictionary according to the length of different Chinese characters and search the dictionary corresponding to the number of Chinese characters each time. Although the code and explanation are different, the essence and result are the same. The longer the word, the higher the priority. This should be paid attention to in the research.The main idea statistics is that the more two or more words appear adjacently in the corpus, the higher the probability that they should be segmented as a word [9]. According to the idea, the adjacent word combinations in the corpus and their corresponding occurrences are counted, and the word combination with the highest probability is used as the optimal word segmentation result under the corpus training model.Li Liangjie combined the idea of statistics and the characteristics of the nature of the Semantic Web, proposed a Chinese word segmentation algorithm based on statistics and semantic information, and discussed the problem. Yuan Xianghui, on the other hand, used the conditional random field model unregistered words in addresses by analyzing the characteristics of Chinese words and expressions and independently built an address model knowledge base and designed an algorithm suitable for standardized word segmentation of Chinese addresses [10]. This also provides a great reference for Chinese word segmentation in specific fields.


## 3. Methodology

### 3.1. Text Mining and Classification

Text mining uses knowledge engineering technology, statistics, and machine learning algorithms to process text data such as documents and web pages stored in semistructured and unstructured forms and mines the potential connections between text content and semantic units and grammatical units, which can be used for text classification, cluster analysis, and other data mining and knowledge discovery processes [11]. Text mining is the mining knowledge from text content, and its general process includes data acquisition, text preprocessing, text representation, text mining analysis, and result evaluation, as shown in Figure 1 [12].Figure 1
General process of text mining.Text mining is an extension of data mining in large-scale text datasets. The acquisition of text data is the first step in text mining. Text corpus can be obtained by means of web crawling, text file reading, etc. [13]. The original text corpus is divided into isolated feature items by word segmentation. There are some meaningless and synonymous units in these feature items. The preprocessed text data needs to undergo feature selection, and these text feature items are represented as a unified data structure for subsequent analysis and modeling. Commonly used feature selections include filtering, wrapping, and embedding [14]. The purpose of feature extraction is to convert text data into numerical vectors in the vector space. Commonly used methods mainly construct the vectorized representation of text from the perspective of word bag and word embedding. Then, for different text mining tasks, different algorithms need to be selected or designed to analyze and display the inherent information of the text, so as to extract the knowledge management process for specific tasks, such as text classification, cluster analysis, and trend prediction. Finally, it is necessary to select specific evaluation indicators according to the text mining task. For example, to evaluate the text classification model, indicators such as accuracy, precision, and recall can be selected, or the mining results can be displayed in the form of result visualization.The study of text classification includes several subject areas, including natural language processing in linguistics, statistics in the field of classification mathematics in library science, and research topics such as pattern recognition, artificial intelligence, and neural networks in the computer field [15]. At first, text classification was carried out through expert rules (patterns), and an expert system was established by using knowledge engineering. The advantage of this is that it can solve the problem intuitively, but it is time-consuming and laborious, and the coverage and accuracy are limited [16].Text classification is the task of mapping a binaryd,c∈D×C to a Boolean value. The mapping is expressed mathematically as follows:
(1)Φ:D×C⟶T,F.HereD is the set of texts to be classified, C is the set of all predefined categories under a given classification system, D can be an infinite set, and C must be a finite set:
(2)di=wi1,wi2,⋯win.Among them,d is the feature vector representing the text i, and the cosine similarity is used to replace the distance metric in the text representation. The calculation formula of the similarity is as follows:
(3)Simdi,dj=∑k=1nwik∗wjk∑k=1nw2ik∗∑k=1nw2jk.Select theK training texts closest to the text di to be classified, that is, the K samples with the largest cosine similarity. The K here is an empirical value, ranging from tens to thousands. Generally, a suitable K value can be selected through cross-validation according to the distribution of the samples. By counting the text weights Pd,ci belonging to class ci in the K training texts, we can further judge the predefined categories they belong to.
(4)Pd,ci=∑d∈KNNdSimd,difdj,cj,fdj,ci=1,dj∈ci,0,dj∉ci,wit=tft,d×logN/nt+0.01∑t∈dttft,d×logN/nt+0.012.


### 3.2. Unsupervised TF-IDF Algorithm

Due to the different lengths of different documents, these frequencies have a large gap, so they need to be normalized so that these frequencies can be compared under the same environment [17].There are many ways of feature weighting, which “equal weight” and “non-equal weight” [18]. “Equal weight” considers that feature items have the same importance in the entire training set, and they will not have any substantial impact on the classification results, so all feature items are given the same weight. In contrast, “nonequal weighting” believes that the importance of feature items is different, and the role of the main feature items can be enhanced. The current research has a high degree of acceptance is the “nonequilibrium weight” method, the most representative of which is the “IDF” weight [19–21].


### 3.3. Improved TF-IDF Algorithm

#### 3.3.1. WF_TF-IDF Algorithm

In the TF-IDF algorithm, since different texts in the text library are usually manually labeled into several different categories, the algorithm only considers the total frequency of feature words in the short text library, ignoring the proportions in several categories. Wang Gensheng et al. introduced the class-frequency variance into TF-IDF. The above method is recorded as WF_TF-IDF. The class frequency variance is calculated as follows:(5)δi=∑j=1Ndfd,wi/Ndfdcj,wi2N.The size of the value indicates the degree of fluctuation of the word in the category. The larger the value, the more obvious the degree of fluctuation, the more unbalanced the distribution, and the stronger the discriminative effect on document classification. Therefore, the improved algorithm formula of TF-IDF is based on the following:(6)wi,j−δi,j=TF×IDF×δi.Formula (6) is called WF_TF-IDF single-model feature representation, which can take into account the distribution of feature words in categories and text libraries at the same time.In view of the shortcomings of the traditional TF-IDF algorithm, Zhu Meng combined CHI with the TF-IDF algorithm to form the CHI_TF-IDF single-model feature representation method. Generally, the distribution of characteristic words is considered from two perspectives: intraclass and interclass distribution. The former refers to the distribution probability ofwi between each text within the category, and the latter refers to the proportion of w in different categories of texts. Use βin to represent the occurrence probability of w within a class and βout to represent the interclass distribution of w. The formulas are as follows:
(7)βin=∑i=1nPi,h−Qh2n−1,(8)βout=∑i=1nP′i,h−Q′h2m−1.In formula (8), the coincidence represents the occurrences of each category z occurrences of feature words in all categories; it represents the mean of the proportion of the hth w in each category; m represents the total number of categories. They are expressed as follows:
(9)Pi,h=TFi,hTFh,(10)Qh=Pi,hn,(11)P′i,h=TFz,hTFh′,(12)Qh′=Pz,h′m.In formula (9), TFi,j is the value of the hth feature word w in the category single text i and TFhhth w in each document within the category. In formula (11), TFz,h is the value of the total frequency of the hth w in all texts of category z, which means the total of the hth w in all documents containing all categories. In general, the intraclass distribution and the interclass distribution show a different relationship with the expressiveness of the w classification, the former is inversely proportional to this relationship, and the latter is the opposite. In this section, we use the index β to represent the expressiveness of w being classified, and the formula is as follows:
(13)β=β0×1−βin,(14)wi,j−β=TF×IDF×β.It can be seen from formula (14) that the expressive power of w classification becomes stronger as the value of β increases and β in the classification of texts. The overall model block diagram is shown in Figure 2.Figure 2
Improved TF-IDF model.Although these commonly used methods can acquire features, they still exist. It is insufficient, resulting in poor classification effect. TF-IDF has been introduced in the previous section. The information gain is similar to its processing, which is to obtain the characteristics of the entire document. Without considering the occurrence of the feature words between categories, the feature vector containing the category difference cannot be well extracted. Mutual information and chi-square tests overestimate the role of low-frequency words in text classification, and believe that low-frequency words contain more information. The reality is that words that can express themes in a document may appear repeatedly.In this section, Word2vec combined with the improved TF-IDF algorithm is recorded as WoTFI for feature acquisition. Word2vec can not only obtain low-dimensional word vectors, but also in the vector and the improved TF-IDF algorithm. It not only fully considers the importance of words in the text but also takes into account the differences in the within and between classes, so the combination. The theoretical support formula for feature selection is as follows:(15)wi,j−δij−β=TF×IDF×δi×β.


## 3.1. Text Mining and Classification

Text mining uses knowledge engineering technology, statistics, and machine learning algorithms to process text data such as documents and web pages stored in semistructured and unstructured forms and mines the potential connections between text content and semantic units and grammatical units, which can be used for text classification, cluster analysis, and other data mining and knowledge discovery processes [11]. Text mining is the mining knowledge from text content, and its general process includes data acquisition, text preprocessing, text representation, text mining analysis, and result evaluation, as shown in Figure 1 [12].Figure 1
General process of text mining.Text mining is an extension of data mining in large-scale text datasets. The acquisition of text data is the first step in text mining. Text corpus can be obtained by means of web crawling, text file reading, etc. [13]. The original text corpus is divided into isolated feature items by word segmentation. There are some meaningless and synonymous units in these feature items. The preprocessed text data needs to undergo feature selection, and these text feature items are represented as a unified data structure for subsequent analysis and modeling. Commonly used feature selections include filtering, wrapping, and embedding [14]. The purpose of feature extraction is to convert text data into numerical vectors in the vector space. Commonly used methods mainly construct the vectorized representation of text from the perspective of word bag and word embedding. Then, for different text mining tasks, different algorithms need to be selected or designed to analyze and display the inherent information of the text, so as to extract the knowledge management process for specific tasks, such as text classification, cluster analysis, and trend prediction. Finally, it is necessary to select specific evaluation indicators according to the text mining task. For example, to evaluate the text classification model, indicators such as accuracy, precision, and recall can be selected, or the mining results can be displayed in the form of result visualization.The study of text classification includes several subject areas, including natural language processing in linguistics, statistics in the field of classification mathematics in library science, and research topics such as pattern recognition, artificial intelligence, and neural networks in the computer field [15]. At first, text classification was carried out through expert rules (patterns), and an expert system was established by using knowledge engineering. The advantage of this is that it can solve the problem intuitively, but it is time-consuming and laborious, and the coverage and accuracy are limited [16].Text classification is the task of mapping a binaryd,c∈D×C to a Boolean value. The mapping is expressed mathematically as follows:
(1)Φ:D×C⟶T,F.HereD is the set of texts to be classified, C is the set of all predefined categories under a given classification system, D can be an infinite set, and C must be a finite set:
(2)di=wi1,wi2,⋯win.Among them,d is the feature vector representing the text i, and the cosine similarity is used to replace the distance metric in the text representation. The calculation formula of the similarity is as follows:
(3)Simdi,dj=∑k=1nwik∗wjk∑k=1nw2ik∗∑k=1nw2jk.Select theK training texts closest to the text di to be classified, that is, the K samples with the largest cosine similarity. The K here is an empirical value, ranging from tens to thousands. Generally, a suitable K value can be selected through cross-validation according to the distribution of the samples. By counting the text weights Pd,ci belonging to class ci in the K training texts, we can further judge the predefined categories they belong to.
(4)Pd,ci=∑d∈KNNdSimd,difdj,cj,fdj,ci=1,dj∈ci,0,dj∉ci,wit=tft,d×logN/nt+0.01∑t∈dttft,d×logN/nt+0.012.


## 3.2. Unsupervised TF-IDF Algorithm

Due to the different lengths of different documents, these frequencies have a large gap, so they need to be normalized so that these frequencies can be compared under the same environment [17].There are many ways of feature weighting, which “equal weight” and “non-equal weight” [18]. “Equal weight” considers that feature items have the same importance in the entire training set, and they will not have any substantial impact on the classification results, so all feature items are given the same weight. In contrast, “nonequal weighting” believes that the importance of feature items is different, and the role of the main feature items can be enhanced. The current research has a high degree of acceptance is the “nonequilibrium weight” method, the most representative of which is the “IDF” weight [19–21].


## 3.3. Improved TF-IDF Algorithm

### 3.3.1. WF_TF-IDF Algorithm

In the TF-IDF algorithm, since different texts in the text library are usually manually labeled into several different categories, the algorithm only considers the total frequency of feature words in the short text library, ignoring the proportions in several categories. Wang Gensheng et al. introduced the class-frequency variance into TF-IDF. The above method is recorded as WF_TF-IDF. The class frequency variance is calculated as follows:(5)δi=∑j=1Ndfd,wi/Ndfdcj,wi2N.The size of the value indicates the degree of fluctuation of the word in the category. The larger the value, the more obvious the degree of fluctuation, the more unbalanced the distribution, and the stronger the discriminative effect on document classification. Therefore, the improved algorithm formula of TF-IDF is based on the following:(6)wi,j−δi,j=TF×IDF×δi.Formula (6) is called WF_TF-IDF single-model feature representation, which can take into account the distribution of feature words in categories and text libraries at the same time.In view of the shortcomings of the traditional TF-IDF algorithm, Zhu Meng combined CHI with the TF-IDF algorithm to form the CHI_TF-IDF single-model feature representation method. Generally, the distribution of characteristic words is considered from two perspectives: intraclass and interclass distribution. The former refers to the distribution probability ofwi between each text within the category, and the latter refers to the proportion of w in different categories of texts. Use βin to represent the occurrence probability of w within a class and βout to represent the interclass distribution of w. The formulas are as follows:
(7)βin=∑i=1nPi,h−Qh2n−1,(8)βout=∑i=1nP′i,h−Q′h2m−1.In formula (8), the coincidence represents the occurrences of each category z occurrences of feature words in all categories; it represents the mean of the proportion of the hth w in each category; m represents the total number of categories. They are expressed as follows:
(9)Pi,h=TFi,hTFh,(10)Qh=Pi,hn,(11)P′i,h=TFz,hTFh′,(12)Qh′=Pz,h′m.In formula (9), TFi,j is the value of the hth feature word w in the category single text i and TFhhth w in each document within the category. In formula (11), TFz,h is the value of the total frequency of the hth w in all texts of category z, which means the total of the hth w in all documents containing all categories. In general, the intraclass distribution and the interclass distribution show a different relationship with the expressiveness of the w classification, the former is inversely proportional to this relationship, and the latter is the opposite. In this section, we use the index β to represent the expressiveness of w being classified, and the formula is as follows:
(13)β=β0×1−βin,(14)wi,j−β=TF×IDF×β.It can be seen from formula (14) that the expressive power of w classification becomes stronger as the value of β increases and β in the classification of texts. The overall model block diagram is shown in Figure 2.Figure 2
Improved TF-IDF model.Although these commonly used methods can acquire features, they still exist. It is insufficient, resulting in poor classification effect. TF-IDF has been introduced in the previous section. The information gain is similar to its processing, which is to obtain the characteristics of the entire document. Without considering the occurrence of the feature words between categories, the feature vector containing the category difference cannot be well extracted. Mutual information and chi-square tests overestimate the role of low-frequency words in text classification, and believe that low-frequency words contain more information. The reality is that words that can express themes in a document may appear repeatedly.In this section, Word2vec combined with the improved TF-IDF algorithm is recorded as WoTFI for feature acquisition. Word2vec can not only obtain low-dimensional word vectors, but also in the vector and the improved TF-IDF algorithm. It not only fully considers the importance of words in the text but also takes into account the differences in the within and between classes, so the combination. The theoretical support formula for feature selection is as follows:(15)wi,j−δij−β=TF×IDF×δi×β.


## 3.3.1. WF_TF-IDF Algorithm

In the TF-IDF algorithm, since different texts in the text library are usually manually labeled into several different categories, the algorithm only considers the total frequency of feature words in the short text library, ignoring the proportions in several categories. Wang Gensheng et al. introduced the class-frequency variance into TF-IDF. The above method is recorded as WF_TF-IDF. The class frequency variance is calculated as follows:(5)δi=∑j=1Ndfd,wi/Ndfdcj,wi2N.The size of the value indicates the degree of fluctuation of the word in the category. The larger the value, the more obvious the degree of fluctuation, the more unbalanced the distribution, and the stronger the discriminative effect on document classification. Therefore, the improved algorithm formula of TF-IDF is based on the following:(6)wi,j−δi,j=TF×IDF×δi.Formula (6) is called WF_TF-IDF single-model feature representation, which can take into account the distribution of feature words in categories and text libraries at the same time.In view of the shortcomings of the traditional TF-IDF algorithm, Zhu Meng combined CHI with the TF-IDF algorithm to form the CHI_TF-IDF single-model feature representation method. Generally, the distribution of characteristic words is considered from two perspectives: intraclass and interclass distribution. The former refers to the distribution probability ofwi between each text within the category, and the latter refers to the proportion of w in different categories of texts. Use βin to represent the occurrence probability of w within a class and βout to represent the interclass distribution of w. The formulas are as follows:
(7)βin=∑i=1nPi,h−Qh2n−1,(8)βout=∑i=1nP′i,h−Q′h2m−1.In formula (8), the coincidence represents the occurrences of each category z occurrences of feature words in all categories; it represents the mean of the proportion of the hth w in each category; m represents the total number of categories. They are expressed as follows:
(9)Pi,h=TFi,hTFh,(10)Qh=Pi,hn,(11)P′i,h=TFz,hTFh′,(12)Qh′=Pz,h′m.In formula (9), TFi,j is the value of the hth feature word w in the category single text i and TFhhth w in each document within the category. In formula (11), TFz,h is the value of the total frequency of the hth w in all texts of category z, which means the total of the hth w in all documents containing all categories. In general, the intraclass distribution and the interclass distribution show a different relationship with the expressiveness of the w classification, the former is inversely proportional to this relationship, and the latter is the opposite. In this section, we use the index β to represent the expressiveness of w being classified, and the formula is as follows:
(13)β=β0×1−βin,(14)wi,j−β=TF×IDF×β.It can be seen from formula (14) that the expressive power of w classification becomes stronger as the value of β increases and β in the classification of texts. The overall model block diagram is shown in Figure 2.Figure 2
Improved TF-IDF model.Although these commonly used methods can acquire features, they still exist. It is insufficient, resulting in poor classification effect. TF-IDF has been introduced in the previous section. The information gain is similar to its processing, which is to obtain the characteristics of the entire document. Without considering the occurrence of the feature words between categories, the feature vector containing the category difference cannot be well extracted. Mutual information and chi-square tests overestimate the role of low-frequency words in text classification, and believe that low-frequency words contain more information. The reality is that words that can express themes in a document may appear repeatedly.In this section, Word2vec combined with the improved TF-IDF algorithm is recorded as WoTFI for feature acquisition. Word2vec can not only obtain low-dimensional word vectors, but also in the vector and the improved TF-IDF algorithm. It not only fully considers the importance of words in the text but also takes into account the differences in the within and between classes, so the combination. The theoretical support formula for feature selection is as follows:(15)wi,j−δij−β=TF×IDF×δi×β.


## 4. Result

### 4.1. Analysis of the Demand for Word Segmentation

(1)
The vocabulary formation method is special, and there are many hot words. Considering the needs of system application, the Chinese word segmentation algorithm should efficiently and accurately segment the domain vocabulary on the basis of meeting the basic needs. The word segmentation results should provide users with a function to further optimize the results(2)
There are many research direction system may not meet the word segmentation needs of individual research directions, and users may have a large number of corpora related to research directions. The system should be able to standardize the corpus provided by users. Use the standardized corpus to train related models, and users should be able to independently add new words for different models, improve the model’s ability to recognize some words, and ensure the accuracy of the system’s word segmentation for individual research directions(3)
To extract the key information of the documents in the massive documents after word segmentation. Therefore, the system should have the function of document feature extraction, including keyword extraction., word frequency statistics, and other functions to summarize relevant literature information and provide users with research reference information(4)
Considering that the documents provided by users may not be plain text or plain text files, in order to deal with various types of documents provided by users and perform word segmentation, the system needs to identify documents of different file types and extract the text in the documents for word segmentation


### 4.2. Implementation of Word Segmentation for Education Based on TF-IDF Algorithm

The hierarchical structure of “two-course” education is divided into four parts. The content includes four parts: presentation, business, support, and data. Figure3 shows the overall structure.
(1)
The hierarchical panel of this module mainly shows the operating functions of various systems(2)
It carries out the business logic processing of the system, mainly including model customization, input and output management, Chinese word segmentation, and word segmentation(3)
This level of panel contains different background function supportFigure 3
Ideological and political education analysis system.


### 4.3. Word Segmentation Results

In the previous section, Chinese word segmentation and word segmentation optimization algorithms were implemented. Since there is no public test corpus in the research, this section is verified by a certain amount of labeled corpus in the test set constructed by self-labeling. Chinese word segmentatioalgorithm and word segmentation optimization algorithm are accurate. This section selects 200 annotated corpora and many more from the test set. The word segmentation results of each word segmentation scheme are compared to determine the pros and cons of different word segmentation schemes. The literature research directions of the test set are different, covering multiple research directions and hotspots.N-gram is a commonly used language model in large vocabulary continuous speech recognition. For Chinese, we call it Chinese language model. The N-gram language types of language models according to the difference of N. Consider the characteristics of long vocabulary and complex structure in the field. The model is based on the assumption that the occurrence of thenth word is only related to the previous n−1 word and is not related to any other word. The probability of the whole sentence is the product of the occurrence probability of each word. If we know more about this sentence segment, we will get a more accurate answer. This tells us that the more the previous (historical) information, the stronger the constraint on the later unknown information. When using Google, enter one or several words, and the search box will usually provide several options like the following figure in the form of pull-down menu. These options are actually guessing the word string you want to search. Moreover, when you use the input method to input a Chinese character, the input method can usually contact a complete word. For example, if I input a “Liu,” the input method will usually prompt me whether I want to input “Liu Bei.” Through the above introduction, you should be able to keenly realize that this is actually based on the N-gram model. The specific comparison test results are shown in Figure 4. The three different colored columns represent different corpora.Figure 4
Comparison of Chinese word segmentation processing time in different order language models.In this paper, related experiments are also carried out on the influence of different amounts of training text on the classification model, and the results are shown in Figure5. The four different feature selection algorithms increase with the increase in the number of training documents, because with the increase of training samples, the more information learned, the better the performance of the model training. Considering the presence and method of feature weight representation, the WF_TF-IDF and CHI_TF-IDF algorithms are better than the traditional TF-IDF, but the WoTFI model performs the best, and the WoTFI model has a very good classification effect. Within classes is considered in the weight calculation, and the text data information is used to the greatest extent. The quality of the text classification effect is mainly represented by the accuracy of text classification, so the improved WoTFI algorithm can represent the text well and obtain a high accuracy rate.Figure 5
The effect of the number of texts on the word segmentation results.TF-IDF and its three weighted improved models, WF_TF-IDF, CHI_TF-IDF, and WoTFI, were tested. The support vector machine classification method tests the classification effects of the four models in different dimensions. The classification results of each model on the corpus are shown in Figure6.Figure 6
Classification effects of four feature selection models in different dimensions.For the traditional TF-IDF and its three weighted models WF_TF-IDF, CHI_TF-IDF, and WoTFI, the classification effect increases with the increase of the dimension, and the accuracy continues to rise. When the dimension reaches 500, the classification effect reaches the best, then the dimension further increases, and the classification effect tends to be stable or slightly decreased. Therefore, appropriate dimension settings can achieve the best classification effect. If the dimension is too low, it may be difficult for the model to achieve the best result state, and the dimension is not as large as possible. If it is too large, many useless feature words are included, which affects the results and slows down the classification speed. Analyze the results of the improved classification model to classify the SST-1 data, as shown in Figure7.Figure 7
The iterations and the accuracy.The analysis and fusion of the text representation model are affected by the single text representation model. It is necessary to set the most suitable dimensions for the two single models, WF_TF-IDF and CHI_TF-IDF, respectively. The results are shown in Figures8 and 9.Figure 8
The impact of WF_TF-IDF on WoTFI when the CHI_TF-IDF dimension is 200.Figure 9
The impact of CHI_TF-IDF on WoTFI when the dimension of WF_TF-IDF is 200.As can be seen from Figure8, when the dimension is 200, the effect of the WoTFI model changes with the change of WF_TF-IDF. When the dimension of the WF_TF-IDF model changes from 100 to 500, the classification result accuracy of WoTFI combined with SVM increases very fast. After the dimension is greater than 500, the increase is slow. When the dimension is too large, the classification effect decreases. As can be seen from Figure 9, when the WF_TF-IDF dimension is fixed to 200, the effect of the WoTFI model also changes with the change of the CHI_TF-IDF dimension. When the CHI_TF-IDF dimension increases from 100 to 500, the classification result accuracy growth rate is very fast. Later, as the dimension increases, the model training time becomes longer, the training speed becomes slower, and the effect is not obvious. Combining the two figures, the CHI_TF-IDF method is more stable than the WF_TF-IDF method.


## 4.1. Analysis of the Demand for Word Segmentation

(1)
The vocabulary formation method is special, and there are many hot words. Considering the needs of system application, the Chinese word segmentation algorithm should efficiently and accurately segment the domain vocabulary on the basis of meeting the basic needs. The word segmentation results should provide users with a function to further optimize the results(2)
There are many research direction system may not meet the word segmentation needs of individual research directions, and users may have a large number of corpora related to research directions. The system should be able to standardize the corpus provided by users. Use the standardized corpus to train related models, and users should be able to independently add new words for different models, improve the model’s ability to recognize some words, and ensure the accuracy of the system’s word segmentation for individual research directions(3)
To extract the key information of the documents in the massive documents after word segmentation. Therefore, the system should have the function of document feature extraction, including keyword extraction., word frequency statistics, and other functions to summarize relevant literature information and provide users with research reference information(4)
Considering that the documents provided by users may not be plain text or plain text files, in order to deal with various types of documents provided by users and perform word segmentation, the system needs to identify documents of different file types and extract the text in the documents for word segmentation


## 4.2. Implementation of Word Segmentation for Education Based on TF-IDF Algorithm

The hierarchical structure of “two-course” education is divided into four parts. The content includes four parts: presentation, business, support, and data. Figure3 shows the overall structure.
(1)
The hierarchical panel of this module mainly shows the operating functions of various systems(2)
It carries out the business logic processing of the system, mainly including model customization, input and output management, Chinese word segmentation, and word segmentation(3)
This level of panel contains different background function supportFigure 3
Ideological and political education analysis system.


## 4.3. Word Segmentation Results

In the previous section, Chinese word segmentation and word segmentation optimization algorithms were implemented. Since there is no public test corpus in the research, this section is verified by a certain amount of labeled corpus in the test set constructed by self-labeling. Chinese word segmentatioalgorithm and word segmentation optimization algorithm are accurate. This section selects 200 annotated corpora and many more from the test set. The word segmentation results of each word segmentation scheme are compared to determine the pros and cons of different word segmentation schemes. The literature research directions of the test set are different, covering multiple research directions and hotspots.N-gram is a commonly used language model in large vocabulary continuous speech recognition. For Chinese, we call it Chinese language model. The N-gram language types of language models according to the difference of N. Consider the characteristics of long vocabulary and complex structure in the field. The model is based on the assumption that the occurrence of thenth word is only related to the previous n−1 word and is not related to any other word. The probability of the whole sentence is the product of the occurrence probability of each word. If we know more about this sentence segment, we will get a more accurate answer. This tells us that the more the previous (historical) information, the stronger the constraint on the later unknown information. When using Google, enter one or several words, and the search box will usually provide several options like the following figure in the form of pull-down menu. These options are actually guessing the word string you want to search. Moreover, when you use the input method to input a Chinese character, the input method can usually contact a complete word. For example, if I input a “Liu,” the input method will usually prompt me whether I want to input “Liu Bei.” Through the above introduction, you should be able to keenly realize that this is actually based on the N-gram model. The specific comparison test results are shown in Figure 4. The three different colored columns represent different corpora.Figure 4
Comparison of Chinese word segmentation processing time in different order language models.In this paper, related experiments are also carried out on the influence of different amounts of training text on the classification model, and the results are shown in Figure5. The four different feature selection algorithms increase with the increase in the number of training documents, because with the increase of training samples, the more information learned, the better the performance of the model training. Considering the presence and method of feature weight representation, the WF_TF-IDF and CHI_TF-IDF algorithms are better than the traditional TF-IDF, but the WoTFI model performs the best, and the WoTFI model has a very good classification effect. Within classes is considered in the weight calculation, and the text data information is used to the greatest extent. The quality of the text classification effect is mainly represented by the accuracy of text classification, so the improved WoTFI algorithm can represent the text well and obtain a high accuracy rate.Figure 5
The effect of the number of texts on the word segmentation results.TF-IDF and its three weighted improved models, WF_TF-IDF, CHI_TF-IDF, and WoTFI, were tested. The support vector machine classification method tests the classification effects of the four models in different dimensions. The classification results of each model on the corpus are shown in Figure6.Figure 6
Classification effects of four feature selection models in different dimensions.For the traditional TF-IDF and its three weighted models WF_TF-IDF, CHI_TF-IDF, and WoTFI, the classification effect increases with the increase of the dimension, and the accuracy continues to rise. When the dimension reaches 500, the classification effect reaches the best, then the dimension further increases, and the classification effect tends to be stable or slightly decreased. Therefore, appropriate dimension settings can achieve the best classification effect. If the dimension is too low, it may be difficult for the model to achieve the best result state, and the dimension is not as large as possible. If it is too large, many useless feature words are included, which affects the results and slows down the classification speed. Analyze the results of the improved classification model to classify the SST-1 data, as shown in Figure7.Figure 7
The iterations and the accuracy.The analysis and fusion of the text representation model are affected by the single text representation model. It is necessary to set the most suitable dimensions for the two single models, WF_TF-IDF and CHI_TF-IDF, respectively. The results are shown in Figures8 and 9.Figure 8
The impact of WF_TF-IDF on WoTFI when the CHI_TF-IDF dimension is 200.Figure 9
The impact of CHI_TF-IDF on WoTFI when the dimension of WF_TF-IDF is 200.As can be seen from Figure8, when the dimension is 200, the effect of the WoTFI model changes with the change of WF_TF-IDF. When the dimension of the WF_TF-IDF model changes from 100 to 500, the classification result accuracy of WoTFI combined with SVM increases very fast. After the dimension is greater than 500, the increase is slow. When the dimension is too large, the classification effect decreases. As can be seen from Figure 9, when the WF_TF-IDF dimension is fixed to 200, the effect of the WoTFI model also changes with the change of the CHI_TF-IDF dimension. When the CHI_TF-IDF dimension increases from 100 to 500, the classification result accuracy growth rate is very fast. Later, as the dimension increases, the model training time becomes longer, the training speed becomes slower, and the effect is not obvious. Combining the two figures, the CHI_TF-IDF method is more stable than the WF_TF-IDF method.


## 5. Conclusion

(1)
This section selects 200 annotated corpora and more corpora from the test set and compares the word segmentation results of each word segmentation scheme. According to the difference ofN, N-gram language model takes into account the characteristics of long vocabulary and complex structure in this field. There are obvious differences in the processing time of Chinese word segmentation under different sequential language models(2)
This paper also carries out relevant experiments on the impact of different numbers of training texts on the classification model. Four different feature selection algorithms increase with the increase of the number of training documents, because with the increase of training samples, the more information learned, the better the performance of model training. The WF_TF-IDF and CHI_TF-IDF algorithms are better than the traditional TF-IDF, and WoTFI model performs best. When the number of training documents exceeds 1400, the accuracy of WoTFI model exceeds 90%, WF_TF-IDF and CHI_TF-IDF. The accuracy of TF-IDF model is more than 85% and has remained stable since then. The accuracy of traditional TF-IDF model is lower than the first three. The improved WoTFI algorithm can represent text well and obtain high accuracy(3)
TF-IDF and its three weighted improved models, WF_TF-IDF, CHI_TF-IDF, and WoTFI, were tested. The classification effect of the four models increases with the increase of dimension, and the accuracy continues to improve. When the dimension reaches 500, the classification effect reaches the best, and then, the dimension further increases, and the classification effect tends to be stable or slightly decreases. Therefore, proper dimension setting can achieve the best classification effect. When the dimension is 200, the effect of the WoTFI model changes with the change of WF_TF-IDF. When the dimension of the WF_TF-IDF model changes from 100 to 500, the classification result accuracy of WoTFI combined with SVM increases very fast. When the WF_TF-IDF dimension is fixed to 200, the effect of the WoTFI model also changes with the change of the CHI_TF-IDF dimension. When the CHI_TF-IDF dimension increases from 100 to 500, the accuracy growth rate is very fast. The classification results show that the algorithm model can not only capture the two-way semantic dependencies, process sequential text data of different lengths, and obtain deep text features by capturing the shape and morphology information of words or phrases but also effectively retain the semantic information of short text and avoid long text at the same time. There were gradient explosion and disappearance in sequence training. The word segmentation system based on the corresponding idea improves the efficiency of text analysis and promotes the related research



---

*Source: 5219117-2022-08-09.xml*