Research on Feature Selection and kNN Classification Method in Chinese Text Classification

Scholars at home and abroad have done lots of research on feature selection methods in Chinese text classification, such as document frequency (DF), information gain (IG), and a 2  -test (CHI). On the basis of their work, we propose a new selection method of counting the unbalanced degree of term distribution, compare it with other feature selection methods using the k-nearest-neighbor (kNN) algorithm, and find that the new method performs as well as CHI and IG. Experiments have shown that whatever the feature selection method we choose, after the number of features reaches a certain value, the gain of classification accuracy becomes very slight. Keep increasing the feature dimension can hardly improve the classification performance, while the time consumed doubles. In that case, we attempts to improve the kNN method by counting the text similarity differently. The improved method will quantify each feature’s weight using a bit string, count the similarity of two documents under their bits mode, and finally remarkably reduce the space required for storing documents and the time consumed by counting their similarity. Experiments have confirmed that the new kNN method can greatly accelerate the speed of classification at the expense of a little loss of classification accuracy. Introduction With the rapid development of information technology, retrieving information becomes very simple. People can get access to information through the Internet conveniently. But when they take a closer look at the information downloaded from the Internet, they will probably find that there are too many useless, repetitive and outdated information and it takes lots of time to filter out the useful information manually. In order to get the valuable information efficiently, we use text categorization to solve this problem. Text categorization is the problem of automatically assigning predefined categories to free text documents. As to Chinese text categorization, the first difficulty is how to divide one document into a list of meaningful terms (words or phrases), since unlike English, Chinese text does not have a natural delimiter between words. Chinese word segmentation methods are not the focus of our work, and we use a segmentation software offered by Professor Zhang Huaping[1] instead. In addition, there is an important thing to mention that perfect precision and disambiguation in Chinese word segmentation can not be reached. As a result, the inherent errors caused by word segmentation always remain as a problem in Chinese information processing. However, we can focus on the performances of different feature selection methods as long as the errors caused by word segmentation are reasonably low. The second difficulty of Chinese text categorization is the high dimensionality of the feature space. The native feature space may consist of tens or hundreds of thousands of unique terms for even a moderate-sized text collection that is prohibitively high for many classification algorithms. It is highly desirable to reduce the native space without sacrificing categorization accuracy. The process of feature selection is designed to achieve such a goal. The final difficulty or characteristic is the applications of different classification algorithms to Chinese text 4th National Conference on Electrical, Electronics and Computer Engineering (NCEECE 2015) © 2016. The authors Published by Atlantis Press 956 categorization. In this paper, we examine the capabilities of the k-nearest-neighbor (kNN) algorithm in mining categorization knowledge from high dimensional document feature vectors. Then we find out an easy way to calculate the text similarity and make the process of classification more quickly with a little loss of classification accuracy. Feature Selection Methods Feature selection has become the focus in the field of machine learning research. Its task in text classification is to use a term-goodness criterion thresholded to eliminate a large number of terms from the vocabulary of a corpus. Document frequency (DF), information gain (IG), and a 2  statistic (CHI) are the three most commonly used criteria in text classification[2]. At the last part of this section we propose a new criterion based on the unbalanced degree of term distribution inspired by the CHI method. All feature selection methods follow the same general module structure[3] as described by the pseudo code below: Input a dictionary D , a class c , and the number of feature words k Output a list of feature words belonging to the class c donated as Sel Step 1 select valid words from the dictionary D to form a vocabulary V ; Step 2 let L to be an empty list; Step 3 for each word t V  , repeating the following sub-steps; Step 3.1 compute the weight of t belonging to the class c denoted as ( , ) A t c ; Step 3.2 make a pair of t and its weight ( , ) A t c , then insert it into the list L in order of weight; Step 4 select the first k elements in list L , and return the corresponding k feature words as Sel Document Frequency (DF). Document frequency is the number of documents in which a term occurs. We compute the document frequency for each term in the training corpus and remove those terms whose document frequency is less than some predetermined threshold. DF thresholding is the simplest feature selection method and it can easily scales to large corpus, with a computational complexity approximately linear in the number of training documents. The principle of this method is based on an easy assumption that rare terms are either non-informative for category predication, or not influential in global performance[4]. As a result, we can aggressively remove rare terms without losing much categorization information and select the terms occurring more frequently to construct the feature space. Information Gain (IG). Information gain measures the number of bits of information obtained from category prediction by knowing the presence or absence of a term in a document[5]. The information gain of term t can be defined as: 1 1 1 ( ) ( ) log ( ) ( ) ( ) log ( ) ( ) ( ) log ( ) m m m r i r i r r i r i r r i r i i i i G t p c p c p t p c t p c t p t p c t p c t           (1) ( ) r i p c denotes the probability of a document belonging to category i c . ( ) r p t denotes the probability of term t occurring in a document while ( ) r p t denotes the probability of term t not occurring in a document. ( ) r i p c t is the conditional probability of category i c given term t , and ( ) r i p c t denotes the conditional probability of category i c when term t is absent in a document. m is the number of categories. Given a training corpus, we compute the information gain for each term, remove those terms whose score is less than some predetermined threshold and finally get the feature space we wanted. 2  Statistic (CHI). The basic idea of the 2  statistic method is to determine whether the assumption we made is correct or not by comparing the theoretical value and the practical value. First we assume that a term t and a category c are independent of each other. Then we calculate the two-way contingency table of them theoretically, and get the practical value of the contingency table