# Text Classification Using Novel Term Weighting Scheme-Based Improved TF-IDF for Internet Media Reports

Authors: Zhiying Jiang; Bo Gao; Yanlin He; Yongming Han; Paul Doyle; Qunxiong Zhu
Journal: Mathematical Problems in Engineering
 (2021)

---

## Abstract

With the rapid development of the internet technology, a large amount of internet text data can be obtained. The text classification (TC) technology plays a very important role in processing massive text data, but the accuracy of classification is directly affected by the performance of term weighting in TC. Due to the original design of information retrieval (IR), term frequency-inverse document frequency (TF-IDF) is not effective enough for TC, especially for processing text data with unbalanced distributions in internet media reports. Therefore, the variance between the DF value of a particular term and the average of all DFsDF¯, namely, the document frequency variance (ADF), is proposed to enhance the ability in processing text data with unbalanced distribution. Then, the normal TF-IDF is modified by the proposed ADF for processing unbalanced text collection in four different ways, namely, TF-IADF, TF-IADF+, TF-IADFnorm, and TF-IADF+norm. As a result, an effective model can be established for the TC task of internet media reports. A series of simulations have been carried out to evaluate the performance of the proposed methods. Compared with TF-IDF on state-of-the-art classification algorithms, the effectiveness and feasibility of the proposed methods are confirmed by simulation results.


---

## Body

## 1. Introduction

Due to the rapid development of internet technology and information infrastructure construction, the volume of text data which can be obtained online has increased dramatically. As the China Internet Network Information Center (CNNIC) stated, the number of netizens in China had increased to 828.51 million by the end of 2018 [1]. The internet has become the main channel for Chinese people to obtain information. The content of internet media is the most important data source, in which textual documents are the main one. And it is increasingly important to effectively analyze massive textual documents such as classification, indexing, and clustering. As a consequence, text classification (TC) is concerned by many researchers working in the field. Based on the previous studies, many applications based on TC technology have been developed, such as author identification [2, 3], spam e-mail filtering [4], medical documents’ classification [5], management of customer relationship, and classification of web pages [6, 7]. Text classification (TC) is a task that assigns textual documents to predefined classes based on knowledge extracted from their content. The process of TC is as follows [8]:(i)
Given a set ofk different discrete class label values C = {C1, …, Ck} and training data and a set of documents D = {D1, …, Dn}, each document of which is labeled with a specific value from set C(ii)
Calculate text representations for documents in setD(iii)
Build a classification model based on the training data, which indicates the relationship between the features in the underlying document and one of the classes(iv)
Predict class labels for class-unknown documents using the trained modelCalculating text representation, training classification models, and predicting class labels for class-unknown documents are the main steps of text classification. The entire steps, factors, and the way they organize in TC are shown in Figure1.Figure 1
The entire steps, factors, and the way they organize in text classification.As shown in Figure1, before documents can be analyzed by a classification model, documents need to be preprocessed in a specific way such as be represented by vectors with numerical values. These values relate to predefined classes that the classification model can understand. This process is called text representation, and it is an essential prerequisite for TC tasks [9]. There are many methods proposed for text representation among which the vector space model (VSM) is the most commonly used one [10]. VSM is a feature vector that consists of numerical values which are also called term weights for representing a document. Components of this kind of model can be of different types such as words, sentences, and phrases [11]. These components are also called terms which are extracted from a document to form a bag of words (BOW) [12]. The abilities of these terms in distinguishing different documents are represented by numerical values (weights) related to the terms [13]. For example, a document can be represented as a vector of weighted features (or terms) dk = (t1, t2, …, tn) and a corresponding weight vector wk = (w1, w2, …, wn), where n is the number of selected features (terms) and w1, w2, …, wn are the weights of t1, t2, …, tn. Then, a collection of documents (corpus) can be represented as shown in Figure 2, where element wi,j represents the weight of tj from di.Figure 2
An example of the weight matrix for a collection of documents in the vector space model.As we can see in the matrix, each term in each document can only be assigned to one weight at the same time in VSM. It is obviously crucial to assign appropriate weights to terms for the performance of text classification. Therefore, many methods which are called term weighting scheme (TWS) are proposed to determine the weights for terms of documents. Different TWSs generate different vectors for the same document, thus attributing to the document with different representations. “Good” term weighting methods are of fundamental importance for guaranteeing good TC performance. So far, there are two main categories of TWSs in the literature: semantic-based TWSs and statistics-based TWSs [14].The semantic-based TWSs focus on the semantic relationships between terms and documents which are hidden behind the extracted features (words) as well as focus on the meaning of words [15]. For example, Rao et al. proposed a new model based on a neural network which captures semantics of continuous text representations [16]. Based on the distributed hypothesis in which meanings of terms from documents with similar meanings will also be similar, a neural network was used to embed words into a continuous vector space (Word2Vec) for capturing the semantic information of words [17]. Doc2Vec, based on Word2Vec, was extended from the word level to the document level by fully using the information of the word sequence [18]. Because the above methods all have limitations when being used on their own, Kim et al. combined the BOW and Doc2Vec and proposed the bag-of-concepts method for overcoming the limitations [19], and Kim et al. also tried different methods, namely, TF-IDF, LDA, and Doc2Vec, for text representation [20]. More recently, Wu et al. proposed a novel phrase-based text representation called Phrase2Vec, which includes skip phrase, CBOP, and GloVeFP. They applied the novel method to text analysis research, and results show that Phrase2Vec can improve the performance of TC and clustering tasks [21]. Jaeyoung Kim et al. first applied capsule networks, which achieved success in image classification, in the TC task and demonstrated comparable performance to well-known schemes at the time [22].For non-English languages, semantic analysis is also widely used for TC. Ye-wang Chen et al. proposed a novel method using the biggest open and free internet knowledge-based Baidu Baike to capture the semantic relationships of words to categories to enhance the performance of TC in Chinese text [23]. Ashraf Elnagar et al. introduced two new datasets for Arabic TC tasks, namely, SANAD and NADiA, both of which are freely available for research studies. In their experiments of extensive comparisons among several deep learning (DL) models for Arabic TC on SANAD and NADiA, their method outperformed others because of no requirement of a preprocessing stage and being completely based on deep learning models [24].However, TWSs based on semantic analysis are more complex than statistical counterparts in analyzing and calculating the process. Furthermore, for semantic-based methods, performance cannot be significantly improved. Therefore, statistics-based TWSs are still major topics in the field of text classification [25]. Normally, most statistics-based TWSs all depend on the following philosophies:(i)
Terms with higher occurrences in a document relate to the document better, which is the basic idea of “term frequency” (TF)(ii)
Terms with occurrence in fewer documents relate to the documents where they occur better, which is the basic idea of “inverse document frequency” (IDF)According to these principles, there are two main factors in a statistics-based TWS as it is shown in the following equation:(1)TWS=term frequency factor∗collection frequency factor.Many methods are proposed in the literature based on different implementations of equation (1). Some popular examples are shown in Table 1, where values of “NONE” indicate there is no corresponding method for the specific parameter. As the table shows, some methods focus on modifying the term frequency factor (i.e., LogTF-RF [11] and SQRT_TF-IGM [25]), while some focus on developing novel methods as the collection frequency factor (i.e., TF-IDF [26], TF-CHI2 [27], TF-IEF [14], and TF-IGM [25]). Nevertheless, TF-IDF is still one of the most preferred methods.Table 1
Some popular examples of statistics-based term weighting schemes.
TWSTerm frequency factorCollection frequency factorTFTerm frequency (TF)NONEDFNONEDocument frequency (DF)TF-IDFTerm frequency (TF)Inverse document frequency (IDF)TF-CHI2Term frequency (TF)Globalized chi-square score (CHI2)LogTF-RFLogarithm of term frequencyRelevance frequency (RF)TF-IGMTerm frequency (TF)Inverse gravity moment (IGM)SQRT_TF-IGMSquare root of term frequencyInverse gravity moment (IGM)TF-IEFTerm frequency (TF)Inverse exponential frequency (IEF)In this paper, enriching the collection frequency factor of statistics-based TWS is concerned, i.e., TF-IDF, for handling situations of imbalanced data distribution. A new formula is designed by using the variance between the DF of a specific term and the average value of all DFsDF¯ instead of original DF in TF-IDF. Based on the new formula, a novel method named TF-IADF and three other TWSs based on the same idea are proposed to enhance the TC performance in the imbalanced situation of internet media reports.The remainder of this paper is organized as follows. An overview of background study on statistics-based term weighting schemes is given in Section2. The main idea of our novel methods is described in Section 3. Section 4 briefly introduces the experimental settings and datasets, including data preprocessing, the classifiers, and the measurements. Experimental results and the analysis are presented in Section 5. The final conclusion is given in Section 6.


## 2. Background Study

When looking at the studies related to TWS in the literature, TF-IDF, originally designed for information retrieval (IR), may be at the top of the list. However, as Chen et al. stated, due to its original design, TF-IDF is not effective enough in the text classification domain [25]. Thus, they proposed a new statistics-based model named inverse gravity moment (IGM) to describe the intercategory distribution. Based on IGM, TF-IGM and sqrt_TF-IGM (RTF) are proposed. In their demonstration on popular classifiers, namely, SVM and kNN, the proposed methods had better performance in measurements such as micro-F1 and macro-F1 than existing TWSs (TF, TF-IDF, TFIDF-ICSDF, TF-CHI, TF-PB, and TF-RF). However, Turgut Dogan et al. reviled that, for each case where the term document frequency changes, the term with the same weight is given by TF-IGM. This means terms with different distinguishing abilities obtain the same weights from the standard IGM method which is unreasonable [28]. In their studies, two novel TWSs, namely, SQRT_TF-IGMimp and TF-IGMimp, are proposed deriving from IGM to overcome its limitations. In other aspects, Zhong Tang et al. described two deficiencies from which TF-IDF suffers, namely, collection frequency factor being undefined (division by zero) or being equal to zero in some special cases. They proposed a novel method, namely, term frequency-inverse exponential frequency (TF-IEF), to overcome these drawbacks [14]. The proposed methods replaced the IDF with a global weighting factor IEF, and a log-like method is used to characterize the collection frequency factor. It greatly reduced the influence caused by terms with high TF values, which helped in generating a more representative vector of terms. The experiments stated that the novel methods had an improved performance than compared schemes. The knowledge about Chinese language and Chinese culture provided by Baidu Baike is learned and organized by Chinese language-speaking people and professional employees of Baidu company. Therefore, Baidu Baike is used for optimizing TC on Chinese text a couple of times in the Chinese language aspect [23, 29]. However, both Baidu Baike-based methods are based on semantic analysis, and huge calculations are required for processing.However, most of these methods are based on the assumption that the dataset is relatively balanced in distribution. In fact, the imbalanced distribution of the dataset occurs frequently in the TC domain [30]. Furthermore, the classification performance is heavily affected by the imbalanced distribution of the dataset in TC [31, 32]. Many studies have been proposed to address this problem, such as [33, 34]. In these proposed studies, two common ways are used to solve the problem of data imbalance, namely, the data-driven methods and the algorithm-driven methods. The data-driven method is to adjust the proportion of data categories by undersampling, oversampling, or a combination of undersampling and oversampling. The algorithm-driven method is to adjust the classification algorithm to achieve the effect of promoting learning without changing the dataset. The simulation results of these proposed methods show that the more unbalanced the proportion of categories is, the lower the overall performance of TC becomes. One of the main reasons is that some less-common terms in large-scale categories are weighted even higher than some more-common terms in small-scale categories due to their frequencies of occurrence.Document classification of Chinese media reports on the internet which is also a TC problem with imbalanced dataset is researched in this paper. And a more representative model in cases of imbalance data is tried to create by modifying the term weighting method.


## 3. Novel Term Weighting Methods Based on Improved TF-IDF

TF-IDF is the most widely used TWS proposed by Karen Spärck Jones [26]. In this section, a new TWS based on TF-IDF, namely TF-IADF, and its variants proposed in this paper are described in specific.


### 3.1. Overview to TF-IDF

TF-IDF [35] is a combination of term frequency (TF) and inverse document frequency (IDF). Since the original value of term frequency in a document is used directly, the TF representation is one of the simplest TWSs. TF is based on the assumption that a term with a higher term frequency value is regarded to be more important than that with a lower term frequency value. It only depends on the number of occurrences of a specific term in a local document. Therefore, the capacity of TF for distinguishing all relevant documents from other irrelevant documents is very low due to its ignorance of collection frequency. To address this problem, the inverse document frequency (IDF) was proposed with a concern of collection frequency which enhanced the discriminative capacity of a term for text classification [36]. IDF extends from document frequency (DF) which means the number of documents where a term occurs. It is proposed based on the assumption that a term which occurs in fewer documents is regarded to be more important than that which occurs in more documents [11]. The IDF value of a specific term can be obtained as shown in the following:(2)IDFt,d,D=logDDFt,D.In equation (2), DFt,D represents the DF value of term t in corpus D. The symbol in equation (2) represents the total number of documents in corpus D. To avoid infinity of some extreme cases, the formula is sometimes optimized as shown in the following:(3)IDFt,d,D=logD+1DFt,D+1.After that, Jones extended the IDF method by adding the TF value into calculation [26]. The proposed combination with TF and IDF is the most well-known term weighting method, namely, TF-IDF. Similar with IDF, TF-IDF is also a global statistical measure. The classical structure of TF-IDF is shown as(4)TF−IDFt,d,D=TFt,d∗IDFt,d,D.In equation (4), TF−IDFt,d,D represents the weight of term t of document d in corpus D, while TFt,d represents the TF value of term t in document d.As we introduced in Section2, TF-IDF is not effective enough in the text classification domain due to its original design. And many research studies have been deployed in optimizing term weighting methods based on TF-IDF from different perspectives. Some of them developed new methods replacing the term frequency factor or document frequency factor of TF-IDF, while some of them modified the existing method of TF-IDF. This paper focuses on modifying IDF to improve the TC performance, especially for Chinese internet media content.


### 3.2. Proposed Methods

When looking into the formula of calculating the value of IDF as shown in equations (2) and (3), we notice that when the corpus is not very balanced which means the size of different categories in a corpus varies from each other, terms from categories with larger size will be assigned smaller values than terms from other categories. This is obviously not in line with the real situation. Moreover, for some low document-frequency terms, the value of IDF is much higher than others even when those low document-frequency terms are meaningless, which is not in line with the true situation either. To address this kind of problems, we focus on the deviation of the DF value between a specific term and the overview of all terms in the whole corpus since when the deviation between the DF value of a specific term and the average of all DF values is large, its discriminative ability is weak. This factor should be considered in the term weighting process.Definition 1 (average document frequency (ADF)).
It is the variance between the DF value of a specific term and the average of all DF values in a corpus.
We modified the collection frequency factor by adding ADF into calculation to address the problems mentioned above. In this study, the average of all DF values in the corpus is represented asDF¯, while the ADF value of term t in document D is represented as ADFt,D. Equations (5) and (6) show how they are calculated, where n is the number of terms:(5)DF¯=∑DFt,Dn,(6)ADFt,D=DFt,D−DF¯2n.
As ADF is extended from DF, the simplest way of optimizing IDF is to replace DF by ADF in the formula. Then, we get a novel formula of collection frequency which is shown as follows:(7)IADFt,D=logD+1ADFt,D+1.
In fact, the IDF method is successful enough in most cases; what we need to do is just to modify it for some extreme cases. Then, we get another novel formula as shown in equation (8), where ADF is used to reduce the weight of the terms with extremely high or extremely low DF value.(8)IADF+t,D=logD+1DFt,D+1∗1logADFt,D+1+1.
The two ADF-based methods can improve the TC performance in some cases we mentioned. However, there are still limitations due to the variance itself that when the size is too large or too small, the variance will be relatively too small or too large. Extreme values for terms will obviously impact the TC performance. Therefore, we further optimized the formula by normalizing the ADF to reduce the effect caused by the extreme value of terms. First,ADFt,D is modified as shown in equation (9) and then using the normalization formula as shown in equation (10):(9)ADF′t,D=log1ADFt,D+1+1,(10)ADF′t,D=ADF′t,D−minADF′t,DmaxADF′t,D−minADF′t,D.
Based onADF″, another two novel formulas are designed as shown in equations (11) and (12), where α (default value is 1) is used as an optional weight proportion to adjust the importance of ADF″ in different cases.(11)IADFnormt,D=logD+1ADF″t,D+1,(12)IADFnorm+t,D=logD+1DFt,D+1∗ADF″t,D∗α.
Based on the above four proposed formulas of collection frequency based on IDF, we get four novel term weighting methods which are shown in equations (13)–(16):(13)TF–IADFt,d,D=TFt,d∗IADFt,D,(14)TF–IADF+t,d,D=TFt,d∗IADF+t,D,(15)TF–IADFnormt,d,D=TFt,d∗IADFnormt,D,(16)TF–IADFnorm+t,d,D=TFt,d∗IADFnorm+t,D.
As a result, the optimized text representation model of processing internet media reports is shown in Figure3. Four new calculation formulas are used to replace IDF of TF-IDF. And four novel term weighting methods are obtained to enhance the performance of processing unbalanced text collection.Figure 3
The flowchart of the text classification method based on the improved TF-IDF.


## 3.1. Overview to TF-IDF

TF-IDF [35] is a combination of term frequency (TF) and inverse document frequency (IDF). Since the original value of term frequency in a document is used directly, the TF representation is one of the simplest TWSs. TF is based on the assumption that a term with a higher term frequency value is regarded to be more important than that with a lower term frequency value. It only depends on the number of occurrences of a specific term in a local document. Therefore, the capacity of TF for distinguishing all relevant documents from other irrelevant documents is very low due to its ignorance of collection frequency. To address this problem, the inverse document frequency (IDF) was proposed with a concern of collection frequency which enhanced the discriminative capacity of a term for text classification [36]. IDF extends from document frequency (DF) which means the number of documents where a term occurs. It is proposed based on the assumption that a term which occurs in fewer documents is regarded to be more important than that which occurs in more documents [11]. The IDF value of a specific term can be obtained as shown in the following:(2)IDFt,d,D=logDDFt,D.In equation (2), DFt,D represents the DF value of term t in corpus D. The symbol in equation (2) represents the total number of documents in corpus D. To avoid infinity of some extreme cases, the formula is sometimes optimized as shown in the following:(3)IDFt,d,D=logD+1DFt,D+1.After that, Jones extended the IDF method by adding the TF value into calculation [26]. The proposed combination with TF and IDF is the most well-known term weighting method, namely, TF-IDF. Similar with IDF, TF-IDF is also a global statistical measure. The classical structure of TF-IDF is shown as(4)TF−IDFt,d,D=TFt,d∗IDFt,d,D.In equation (4), TF−IDFt,d,D represents the weight of term t of document d in corpus D, while TFt,d represents the TF value of term t in document d.As we introduced in Section2, TF-IDF is not effective enough in the text classification domain due to its original design. And many research studies have been deployed in optimizing term weighting methods based on TF-IDF from different perspectives. Some of them developed new methods replacing the term frequency factor or document frequency factor of TF-IDF, while some of them modified the existing method of TF-IDF. This paper focuses on modifying IDF to improve the TC performance, especially for Chinese internet media content.


## 3.2. Proposed Methods

When looking into the formula of calculating the value of IDF as shown in equations (2) and (3), we notice that when the corpus is not very balanced which means the size of different categories in a corpus varies from each other, terms from categories with larger size will be assigned smaller values than terms from other categories. This is obviously not in line with the real situation. Moreover, for some low document-frequency terms, the value of IDF is much higher than others even when those low document-frequency terms are meaningless, which is not in line with the true situation either. To address this kind of problems, we focus on the deviation of the DF value between a specific term and the overview of all terms in the whole corpus since when the deviation between the DF value of a specific term and the average of all DF values is large, its discriminative ability is weak. This factor should be considered in the term weighting process.Definition 1 (average document frequency (ADF)).
It is the variance between the DF value of a specific term and the average of all DF values in a corpus.
We modified the collection frequency factor by adding ADF into calculation to address the problems mentioned above. In this study, the average of all DF values in the corpus is represented asDF¯, while the ADF value of term t in document D is represented as ADFt,D. Equations (5) and (6) show how they are calculated, where n is the number of terms:(5)DF¯=∑DFt,Dn,(6)ADFt,D=DFt,D−DF¯2n.
As ADF is extended from DF, the simplest way of optimizing IDF is to replace DF by ADF in the formula. Then, we get a novel formula of collection frequency which is shown as follows:(7)IADFt,D=logD+1ADFt,D+1.
In fact, the IDF method is successful enough in most cases; what we need to do is just to modify it for some extreme cases. Then, we get another novel formula as shown in equation (8), where ADF is used to reduce the weight of the terms with extremely high or extremely low DF value.(8)IADF+t,D=logD+1DFt,D+1∗1logADFt,D+1+1.
The two ADF-based methods can improve the TC performance in some cases we mentioned. However, there are still limitations due to the variance itself that when the size is too large or too small, the variance will be relatively too small or too large. Extreme values for terms will obviously impact the TC performance. Therefore, we further optimized the formula by normalizing the ADF to reduce the effect caused by the extreme value of terms. First,ADFt,D is modified as shown in equation (9) and then using the normalization formula as shown in equation (10):(9)ADF′t,D=log1ADFt,D+1+1,(10)ADF′t,D=ADF′t,D−minADF′t,DmaxADF′t,D−minADF′t,D.
Based onADF″, another two novel formulas are designed as shown in equations (11) and (12), where α (default value is 1) is used as an optional weight proportion to adjust the importance of ADF″ in different cases.(11)IADFnormt,D=logD+1ADF″t,D+1,(12)IADFnorm+t,D=logD+1DFt,D+1∗ADF″t,D∗α.
Based on the above four proposed formulas of collection frequency based on IDF, we get four novel term weighting methods which are shown in equations (13)–(16):(13)TF–IADFt,d,D=TFt,d∗IADFt,D,(14)TF–IADF+t,d,D=TFt,d∗IADF+t,D,(15)TF–IADFnormt,d,D=TFt,d∗IADFnormt,D,(16)TF–IADFnorm+t,d,D=TFt,d∗IADFnorm+t,D.
As a result, the optimized text representation model of processing internet media reports is shown in Figure3. Four new calculation formulas are used to replace IDF of TF-IDF. And four novel term weighting methods are obtained to enhance the performance of processing unbalanced text collection.Figure 3
The flowchart of the text classification method based on the improved TF-IDF.


## 4. Case Study

To evaluate our proposed TWSs, experiments are carried out by using proposed methods in state-of-the-art classification algorithms on both Chinese and English corpuses. In this section, datasets used in experiments are briefly described. Then, algorithms utilized for the classification process and the measurements used for performance evaluation in this study are introduced. Finally, the experiment settings are also presented.


### 4.1. The Data Source

This study carried out experiments on three different datasets, i.e., standard dataset of English text, namely, Reuters-21578 corpus, classic dataset of Chinese text, namely, Fudan corpus, and a collection of Chinese internet media reports named Internet corpus, which were crawled off web and transformed into forms of Chinese textual document.


#### 4.1.1. Reuters-21578 Corpus

The Reuters-21578 corpus contains top-10 categories of Reuters-ModApte separately split which is most preferred in the TC domain [37]. In this study, multilabeled samples are removed since single-label-classification is focused. So, only 8 categories of 5607 training samples and 2270 test samples in Reuters-21578 were used in our experiments. The detail of data distribution of this corpus is shown in Figure 4 and Table 2.Figure 4
Data distribution of Reuters-21578 (unbalanced).Table 2
Reuters-21578 corpus.
No.Class labelTraining samplesTesting samples1Acq16347282Crude2771313Earn285910864Grain41105Interest198876Money-fx214937Ship115438Trade26992


#### 4.1.2. Fudan Corpus

The Fudan University TC corpus is from the Chinese NLP group in Department of Computer Information and Technology, Fudan University of China. There are 20 categories of which the data distributions are shown in Figure5 and Table 3. Similar to Reuters-21578, Fudan corpus is also an unbalanced dataset, but in Chinese language.Figure 5
Data distribution of Fudan corpus (unbalanced).Table 3
Fudan corpus.
No.Class labelTraining samplesTesting samples1C11-Space6406422C15-Energy32333C16-Electronics27284C17-Communication25275C19-Computer135713586C23-Mine33347C29-Transport57598C3-Art7407429C31-Enviornment1217121810C32-Agriculture1021102211C34-Economy1600160112C35-Law515213C36-Medical515314C37-Military747615C38-Politics1024102616C39-Sports1253125417C4-Literature333418C5-Education596119C6-Philosophy444520C7-History466468


#### 4.1.3. Internet Corpus

To test the performance of our proposed methods on Chinese internet media reports, some reports from the web are crawled, and this corpus is formed. There are six categories, namely, sport, education, tourism, traffic, tech, finance, and food. The data format is shown in Figure6. There are three parts in each instance of the test data which are the category index, the article content, and the total number of words. In this study, both balanced and unbalanced data distributions of this corpus are tried.Figure 6
Style of test data.


### 4.2. Classification Algorithms Used for Experiments and Measurements

Three popular classification algorithms, namely, naïve Bayes (NB), support vector machine (SVM), and random forests (RF), are utilized using our proposed methods and existing methods for a brief comparison. Introductions about these algorithms and measurements for evaluation are given as follows.


#### 4.2.1. Naïve Bayes

Naïve Bayes algorithm [38] is a well-known TC classifier based on Bayes’ assumption that the features are regarded to be independent from each other. In the TC process, document dk can be represented as a vector of terms (t1, t2, …, tn). The probability that dk belongs to a specific category ci can be calculated using equation (17). More details about the NB classifier can be accessed in [39]. In this study, a NB classifier is used for evaluating the text weighting performance.(17)Pci|dk=Pci∏j=1nPtjk|ciPdk.


#### 4.2.2. Support Vector Machine

SVM [40, 41] is one of the most preferred algorithms for TC and many other patter recognition problems. Since it is a learning algorithm, it can handle problems with high dimensions well. The main principle of the SVM is to create linear or nonlinear hyperplanes to separate positive and negative samples. SVM uses some samples in the training set (called support vectors) to create hyperplanes at locations maximizing margins between negative and positive samples. In this study, a classic SVM classifier is used for evaluating the text weighting performance.


#### 4.2.3. Random Forests

The random forest (RF) algorithm [42] is a parallelizable integration method which is one of the most preferred classifiers in the field of TC [43]. RF is composed of multiple decision trees. It is used to build a forest in a random way, which consists of many decision trees (DT). There is no correlation between each decision tree in the RF. After the RF is obtained, for each sample input, each decision tree in the forest is judged to see which category this sample belongs to, which category ultimately gets the most results, and which type of input prediction is. The architecture of RF is shown in Figure 7, where DT refers to the decision tree. In this study, a RF classifier is used for evaluating the text weighting performance.Figure 7
The architecture of random forests.


#### 4.2.4. Measurements for TC Performance

To evaluate the classification performance on the aforementioned datasets, accuracy, precision, recall, andF1 score are calculated for validation according to equations (18)–(21), where ci refers to a predefined category, while TPci refers to the number of documents which belong to ci resulting in ci, FNci refers to the number of documents which do not belong to ci resulting in ci, FPci refers to the number of documents which belong to ci not resulting in ci, and TN ci refers to the number of documents which do not belong to ci not resulting in ci. The relationship between them and the classification result are shown in Table 4.(18)Accuracyci=TPci+TNciTPci+TNci+FPci+FNci,(19)precisionci=TPciTPci+FPci,(20)recallci=TPciTPci+FNci,(21)F1ci=2∗precisionci∗recallciprecisionci+recallci.Table 4
Meanings of TP, TN, FP, and FN.
An instance of documents in the corpus (dk)SampledkResult inciNot a result inciBelongs tociTPciFPciDoes not belong tociFNciTNciIn multiclass classification problems, the overall performance can be measured by averaging the evaluation methods. Microaverage and macroaverage are used widely for this purpose. In this study, the microaveragedF1 (micro-F1) and macroaveraged F1 (macro-F1) measurements are also calculated to evaluate the experimental methods. The definition of macro-F1 and micro-F1 is as shown in equations (22) and (23):(22)macro−F1=1m∑i=1mF1ci,(23)micro−F1=2∗∑i=1mTPci2∗∑i=1mTPci+∑i=1mFPci+∑i=1mFNci.In cases of unbalanced distribution, it is better to use micro-F1 than macro-F1 since the data size of categories is not considered in micro-F1 score calculation.


### 4.3. Experiment Settings

In this study, we carried out three experiments on the aforementioned datasets. All experiments were implemented on a 64 bit Windows 10 computer with 8 GB internal storage. The experimental code was written in Python language using Scikit-learn (sk-learn). sk-learn is a commonly used third-party module in machine learning which encapsulates many commonly used machine learning algorithms such as regression, dimension reduction, clustering [44], and classification. For each dataset, in preprocessing, term weighting, and term extraction, term representation and classification were utilized. In preprocessing, all documents were segmented into words by the open-source tool Jieba, and stop words were removed in this process. After that, a vector space model (VSM) was used for term representation using words as terms. Term weighting methods including TF-IDF and proposed methods, i.e., TF-IADF, TF-IADFnorm, TF-IADF+, and TF-IADF+norm, were used here to form a final representation for each document. Finally, NB classifier, SVM classifier, and RF classifier were utilized for TC purpose. Combinations of different term weighting methods and different classification algorithms on different datasets were compared for a brief analysis.


## 4.1. The Data Source

This study carried out experiments on three different datasets, i.e., standard dataset of English text, namely, Reuters-21578 corpus, classic dataset of Chinese text, namely, Fudan corpus, and a collection of Chinese internet media reports named Internet corpus, which were crawled off web and transformed into forms of Chinese textual document.


### 4.1.1. Reuters-21578 Corpus

The Reuters-21578 corpus contains top-10 categories of Reuters-ModApte separately split which is most preferred in the TC domain [37]. In this study, multilabeled samples are removed since single-label-classification is focused. So, only 8 categories of 5607 training samples and 2270 test samples in Reuters-21578 were used in our experiments. The detail of data distribution of this corpus is shown in Figure 4 and Table 2.Figure 4
Data distribution of Reuters-21578 (unbalanced).Table 2
Reuters-21578 corpus.
No.Class labelTraining samplesTesting samples1Acq16347282Crude2771313Earn285910864Grain41105Interest198876Money-fx214937Ship115438Trade26992


### 4.1.2. Fudan Corpus

The Fudan University TC corpus is from the Chinese NLP group in Department of Computer Information and Technology, Fudan University of China. There are 20 categories of which the data distributions are shown in Figure5 and Table 3. Similar to Reuters-21578, Fudan corpus is also an unbalanced dataset, but in Chinese language.Figure 5
Data distribution of Fudan corpus (unbalanced).Table 3
Fudan corpus.
No.Class labelTraining samplesTesting samples1C11-Space6406422C15-Energy32333C16-Electronics27284C17-Communication25275C19-Computer135713586C23-Mine33347C29-Transport57598C3-Art7407429C31-Enviornment1217121810C32-Agriculture1021102211C34-Economy1600160112C35-Law515213C36-Medical515314C37-Military747615C38-Politics1024102616C39-Sports1253125417C4-Literature333418C5-Education596119C6-Philosophy444520C7-History466468


### 4.1.3. Internet Corpus

To test the performance of our proposed methods on Chinese internet media reports, some reports from the web are crawled, and this corpus is formed. There are six categories, namely, sport, education, tourism, traffic, tech, finance, and food. The data format is shown in Figure6. There are three parts in each instance of the test data which are the category index, the article content, and the total number of words. In this study, both balanced and unbalanced data distributions of this corpus are tried.Figure 6
Style of test data.


## 4.1.1. Reuters-21578 Corpus

The Reuters-21578 corpus contains top-10 categories of Reuters-ModApte separately split which is most preferred in the TC domain [37]. In this study, multilabeled samples are removed since single-label-classification is focused. So, only 8 categories of 5607 training samples and 2270 test samples in Reuters-21578 were used in our experiments. The detail of data distribution of this corpus is shown in Figure 4 and Table 2.Figure 4
Data distribution of Reuters-21578 (unbalanced).Table 2
Reuters-21578 corpus.
No.Class labelTraining samplesTesting samples1Acq16347282Crude2771313Earn285910864Grain41105Interest198876Money-fx214937Ship115438Trade26992


## 4.1.2. Fudan Corpus

The Fudan University TC corpus is from the Chinese NLP group in Department of Computer Information and Technology, Fudan University of China. There are 20 categories of which the data distributions are shown in Figure5 and Table 3. Similar to Reuters-21578, Fudan corpus is also an unbalanced dataset, but in Chinese language.Figure 5
Data distribution of Fudan corpus (unbalanced).Table 3
Fudan corpus.
No.Class labelTraining samplesTesting samples1C11-Space6406422C15-Energy32333C16-Electronics27284C17-Communication25275C19-Computer135713586C23-Mine33347C29-Transport57598C3-Art7407429C31-Enviornment1217121810C32-Agriculture1021102211C34-Economy1600160112C35-Law515213C36-Medical515314C37-Military747615C38-Politics1024102616C39-Sports1253125417C4-Literature333418C5-Education596119C6-Philosophy444520C7-History466468


## 4.1.3. Internet Corpus

To test the performance of our proposed methods on Chinese internet media reports, some reports from the web are crawled, and this corpus is formed. There are six categories, namely, sport, education, tourism, traffic, tech, finance, and food. The data format is shown in Figure6. There are three parts in each instance of the test data which are the category index, the article content, and the total number of words. In this study, both balanced and unbalanced data distributions of this corpus are tried.Figure 6
Style of test data.


## 4.2. Classification Algorithms Used for Experiments and Measurements

Three popular classification algorithms, namely, naïve Bayes (NB), support vector machine (SVM), and random forests (RF), are utilized using our proposed methods and existing methods for a brief comparison. Introductions about these algorithms and measurements for evaluation are given as follows.


### 4.2.1. Naïve Bayes

Naïve Bayes algorithm [38] is a well-known TC classifier based on Bayes’ assumption that the features are regarded to be independent from each other. In the TC process, document dk can be represented as a vector of terms (t1, t2, …, tn). The probability that dk belongs to a specific category ci can be calculated using equation (17). More details about the NB classifier can be accessed in [39]. In this study, a NB classifier is used for evaluating the text weighting performance.(17)Pci|dk=Pci∏j=1nPtjk|ciPdk.


### 4.2.2. Support Vector Machine

SVM [40, 41] is one of the most preferred algorithms for TC and many other patter recognition problems. Since it is a learning algorithm, it can handle problems with high dimensions well. The main principle of the SVM is to create linear or nonlinear hyperplanes to separate positive and negative samples. SVM uses some samples in the training set (called support vectors) to create hyperplanes at locations maximizing margins between negative and positive samples. In this study, a classic SVM classifier is used for evaluating the text weighting performance.


### 4.2.3. Random Forests

The random forest (RF) algorithm [42] is a parallelizable integration method which is one of the most preferred classifiers in the field of TC [43]. RF is composed of multiple decision trees. It is used to build a forest in a random way, which consists of many decision trees (DT). There is no correlation between each decision tree in the RF. After the RF is obtained, for each sample input, each decision tree in the forest is judged to see which category this sample belongs to, which category ultimately gets the most results, and which type of input prediction is. The architecture of RF is shown in Figure 7, where DT refers to the decision tree. In this study, a RF classifier is used for evaluating the text weighting performance.Figure 7
The architecture of random forests.


### 4.2.4. Measurements for TC Performance

To evaluate the classification performance on the aforementioned datasets, accuracy, precision, recall, andF1 score are calculated for validation according to equations (18)–(21), where ci refers to a predefined category, while TPci refers to the number of documents which belong to ci resulting in ci, FNci refers to the number of documents which do not belong to ci resulting in ci, FPci refers to the number of documents which belong to ci not resulting in ci, and TN ci refers to the number of documents which do not belong to ci not resulting in ci. The relationship between them and the classification result are shown in Table 4.(18)Accuracyci=TPci+TNciTPci+TNci+FPci+FNci,(19)precisionci=TPciTPci+FPci,(20)recallci=TPciTPci+FNci,(21)F1ci=2∗precisionci∗recallciprecisionci+recallci.Table 4
Meanings of TP, TN, FP, and FN.
An instance of documents in the corpus (dk)SampledkResult inciNot a result inciBelongs tociTPciFPciDoes not belong tociFNciTNciIn multiclass classification problems, the overall performance can be measured by averaging the evaluation methods. Microaverage and macroaverage are used widely for this purpose. In this study, the microaveragedF1 (micro-F1) and macroaveraged F1 (macro-F1) measurements are also calculated to evaluate the experimental methods. The definition of macro-F1 and micro-F1 is as shown in equations (22) and (23):(22)macro−F1=1m∑i=1mF1ci,(23)micro−F1=2∗∑i=1mTPci2∗∑i=1mTPci+∑i=1mFPci+∑i=1mFNci.In cases of unbalanced distribution, it is better to use micro-F1 than macro-F1 since the data size of categories is not considered in micro-F1 score calculation.


## 4.2.1. Naïve Bayes

Naïve Bayes algorithm [38] is a well-known TC classifier based on Bayes’ assumption that the features are regarded to be independent from each other. In the TC process, document dk can be represented as a vector of terms (t1, t2, …, tn). The probability that dk belongs to a specific category ci can be calculated using equation (17). More details about the NB classifier can be accessed in [39]. In this study, a NB classifier is used for evaluating the text weighting performance.(17)Pci|dk=Pci∏j=1nPtjk|ciPdk.


## 4.2.2. Support Vector Machine

SVM [40, 41] is one of the most preferred algorithms for TC and many other patter recognition problems. Since it is a learning algorithm, it can handle problems with high dimensions well. The main principle of the SVM is to create linear or nonlinear hyperplanes to separate positive and negative samples. SVM uses some samples in the training set (called support vectors) to create hyperplanes at locations maximizing margins between negative and positive samples. In this study, a classic SVM classifier is used for evaluating the text weighting performance.


## 4.2.3. Random Forests

The random forest (RF) algorithm [42] is a parallelizable integration method which is one of the most preferred classifiers in the field of TC [43]. RF is composed of multiple decision trees. It is used to build a forest in a random way, which consists of many decision trees (DT). There is no correlation between each decision tree in the RF. After the RF is obtained, for each sample input, each decision tree in the forest is judged to see which category this sample belongs to, which category ultimately gets the most results, and which type of input prediction is. The architecture of RF is shown in Figure 7, where DT refers to the decision tree. In this study, a RF classifier is used for evaluating the text weighting performance.Figure 7
The architecture of random forests.


## 4.2.4. Measurements for TC Performance

To evaluate the classification performance on the aforementioned datasets, accuracy, precision, recall, andF1 score are calculated for validation according to equations (18)–(21), where ci refers to a predefined category, while TPci refers to the number of documents which belong to ci resulting in ci, FNci refers to the number of documents which do not belong to ci resulting in ci, FPci refers to the number of documents which belong to ci not resulting in ci, and TN ci refers to the number of documents which do not belong to ci not resulting in ci. The relationship between them and the classification result are shown in Table 4.(18)Accuracyci=TPci+TNciTPci+TNci+FPci+FNci,(19)precisionci=TPciTPci+FPci,(20)recallci=TPciTPci+FNci,(21)F1ci=2∗precisionci∗recallciprecisionci+recallci.Table 4
Meanings of TP, TN, FP, and FN.
An instance of documents in the corpus (dk)SampledkResult inciNot a result inciBelongs tociTPciFPciDoes not belong tociFNciTNciIn multiclass classification problems, the overall performance can be measured by averaging the evaluation methods. Microaverage and macroaverage are used widely for this purpose. In this study, the microaveragedF1 (micro-F1) and macroaveraged F1 (macro-F1) measurements are also calculated to evaluate the experimental methods. The definition of macro-F1 and micro-F1 is as shown in equations (22) and (23):(22)macro−F1=1m∑i=1mF1ci,(23)micro−F1=2∗∑i=1mTPci2∗∑i=1mTPci+∑i=1mFPci+∑i=1mFNci.In cases of unbalanced distribution, it is better to use micro-F1 than macro-F1 since the data size of categories is not considered in micro-F1 score calculation.


## 4.3. Experiment Settings

In this study, we carried out three experiments on the aforementioned datasets. All experiments were implemented on a 64 bit Windows 10 computer with 8 GB internal storage. The experimental code was written in Python language using Scikit-learn (sk-learn). sk-learn is a commonly used third-party module in machine learning which encapsulates many commonly used machine learning algorithms such as regression, dimension reduction, clustering [44], and classification. For each dataset, in preprocessing, term weighting, and term extraction, term representation and classification were utilized. In preprocessing, all documents were segmented into words by the open-source tool Jieba, and stop words were removed in this process. After that, a vector space model (VSM) was used for term representation using words as terms. Term weighting methods including TF-IDF and proposed methods, i.e., TF-IADF, TF-IADFnorm, TF-IADF+, and TF-IADF+norm, were used here to form a final representation for each document. Finally, NB classifier, SVM classifier, and RF classifier were utilized for TC purpose. Combinations of different term weighting methods and different classification algorithms on different datasets were compared for a brief analysis.


## 5. Results’ Analysis and Discussion

### 5.1. Results on the Internet Corpus

In this part, the experiments on both a balanced dataset and an unbalanced dataset of internet media reports in the Chinese language are carried out.


#### 5.1.1. Balanced Dataset

In this dataset, there are 1000 training samples and 1000 test samples in each category. This is an exactly balanced dataset. SVM, NB, and RF classifiers are utilized as TWS for a comparison with TF-IDF. The overall performance of the proposed TF-IADF outperformed all other methods in SVM and RF classifiers as shown in Figure8. The details of experimental results are shown in Table 5 in that the proposed TF-IADF+norm demonstrates better performance than TF-IDF in all cases. Furthermore, all proposed methods outperformed the TF-IDF, in some cases, respectively.Figure 8
Data distribution of the Fudan corpus (unbalanced).Table 5
Experiments on the balanced dataset of the internet corpus.
Balanced datasetTF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-IADF+norm (%)micro-F1 (SVM)94.2394.5494.2394.3294.35macro-F1 (SVM)94.1694.4894.1894.2594.28micro-F1 (RF)84.5186.3785.2684.2385.52macro-F1 (RF)84.4186.2685.1684.0885.41micro-F1 (NB)92.1391.7391.4292.2692.33macro-F1 (NB)92.0591.6691.3692.1892.24For the SVM classifier, the overall classification effect is better than the other two classifiers, and the micro-F1 value is over 94%. TF-IADF has achieved the best effect of 94.45% (increased by 0.31% than TF-IDF). For the RF classifier, TF-IADF achieves the best effect. In addition, TF-IADFnorm and TF-IADF+norm also come with some improvement. The micro-F1 value of TF-IDF is 84.51%, while that of TF-IADF+norm is 85.52%, and that of TF-IADFnorm is 85.26%. The micro-F1 value of TF-IADF reaches to 86.37%, which is the best among all methods, an improvement of 1.86% from TF-IDF. For the NB classifier, both TF-IADF+ and TF-IADF+norm show improvements for the micro-F1 value of TF-IADF+ and TF-IADF+norm reaching 92.26% and 92.33%, while that of TF-IDF is only 92.13%. TF-IADF+norm achieves the best effect with an increase of 1.2% over TF-IDF. In the classification results, we notice that the precision rate and recall rate of finance are relatively low, at only 80% and 85%, respectively. The reason may be that the terms of this category are not obvious enough, and the scope involved is relatively wide, which may cover some contents from tourism and traffic categories, resulting in the poor classification effect of the whole category.The results show that, in the case of balanced dataset which is not focused by our design, the proposed methods can improve the classifiers to a certain extent even though the improvement range is not so obvious.


#### 5.1.2. Unbalanced Dataset

In this section, we investigate how TC performance is impacted by unbalanced datasets. Dataset here is specially designed. The size of food is increased to 5000, and the sizes of other categories are kept the same as in the balanced dataset (1000) except sport. For sport, the size is gradually reduced from 400 to 50, decreasing 50 at a time.Definition 2 (balance ratio).
It is the proportion between the sizes of the category with the smallest size and the category with the largest size.
In this section, we call the proportion between the size of sport and that of food as balance ratio which drops from 8% to 1% with the decrease of sport’s size. Experiments by using the SVM classifier with TF-IDF and our proposed methods are carried out. The experimental results show that performances on categories other than sport and food are basically the same with the balance ratio changing. However, the recall andF1 score of sport and the precision of food are impacted heavily by decreasing the balance ratio. The details are shown in Tables 6–8. As shown, the balance ratio is decreasing, with the recall and F1 score of sport also decreasing in all methods. However, the proposed TF-IADF and TF-IADFnorm outperformed TF-IDF in all cases. Especially, when the balance ratio decreased to an extreme value (1%), TF-IADFnorm came with a recall of 0.7168 which is almost 170% of that of TF-IDF. Due to the balance ratio changes, precision of the category with a relatively large size (food) was also impacted which can be seen from Table 8.Table 6
Recall (sport).
Balance ratio (%)TF-IDFTF-IADFTF-IADFnormTF-IADF+TF-IADF+norm80.94790.95810.95810.93150.916270.93760.94580.94790.89880.888660.91820.93970.94170.88240.869150.90180.93460.94070.86500.850740.87630.92430.93150.82620.812930.82720.89370.91410.74640.730120.74230.82520.85070.60120.591010.42740.66260.71680.25670.2413Table 7
F1 score (sport).
Balance ratio (%)TF-IDFTF-IADFTF-IADFnormTF-IADF+TF-IADF+norm80.97320.97860.97760.96450.956270.96780.97220.97320.94670.941060.95740.96890.97000.93750.930050.94840.96620.96940.92760.919340.93410.96070.96450.90480.896830.90540.94380.95510.85480.844020.85210.90420.91930.75100.742910.59890.79710.83500.40850.3888Table 8
Precision (food).
Balance ratio (%)TF-IDFTF-IADFTF-IADFnormTF-IADF+TF-IADF+norm80.87260.91850.93050.83100.778870.87110.91600.92880.82410.771660.86880.91600.92960.82130.766850.86500.91430.92960.81460.758040.85830.91260.92790.80410.744430.84800.91010.92530.78750.717020.82750.88490.90920.75170.672810.78130.87250.89760.67280.5691Definition 3 (decline ratio).
It is the ratio of the current value to the initial value in a declining trend.
To make the relationship between the balance ratio and the performance clear, the decline ratio is calculated. It is the ratio of the value for each balance ratio to its initial value (value at 8%). This can be seen in Figure9, where the ordinate refers to the decline ratio of the corresponding performance factor. As it is shown, with the balance ratio decreasing, the growth rate of decline ratio becomes faster and faster. It is obvious that, for datasets with extreme categories such as food (relatively too large) and sport (relatively too small), performances of TF-IADF and TF-IADFnorm are more stable than TF-IDF.
The overall performance on this dataset is impacted by the balance ratio also. The details of micro-F1 and macro-F1 are shown in Tables 9 and 10, respectively. The proposed TF-IADF and TF-IADFnorm outperformed TF-IDF in all cases. To make it clear how balance ratio impacts the performance, we calculated the decline ratio which is shown in Figure 10. As it is shown, with the decrease of balance ratio, both micro-F1 and macro-F1 decrease gradually. For example, when looking at micro-F1, TF-IADFnorm was with a decrease of just 3.65%, while TF-IDF was with a decrease of 8.06% which is more than twice of that of the proposed TF-IADFnorm, meaning the performance of TF-IADFnorm is much more stable than that of TF-IDF in this dataset.
Even though TF-IADF+ and its variance do not achieve improvement in this experiment, TF-IADF and its variance outperformed TF-IDF significantly. Furthermore, it can be seen from Figure 10 that our proposed TF-IADF and TF-IADFnorm are not only numerically better but also more stable than TF-IDF. Therefore, considering ADF in the term weighting method can actually improve the performance of text classification considerably in unbalanced cases.Figure 9
The decline ratio of the performance with balance ratio changing.
(a)vpos: 38.5pt(b)(c)Table 9
Micro-F1.
Balance ratio (%)TF-IDFTF-IADFTF-IADFnormTF-IADF+TF-IADF+norm80.92920.93720.93720.92260.914070.92780.93540.93600.91790.909460.92500.93450.93520.91540.906350.92260.93380.93510.91290.903240.91890.93230.93380.90730.897730.91190.92790.93130.89600.885920.89940.91790.92220.87530.865810.85420.89460.90290.82580.8156Table 10
Macro-F1.
Balance ratio (%)TF-IDFTF-IADFTF-IADFnormTF-IADF+TF-IADF+norm80.92870.93660.93660.92250.914770.92730.93490.93550.91790.910260.92460.93410.93480.91540.907250.92230.93330.93460.91300.904240.91860.93190.93330.90740.898730.91150.92750.93090.89560.886820.89850.91740.92170.87260.864910.84430.89220.90140.80550.7987Figure 10
The decline ratio of the overall performance with balance ratio changing.
(a)vpos: 11.2pt(b)


#### 5.1.3. Analysis

For this corpus, the TF-IADF+ and TF-IADF+norm methods are more suitable for the NB classifier, and TF-IADF+norm is the best. For the RF and SVM classifiers, TF-IADF+norm and TF-IADF are more suitable. In addition, TF-IADF is better in case of a balanced dataset, while TF-IADFnorm is better in case of an unbalanced dataset. This means that the processed TF-IADFnorm method is more sensitive and stable in the case of unbalanced datasets, while TF-IADF improves the classification effect more when the datasets are relatively balanced. For several mathematical models proposed by different algorithms, it can be concluded that the formula suitable for different algorithms may be different, and for the improvement of the corresponding algorithm effect, it can also be concluded that the ADF index has improved the effect of text classification, which confirms the conjecture; especially, when the dataset is not evenly distributed, the effect of text classification is more stable.


### 5.2. Results on the Fudan Corpus

#### 5.2.1. Results’ Analysis

Table11 and Figure 11 show the micro-F1 and macro-F1 scores obtained on Fudan corpus using SVM, RF, and NB algorithms with different TWSs, while Tables 12–14 show the detailed results.Table 11
Overall performances on the Fudan corpus.
Fudan corpusTF-IDFTF-IADFTF-IADFnormTF-IADF+ (%)TF-IADF+norm (%)micro-F1 (SVM)50.0254.5754.9348.8547.90macro-F1 (SVM)90.3191.0490.9490.2489.82micro-F1 (RF)48.5751.4348.8049.7851.64macro-F1 (RF)80.3281.9580.8381.2079.85micro-F1 (NB)65.4462.2859.9466.3966.97macro-F1 (NB)88.9988.2087.5589.3189.38Figure 11
Overall performances of classification on the Fudan corpus.Table 12
Performance on the Fudan corpus using the SVM classifier.
SVMPrecisionF1 scoreRecallTF-IDFTF-IADFTF-IADFnormTF-IADF+TF-IADF+normTF-IDFTF-IADFTF-IADFnormTF-IADF+TF-IADF+normTF-IDFTF-IADFTF-IADFnormTF-IADF+TF-IADF+normC1196.1095.9896.1396.24%96.24%92.0692.9992.8491.75%91.75%94.0394.4694.4593.94%93.94%C15100.00100.00100.00100.00%100.00%9.099.099.099.09%9.09%16.6716.6716.6716.67%16.67%C16100.00100.00100.00100.00%100.00%3.573.573.573.57%3.57%6.906.906.906.90%6.90%C17100.00100.00100.00100.00%100.00%18.5222.2222.2218.52%18.52%31.2536.3636.3631.25%31.25%C1995.6495.9395.5395.50%95.56%98.5398.9799.1298.53%98.31%97.0697.4397.2996.99%96.92%C23100.00100.00100.00100.00%100.00%5.885.885.885.88%5.88%11.1111.1111.1111.11%11.11%C29100.0096.1595.65100.00%100.00%8.4842.3737.295.09%3.39%15.6358.8253.669.68%6.56%C387.1088.8589.1787.18%86.90%95.5595.5595.4295.28%94.74%91.1392.0892.1991.05%90.65%C3195.6196.9797.7795.80%95.86%96.6397.2196.9697.29%97.04%96.1297.0997.3696.54%96.45%C3294.1794.1093.4794.43%94.37%96.3896.6796.5896.18%95.21%95.2695.3795.0095.30%94.79%C3490.2090.1890.2389.67%87.03%94.8895.1994.6394.88%94.75%92.4892.6292.3892.20%90.73%C35100.0095.2492.31100.00%100.00%23.0838.4646.1511.54%7.69%37.5054.8061.5420.69%14.29%C36100.00100.00100.00100.00%100.00%1.8913.2115.091.89%1.89%3.7023.3326.233.70%3.70%C37100.00100.00100.00100.00%100.00%5.265.265.265.26%5.26%10.0010.0010.0010.00%10.00%C3882.6582.3682.7883.19%82.66%95.6194.6494.6495.52%95.22%88.6688.0788.3188.93%88.50%C3985.3688.6987.6884.86%85.93%96.7396.9796.4996.97%96.89%90.6992.6591.8890.51%91.08%C4100.00100.00100.00100.00%100.00%5.885.885.885.88%2.94%11.1111.1111.1111.11%5.71%C566.6766.6766.6766.67%66.67%3.283.283.283.28%3.28%6.256.256.256.25%6.25%C688.8988.8990.0088.89%88.89%17.7817.7820.0017.78%17.78%29.6329.6332.7329.63%29.63%C782.7881.3080.9982.43%81.70%68.8072.4473.7268.16%65.81%75.1576.6177.1874.62%72.90%Table 13
Performance on the Fudan corpus using the RF classifier.
RFPrecisionRecallF1 scoreTF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-IADF+norm (%)TF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-+norm (%)TF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-IADF+norm (%)C1182.9490.8585.5287.6186.5082.5685.0579.1378.1983.8082.7587.8582.2082.6385.13C1533.3380.0044.4466.6783.3315.1512.1212.1212.1215.1520.8321.0519.0520.5125.64C1640.0050.0057.1430.7757.147.1417.8614.2914.2914.2912.1226.3222.8619.5122.86C1755.5650.0038.7158.3361.9137.0437.0444.4425.9348.1544.4442.5541.3835.9054.17C1989.9090.5587.6186.3188.2496.3296.6196.3297.0596.6993.0093.4891.7691.3792.27C2343.7538.4633.3334.7850.0020.5914.718.8223.5314.7128.0021.2813.9528.0722.73C2943.5969.5750.0067.7460.8728.8127.1222.0335.5947.4634.6939.0230.5946.6753.33C375.7774.8973.4175.1471.7386.3990.4390.4389.6287.8780.7381.9381.0481.7578.98C3192.0493.1791.7792.8693.4990.2391.7991.5490.7288.4291.1392.4791.6691.7890.89C3284.1087.0186.3088.0881.9684.3486.5083.8686.7980.9284.2286.7585.0687.4381.44C3474.4975.8573.5975.0372.6188.2690.2689.8289.6988.0780.7982.4380.9081.7179.59C3566.6777.7856.5268.7560.007.6913.4625.0021.1517.3113.7922.9534.6732.3526.87C3657.1466.6751.8575.0051.5222.6430.1926.4228.3032.0832.4341.5635.0041.1039.54C3742.8642.8645.4613.6466.6711.8415.796.583.9510.5318.5623.0811.496.1218.18C3865.6666.4570.0066.8666.8176.2277.9780.5180.0276.5170.5571.7574.8972.8571.33C3984.7986.2987.5487.6586.6784.4585.3384.0582.6281.4284.6285.8185.7685.0683.96C4100.0075.00100.00100.00100.005.888.825.885.888.8211.1115.7911.1111.1116.22C555.5685.7150.0062.5077.788.209.8411.488.2011.4814.2917.6518.6714.4920.00C690.9184.6283.3363.6490.0022.2224.4422.2215.5620.0035.7137.9335.0925.0032.73C769.5470.4873.0477.9165.0525.8625.0017.9527.1425.8637.7036.9128.8240.2537.00Table 14
Performance on the Fudan corpus using the NB classifier.
NBPrecisionRecallF1 scoreTF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+TF-IADF+norm (%)TF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-IADF+norm (%)TF-IDF (%)TF-IADF (%)TF-IADFnormTF-IADF+ (%)TF-IADF+norm (%)C1194.9894.5794.5394.9994.8391.2889.5688.7991.5991.4393.0992.0091.5793.2693.10C1593.75100.00100.0093.7593.7545.4630.3027.2745.4645.4661.2246.5142.8661.2261.22C1666.67100.00100.0075.0080.007.143.573.5710.7114.2912.906.906.9018.7524.24C1787.5086.6783.3393.7594.1251.8548.1537.0455.5659.2665.1261.9151.2869.7772.73C1996.3896.0295.8796.4696.3896.1796.0295.8096.1796.1796.2896.0295.8496.3196.28C2393.3392.31100.0094.7490.0041.1835.2920.5952.9452.9457.1451.0634.1567.9366.67C2986.3687.8887.1086.3688.8964.4149.1545.7664.4167.8073.7963.0460.0073.7976.92C382.9481.8181.1983.4383.5194.3495.1595.4294.3494.2188.2787.9887.7388.5588.54C3197.5997.8997.7897.6997.6992.9491.3890.4893.7693.7695.2194.5293.9995.6995.69C3291.9290.7090.3591.6891.6090.2288.7587.9790.6190.7191.0689.7189.1491.1491.15C3485.0582.8681.3285.7686.2092.7692.6992.4492.9492.8888.7487.5086.5289.2189.42C3587.5086.9687.5086.9686.9640.3938.4640.3938.4638.4655.2653.3355.2653.3353.33C3690.9188.8993.3390.9190.9137.7430.1926.4237.7437.7453.3345.0741.1853.3353.33C3778.8584.0981.4078.4378.8553.9548.6846.0552.6353.9564.0661.6758.8262.9964.06C3884.3083.1282.1685.4685.5592.1192.5992.0192.2092.3088.0387.6086.8188.7088.80C3989.3088.2987.3089.3989.3993.8693.2292.6694.0294.0291.5290.6989.9091.6491.64C450.00100.00100.0050.0050.005.885.885.885.885.8810.5311.1111.1110.5310.53C580.0075.0071.4387.5077.7813.129.848.2011.4811.4822.5417.3914.7120.2920.00C690.0090.0090.0090.0090.0020.0020.0020.0020.0020.0032.7332.7332.7332.7332.73C769.1071.7672.4968.9769.1766.8866.2464.7468.3869.0267.9768.8968.4068.6769.09For the NB classifier, TF-IADF+norm has the best performance, which is 89.38%, an increase of 0.39% compared to TF-IDF. Meanwhile, TF-IADF+ also gets an increase of 0.32%. This means for Chinese text datasets, these two models are more suitable for a NB classifier and can achieve an improvement on the overall classification effect. When comparing the specific measurements which are shown in Figure 12 and more details can be seen in Table 14, TF-IADF+norm has achieved a lot of the highest performance-score items. Furthermore, the difference between performance scores of TF-IDF and those of TF-IADF+norm is relatively small in categories, where TF- IADF+norm is not as good as TF-IDF. For example, in C32 and C35, where TF-IDF achieves the best precision score, the difference between the precision score of TF-IDF and that of TF-IADF+norm is less than 1%. However, in categories such as C17, where TF-IADF+norm achieves the highest precision score, the difference between that and precision score of TF-IDF is more than 6% which is six times of the difference occurring in categories where TF-IDF achieves the higher score. In fact, in C17, when comparing TF-IADF+norm to TF-IDF, the precision is improved by 6.62%, and the recall score of C17 is also increased by 7.41%, which is obvious. In an overall view of the F1 scores, it can be noticed that, in all the 20 categories except C35 and C5, the scores of TF-IADF+norm are not lower than those of TF-IDF. Especially, in cases of TF-IADF+norm with higher elevation such as C16, the F1 score is increased by nearly 10%.Figure 12
Detailed performances on the Fudan corpus using the NB classifier.
(a)(b)(c)For the RF classifier, the micro-F1 score obtained by TF-IADF is 81.95%, which is 1.63% higher than that obtained by TF-IDF. Meanwhile, TF-IADF+ and TF-IADFnorm also have achieved improvements in different extents. It is known that RF algorithm has certain randomness, but our proposed TF-IADF method is more stable and has achieved better results which can be seen in Figure 13. When looking into detailed results as shown in Table 13, in some categories, such as C29 and C5, the precision score has been improved by 25.98% and 30.16%, respectively, and the recall score has remained basically unchanged. For the performance of F1 score, in some categories, such as C16 and C36, the score obtained by TF-IADF is about 10% higher than that obtained by TF-IDF. Furthermore, there are only three categories where F1 of TF-IADF is not as good as TF-IDF.Figure 13
Detailed performances on the Fudan corpus using the RF classifier.
(a)(b)(c)For the SVM classifier, it is still the case that TF-IADF and TF-IADFnorm have achieved improved performances. When comparing with TF-IDF in the micro-F1 score, TF-IADF has increased by 0.73%, while TF-IADFnorm has increased by 0.63%. It can be seen in Figure 14 that, for the SVM classifier, these two improved methods are more stable and have achieved some improvements. As shown in Table 12, there are many categories where the precision scores are very high, even up to 1. It can easily be seen in Figure 14 that the size (data amount) of those categories is very small. For example, there are only 32 training samples in C15. The reason is that due to the small eigenvalues, no other categories being considered as this category are responsible for the high precision score. However, it can also be concluded from the detailed results that, for most of those categories with a high precision but a small size, the recall score is significantly reduced; that is to say, many documents are assigned to the wrong categories, which may be due to the impact caused by a small number of features. For example, the recall score of C15 is only 0.09. TF-IADF and TF-IADFnorm perform better in most of the categories, especially in C29, where the size is small. Although the precision scores obtained by the two methods are decreased by about 4%, the recall scores are improved by about 30% which is significant. In addition, there are only two categories where F1 scores of TF-IADF are lower than those of TF-IDF, while for most categories, TF-IADF is better.Figure 14
Detailed performances on the Fudan corpus using the SVM classifier.
(a)(b)(c)


#### 5.2.2. Further Discussion

Due to the unbalanced distribution of this corpus, all categories can be roughly divided into three groups by their size for specific analysis. The first group refers to categories with size in the range of 0 to 100, the second refers to categories with size in the range of 101 to 1000, and the third group refers to categories with size over 1000. First of all, for the first group, due to the insufficient size of training data, the same property is reflected in all term weighting methods, i.e., high precision and low recall scores, such as C16. Taking SVM combined with TF-IDF as an example, the precision score reaches 100%, while the recall score is only 3.57%, resulting in poor classification effect. In short, it assigns a few samples correctly, while a large number of test samples are assigned to the wrong categories. And the performance is similar in the RF and NB classifiers. This is the impact caused by the unbalanced distribution and the small size of the training data. For the second group, the number of training sets has been improved to a higher level. For example, in C11 when using the SVM classifier with TF-IDF, the precision score and recall score are 96.10% and 92.06%, respectively, which is a significant improvement compared to the first group. For the third category, such as C19, the precision score and recall score are 95.64% and 98.53%, respectively. It can be concluded that the precision score in categories with large training set size is relatively low, while the recall rate is relatively high. In categories with a small training set size, it will have better precision but very low recall score. In these kinds of conditions, our proposed methods will demonstrate a similar but more stable performance compared to TF-IDF. Taking C29 as an example, there are only 57 training samples. Comparing TF-IDF with TF-IADF using the SVM classifier, the precision scores are 100% to 96.15%, while the recall scores are 8.48% to 42.37%. It can be seen that although the accuracy rate of our algorithm is slightly reduced, the recall rate is greatly increased by nearly 500%, and theF1 score is also greatly improved, from 15.63% to 58.82%, showing a very obvious improvement.In the Fudan corpus, a similar phenomenon with the internet corpus can be seen. For example, the TF-IADF+norm method is the best one in the NB classifier, while TF-IADF+ is also better than TF-IDF. And with RF and SVM classifiers, both TF-IADF and TF-IADFnorm achieve a relatively stable performance. The difference is that TF-IADF achieves the best effect in the Fudan unbalanced dataset. It is also noticed that although the micro-F1 score of the SVM classifier is the best, the macro-F1 score, which is about 50%, is not as good as that of the NB classifier which is over 65%. That is to say, although the overall accuracy of the SVM classifier is high, the effect of the NB classifier is better when each category is regarded as equally important.


### 5.3. Results on Reuters-21578

The overall performance on Reuters-21758 of all methods is shown in Table15, while Figure 15 shows a brief comparison between all proposed methods and the well-known TF-IDF method when combining with different classifiers. More detailed results are shown in Tables 16–18.Table 15
Overall performances on Reuters-21578.
TF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-IADF+norm (%)micro-F1 (SVM)91.0091.4191.8690.5990.23macro-F1 (SVM)79.6380.3381.9778.4275.77micro-F1 (RF)87.4585.2387.7787.4188.14macro-F1 (RF)69.9059.0365.4367.9167.76micro-F1 (NB)89.8290.4190.3289.4189.36macro-F1 (NB)76.2277.8677.7776.2175.95Figure 15
Performance of the NB classifier on Reuters-21578 (precision, recall, andF1 score).Table 16
Performance on Reuters-21758 using the SVM classifier.
SVMPrecisionRecallF1 scoreTF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-IADF+norm (%)TF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TFIADF+norm (%)TF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-IADF+norm (%)Acq90.1791.4393.0390.0690.4292.2491.9592.1092.3992.2491.1991.6992.5691.2191.32Crude90.3989.7289.9192.6392.5577.6979.3480.9972.7371.9083.5684.2185.2281.4880.93Earn92.0692.0092.4891.3590.2197.4297.6997.6997.5197.8894.6694.7695.0294.3393.89Grain100.00100.00100.00100.00100.0040.0040.0050.0040.0030.0057.1457.1466.6757.1446.15Interest91.3090.0088.2491.1190.4856.0060.0060.0054.6750.6769.4272.0071.4368.3364.96Moneyfx79.7184.0680.0079.1079.0363.2266.6773.5660.9256.3270.5174.3676.6568.8365.77Ship100.00100.0095.8395.6595.2463.8961.1163.8961.1155.5677.9775.8676.6774.5870.18Trade90.6389.8088.7889.5892.4794.5795.6594.5793.4893.4892.5592.6391.5891.4992.97Table 17
Performance on Reuters-21758 using the RF classifier.
RFPrecisionRecallF1 scoreTF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-IADF+norm (%)TF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-IADF+norm (%)TF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-IADF+norm (%)Acq86.5282.2585.4181.7788.0690.3790.5292.5394.1190.0988.4186.1888.8387.5189.06Crude92.6893.2484.8891.2586.8162.8157.0360.3360.3365.2974.8870.7770.5372.6474.53Earn89.8987.7890.4893.1189.7496.8696.8696.5894.8397.6993.2492.1093.4493.9693.55Grain100.00100.0033.33100.0050.0030.0010.0010.0020.0010.0046.1518.1815.3933.3316.67Interest75.5174.2978.0072.7386.0549.3334.6752.0053.3349.3359.6847.2762.4061.5462.71Moneyfx75.4773.1782.6972.4171.4345.9834.4849.4348.2845.9857.1446.8861.8757.9355.94Ship90.0090.0092.31100.0090.4850.0025.0033.3341.6752.7864.2939.1348.9858.8266.67Trade72.7378.2184.8880.2381.2578.2666.3079.3575.0084.7875.3971.7782.0277.5382.98Table 18
Performance on Reuters-21758 using the NB classifier.
NBPrecisionRecallF1 scoreTF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-IADF+norm (%)TF-IDF (%)TF-IADF (%)TF (%)TF-IADF+ (%)TF-IADF+norm (%)TF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-IADF+norm (%)Acq91.8391.7491.6091.1291.3492.1092.5392.3991.3890.9591.9792.1391.9991.2591.15Crude85.7188.0788.0786.4987.1679.3479.3479.3479.3478.5182.4083.4883.4882.7682.61Earn92.6592.6892.6792.1491.8495.4895.8595.7595.2095.5794.0494.2494.1993.6493.67Grain71.4371.4371.4371.4371.4350.0050.0050.0050.0050.0058.8258.8258.8258.8258.82Interest78.2684.4482.6178.2679.5548.0050.6750.6748.0046.6759.5063.3362.8159.5058.82Moneyfx76.0077.2277.2277.0377.0365.5270.1270.1265.5265.5270.3773.4973.4970.8170.81Ship85.1992.0092.0088.4685.1963.8963.8963.8963.8963.8973.0275.4175.4174.1973.02Trade70.5974.3474.3469.7569.7591.3091.3091.3090.2290.2279.6281.9581.9578.6778.67For the SVM classifier, TF-IADFnorm comes with the best performance which is 0.86% higher than TF-IDF in the micro-F1 score. According to the detailed results shown in Table 15 and Figure 16, the improvement of TF-IADFnorm is mainly contributed by the recall score, which is greatly improved in the category of crude, grain, interest, and money-fx. Especially in the category of money-fx, the recall score has increased by 10.35%. The other categories also improved in different extents, namely, crude increased by 3.31%, interest increased by 4%, and grain increased by 10%. From the perspective of the F1 score, in the total eight categories, the max F1 score of five categories is obtained by TF-IADFnorm, and the largest increase is obtained in the category of grain, where the F1 score increased from 57.14% of TF-IDF to 66.67% of TF-IADFnorm, close to 10%.Figure 16
Performance of the SVM classifier on Reuters-21578 (precision, recall, andF1 score).
(a)(b)(c)For the NB classifier, the results are different from those on the Chinese dataset, where TF-IADF+ and TF-IADF+norm achieved better performance, and performances of TF-IADF+ and TF-IADF+norm were worse on this corpus. However, the proposed TF-IADF and TF-IADFnorm showed an improvement, the micro-F1 score of which is 0.59% and 0.5% higher than that of TF-IDF, respectively. As shown in Table 16 and Figure 17, TF-IADF has achieved the highest F1 score among all methods in all categories of this corpus, and the maximum improvement, 3.83%, occurs in the category of interest. Specifically, in terms of precision, we see that the four categories of cloud, interest, ship, and trade have improved significantly, with an improvement of 2.36%, 4.35%, 6.82%, and 3.75%, respectively. In terms of the recall score, the increase in the money-fx category is 4.60%.Figure 17
Performance of the NB classifier on Reuters-21578 (precision, recall, andF1 score).
(a)(b)(c)For the RF algorithm, TF-IADF+norm and TF-IADFnorm have improved the effect, micro-F1 score of which is 0.69% and 0.32% higher than that of TF-IDF, respectively. TF-IADF, which has better performance in the Chinese dataset, has worse performance on this corpus. It can be seen in Table 17 and Figure 18 that the better performance of TF-IADF+norm is mainly due to the higher recall score. Comparing with TF-IDF, the recall score obtained in the trade category is 6.52% higher, and in the crude category, it is 2.48% higher. In terms of precision, it has achieved good performance in the categories of interest and trade, with an increase of 10.54% and 8.52%, respectively. On the side of F1 score, the trade category improved significantly, up to 7.59%. However, we also see a decrease in the performance on the categories of grain and money-fx. This may be due to the small size of the test dataset, which changes greatly, and so, it is not easy to draw more accurate conclusions.Figure 18
Performance of the RF classifier on Reuters-21578 (precision, recall, andF1 score).
(a)(b)(c)According to all experimental results on Reuters-21578, the proposed TF-IADFnorm outperformed TF-IDF in nearly all conditions except the micro-F1 score of the RF classifier. In fact, the RF classifier’s performance was worse among all classifiers, suggesting that the RF classifier may not be suitable for this corpus.


### 5.4. Discussion

The experimental results obtained on the English dataset (Reuters) are somehow different from those on the Chinese dataset, and the most effective combination of a term weighting method and classification algorithm is different. For example, TF-IADF+norm is suitable for the Chinese internet corpus using the NB algorithm, whereas TF-IADF performs best in the English unbalanced dataset. It can be inferred from the experimental results that all of the proposed algorithms can generally always be combined with a suitable mathematical model that shows a better performance than the original TF-IDF. For example, TF-IADF+norm performs better in the RF algorithm, while TF-IADFnorm performs better in the SVM. The best combinations concluded from the experiments are shown in Table 19. For the two Chinese datasets in the experiment, we draw the following conclusions: (1) for the NB algorithm, TF-IADF+ and TF-IADF+norm are more suitable, and TF-IADF+norm can achieve better performance whether in balanced or unbalanced datasets; (2) for the RF algorithm and SVM algorithm, TF-IADF and TF-IADFnorm can achieve a relatively stable improvement effect. In the experiments with the internet corpus, it can be concluded that TF-IADF has a better improvement effect when the dataset is relatively balanced, and TF-IADFnorm has a better classification effect when the dataset is unbalanced. However, in the Fudan corpus (unbalanced), TF-IADF has achieved better performance than TF-IADFnorm, while TF-IADFnorm has also improved. It is possible that TF-IADF is more suitable for RF and SVM algorithms when there are many categories which are unbalanced in the corpus.Table 19
The best combinations of TWSs with classifiers.
Dataset/classifierNBRFSVMBalanced Chinese datasetTF-IADF+normTF-IADFTF-IADFUnbalanced Chinese datasetTF-IADF+normTF-IADF, TF-IADFnormTF-IADF, TF-IADFnormUnbalanced English datasetTF-IADFTF-IADF+normTF-IADFnorm


## 5.1. Results on the Internet Corpus

In this part, the experiments on both a balanced dataset and an unbalanced dataset of internet media reports in the Chinese language are carried out.


### 5.1.1. Balanced Dataset

In this dataset, there are 1000 training samples and 1000 test samples in each category. This is an exactly balanced dataset. SVM, NB, and RF classifiers are utilized as TWS for a comparison with TF-IDF. The overall performance of the proposed TF-IADF outperformed all other methods in SVM and RF classifiers as shown in Figure8. The details of experimental results are shown in Table 5 in that the proposed TF-IADF+norm demonstrates better performance than TF-IDF in all cases. Furthermore, all proposed methods outperformed the TF-IDF, in some cases, respectively.Figure 8
Data distribution of the Fudan corpus (unbalanced).Table 5
Experiments on the balanced dataset of the internet corpus.
Balanced datasetTF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-IADF+norm (%)micro-F1 (SVM)94.2394.5494.2394.3294.35macro-F1 (SVM)94.1694.4894.1894.2594.28micro-F1 (RF)84.5186.3785.2684.2385.52macro-F1 (RF)84.4186.2685.1684.0885.41micro-F1 (NB)92.1391.7391.4292.2692.33macro-F1 (NB)92.0591.6691.3692.1892.24For the SVM classifier, the overall classification effect is better than the other two classifiers, and the micro-F1 value is over 94%. TF-IADF has achieved the best effect of 94.45% (increased by 0.31% than TF-IDF). For the RF classifier, TF-IADF achieves the best effect. In addition, TF-IADFnorm and TF-IADF+norm also come with some improvement. The micro-F1 value of TF-IDF is 84.51%, while that of TF-IADF+norm is 85.52%, and that of TF-IADFnorm is 85.26%. The micro-F1 value of TF-IADF reaches to 86.37%, which is the best among all methods, an improvement of 1.86% from TF-IDF. For the NB classifier, both TF-IADF+ and TF-IADF+norm show improvements for the micro-F1 value of TF-IADF+ and TF-IADF+norm reaching 92.26% and 92.33%, while that of TF-IDF is only 92.13%. TF-IADF+norm achieves the best effect with an increase of 1.2% over TF-IDF. In the classification results, we notice that the precision rate and recall rate of finance are relatively low, at only 80% and 85%, respectively. The reason may be that the terms of this category are not obvious enough, and the scope involved is relatively wide, which may cover some contents from tourism and traffic categories, resulting in the poor classification effect of the whole category.The results show that, in the case of balanced dataset which is not focused by our design, the proposed methods can improve the classifiers to a certain extent even though the improvement range is not so obvious.


### 5.1.2. Unbalanced Dataset

In this section, we investigate how TC performance is impacted by unbalanced datasets. Dataset here is specially designed. The size of food is increased to 5000, and the sizes of other categories are kept the same as in the balanced dataset (1000) except sport. For sport, the size is gradually reduced from 400 to 50, decreasing 50 at a time.Definition 2 (balance ratio).
It is the proportion between the sizes of the category with the smallest size and the category with the largest size.
In this section, we call the proportion between the size of sport and that of food as balance ratio which drops from 8% to 1% with the decrease of sport’s size. Experiments by using the SVM classifier with TF-IDF and our proposed methods are carried out. The experimental results show that performances on categories other than sport and food are basically the same with the balance ratio changing. However, the recall andF1 score of sport and the precision of food are impacted heavily by decreasing the balance ratio. The details are shown in Tables 6–8. As shown, the balance ratio is decreasing, with the recall and F1 score of sport also decreasing in all methods. However, the proposed TF-IADF and TF-IADFnorm outperformed TF-IDF in all cases. Especially, when the balance ratio decreased to an extreme value (1%), TF-IADFnorm came with a recall of 0.7168 which is almost 170% of that of TF-IDF. Due to the balance ratio changes, precision of the category with a relatively large size (food) was also impacted which can be seen from Table 8.Table 6
Recall (sport).
Balance ratio (%)TF-IDFTF-IADFTF-IADFnormTF-IADF+TF-IADF+norm80.94790.95810.95810.93150.916270.93760.94580.94790.89880.888660.91820.93970.94170.88240.869150.90180.93460.94070.86500.850740.87630.92430.93150.82620.812930.82720.89370.91410.74640.730120.74230.82520.85070.60120.591010.42740.66260.71680.25670.2413Table 7
F1 score (sport).
Balance ratio (%)TF-IDFTF-IADFTF-IADFnormTF-IADF+TF-IADF+norm80.97320.97860.97760.96450.956270.96780.97220.97320.94670.941060.95740.96890.97000.93750.930050.94840.96620.96940.92760.919340.93410.96070.96450.90480.896830.90540.94380.95510.85480.844020.85210.90420.91930.75100.742910.59890.79710.83500.40850.3888Table 8
Precision (food).
Balance ratio (%)TF-IDFTF-IADFTF-IADFnormTF-IADF+TF-IADF+norm80.87260.91850.93050.83100.778870.87110.91600.92880.82410.771660.86880.91600.92960.82130.766850.86500.91430.92960.81460.758040.85830.91260.92790.80410.744430.84800.91010.92530.78750.717020.82750.88490.90920.75170.672810.78130.87250.89760.67280.5691Definition 3 (decline ratio).
It is the ratio of the current value to the initial value in a declining trend.
To make the relationship between the balance ratio and the performance clear, the decline ratio is calculated. It is the ratio of the value for each balance ratio to its initial value (value at 8%). This can be seen in Figure9, where the ordinate refers to the decline ratio of the corresponding performance factor. As it is shown, with the balance ratio decreasing, the growth rate of decline ratio becomes faster and faster. It is obvious that, for datasets with extreme categories such as food (relatively too large) and sport (relatively too small), performances of TF-IADF and TF-IADFnorm are more stable than TF-IDF.
The overall performance on this dataset is impacted by the balance ratio also. The details of micro-F1 and macro-F1 are shown in Tables 9 and 10, respectively. The proposed TF-IADF and TF-IADFnorm outperformed TF-IDF in all cases. To make it clear how balance ratio impacts the performance, we calculated the decline ratio which is shown in Figure 10. As it is shown, with the decrease of balance ratio, both micro-F1 and macro-F1 decrease gradually. For example, when looking at micro-F1, TF-IADFnorm was with a decrease of just 3.65%, while TF-IDF was with a decrease of 8.06% which is more than twice of that of the proposed TF-IADFnorm, meaning the performance of TF-IADFnorm is much more stable than that of TF-IDF in this dataset.
Even though TF-IADF+ and its variance do not achieve improvement in this experiment, TF-IADF and its variance outperformed TF-IDF significantly. Furthermore, it can be seen from Figure 10 that our proposed TF-IADF and TF-IADFnorm are not only numerically better but also more stable than TF-IDF. Therefore, considering ADF in the term weighting method can actually improve the performance of text classification considerably in unbalanced cases.Figure 9
The decline ratio of the performance with balance ratio changing.
(a)vpos: 38.5pt(b)(c)Table 9
Micro-F1.
Balance ratio (%)TF-IDFTF-IADFTF-IADFnormTF-IADF+TF-IADF+norm80.92920.93720.93720.92260.914070.92780.93540.93600.91790.909460.92500.93450.93520.91540.906350.92260.93380.93510.91290.903240.91890.93230.93380.90730.897730.91190.92790.93130.89600.885920.89940.91790.92220.87530.865810.85420.89460.90290.82580.8156Table 10
Macro-F1.
Balance ratio (%)TF-IDFTF-IADFTF-IADFnormTF-IADF+TF-IADF+norm80.92870.93660.93660.92250.914770.92730.93490.93550.91790.910260.92460.93410.93480.91540.907250.92230.93330.93460.91300.904240.91860.93190.93330.90740.898730.91150.92750.93090.89560.886820.89850.91740.92170.87260.864910.84430.89220.90140.80550.7987Figure 10
The decline ratio of the overall performance with balance ratio changing.
(a)vpos: 11.2pt(b)


### 5.1.3. Analysis

For this corpus, the TF-IADF+ and TF-IADF+norm methods are more suitable for the NB classifier, and TF-IADF+norm is the best. For the RF and SVM classifiers, TF-IADF+norm and TF-IADF are more suitable. In addition, TF-IADF is better in case of a balanced dataset, while TF-IADFnorm is better in case of an unbalanced dataset. This means that the processed TF-IADFnorm method is more sensitive and stable in the case of unbalanced datasets, while TF-IADF improves the classification effect more when the datasets are relatively balanced. For several mathematical models proposed by different algorithms, it can be concluded that the formula suitable for different algorithms may be different, and for the improvement of the corresponding algorithm effect, it can also be concluded that the ADF index has improved the effect of text classification, which confirms the conjecture; especially, when the dataset is not evenly distributed, the effect of text classification is more stable.


## 5.1.1. Balanced Dataset

In this dataset, there are 1000 training samples and 1000 test samples in each category. This is an exactly balanced dataset. SVM, NB, and RF classifiers are utilized as TWS for a comparison with TF-IDF. The overall performance of the proposed TF-IADF outperformed all other methods in SVM and RF classifiers as shown in Figure8. The details of experimental results are shown in Table 5 in that the proposed TF-IADF+norm demonstrates better performance than TF-IDF in all cases. Furthermore, all proposed methods outperformed the TF-IDF, in some cases, respectively.Figure 8
Data distribution of the Fudan corpus (unbalanced).Table 5
Experiments on the balanced dataset of the internet corpus.
Balanced datasetTF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-IADF+norm (%)micro-F1 (SVM)94.2394.5494.2394.3294.35macro-F1 (SVM)94.1694.4894.1894.2594.28micro-F1 (RF)84.5186.3785.2684.2385.52macro-F1 (RF)84.4186.2685.1684.0885.41micro-F1 (NB)92.1391.7391.4292.2692.33macro-F1 (NB)92.0591.6691.3692.1892.24For the SVM classifier, the overall classification effect is better than the other two classifiers, and the micro-F1 value is over 94%. TF-IADF has achieved the best effect of 94.45% (increased by 0.31% than TF-IDF). For the RF classifier, TF-IADF achieves the best effect. In addition, TF-IADFnorm and TF-IADF+norm also come with some improvement. The micro-F1 value of TF-IDF is 84.51%, while that of TF-IADF+norm is 85.52%, and that of TF-IADFnorm is 85.26%. The micro-F1 value of TF-IADF reaches to 86.37%, which is the best among all methods, an improvement of 1.86% from TF-IDF. For the NB classifier, both TF-IADF+ and TF-IADF+norm show improvements for the micro-F1 value of TF-IADF+ and TF-IADF+norm reaching 92.26% and 92.33%, while that of TF-IDF is only 92.13%. TF-IADF+norm achieves the best effect with an increase of 1.2% over TF-IDF. In the classification results, we notice that the precision rate and recall rate of finance are relatively low, at only 80% and 85%, respectively. The reason may be that the terms of this category are not obvious enough, and the scope involved is relatively wide, which may cover some contents from tourism and traffic categories, resulting in the poor classification effect of the whole category.The results show that, in the case of balanced dataset which is not focused by our design, the proposed methods can improve the classifiers to a certain extent even though the improvement range is not so obvious.


## 5.1.2. Unbalanced Dataset

In this section, we investigate how TC performance is impacted by unbalanced datasets. Dataset here is specially designed. The size of food is increased to 5000, and the sizes of other categories are kept the same as in the balanced dataset (1000) except sport. For sport, the size is gradually reduced from 400 to 50, decreasing 50 at a time.Definition 2 (balance ratio).
It is the proportion between the sizes of the category with the smallest size and the category with the largest size.
In this section, we call the proportion between the size of sport and that of food as balance ratio which drops from 8% to 1% with the decrease of sport’s size. Experiments by using the SVM classifier with TF-IDF and our proposed methods are carried out. The experimental results show that performances on categories other than sport and food are basically the same with the balance ratio changing. However, the recall andF1 score of sport and the precision of food are impacted heavily by decreasing the balance ratio. The details are shown in Tables 6–8. As shown, the balance ratio is decreasing, with the recall and F1 score of sport also decreasing in all methods. However, the proposed TF-IADF and TF-IADFnorm outperformed TF-IDF in all cases. Especially, when the balance ratio decreased to an extreme value (1%), TF-IADFnorm came with a recall of 0.7168 which is almost 170% of that of TF-IDF. Due to the balance ratio changes, precision of the category with a relatively large size (food) was also impacted which can be seen from Table 8.Table 6
Recall (sport).
Balance ratio (%)TF-IDFTF-IADFTF-IADFnormTF-IADF+TF-IADF+norm80.94790.95810.95810.93150.916270.93760.94580.94790.89880.888660.91820.93970.94170.88240.869150.90180.93460.94070.86500.850740.87630.92430.93150.82620.812930.82720.89370.91410.74640.730120.74230.82520.85070.60120.591010.42740.66260.71680.25670.2413Table 7
F1 score (sport).
Balance ratio (%)TF-IDFTF-IADFTF-IADFnormTF-IADF+TF-IADF+norm80.97320.97860.97760.96450.956270.96780.97220.97320.94670.941060.95740.96890.97000.93750.930050.94840.96620.96940.92760.919340.93410.96070.96450.90480.896830.90540.94380.95510.85480.844020.85210.90420.91930.75100.742910.59890.79710.83500.40850.3888Table 8
Precision (food).
Balance ratio (%)TF-IDFTF-IADFTF-IADFnormTF-IADF+TF-IADF+norm80.87260.91850.93050.83100.778870.87110.91600.92880.82410.771660.86880.91600.92960.82130.766850.86500.91430.92960.81460.758040.85830.91260.92790.80410.744430.84800.91010.92530.78750.717020.82750.88490.90920.75170.672810.78130.87250.89760.67280.5691Definition 3 (decline ratio).
It is the ratio of the current value to the initial value in a declining trend.
To make the relationship between the balance ratio and the performance clear, the decline ratio is calculated. It is the ratio of the value for each balance ratio to its initial value (value at 8%). This can be seen in Figure9, where the ordinate refers to the decline ratio of the corresponding performance factor. As it is shown, with the balance ratio decreasing, the growth rate of decline ratio becomes faster and faster. It is obvious that, for datasets with extreme categories such as food (relatively too large) and sport (relatively too small), performances of TF-IADF and TF-IADFnorm are more stable than TF-IDF.
The overall performance on this dataset is impacted by the balance ratio also. The details of micro-F1 and macro-F1 are shown in Tables 9 and 10, respectively. The proposed TF-IADF and TF-IADFnorm outperformed TF-IDF in all cases. To make it clear how balance ratio impacts the performance, we calculated the decline ratio which is shown in Figure 10. As it is shown, with the decrease of balance ratio, both micro-F1 and macro-F1 decrease gradually. For example, when looking at micro-F1, TF-IADFnorm was with a decrease of just 3.65%, while TF-IDF was with a decrease of 8.06% which is more than twice of that of the proposed TF-IADFnorm, meaning the performance of TF-IADFnorm is much more stable than that of TF-IDF in this dataset.
Even though TF-IADF+ and its variance do not achieve improvement in this experiment, TF-IADF and its variance outperformed TF-IDF significantly. Furthermore, it can be seen from Figure 10 that our proposed TF-IADF and TF-IADFnorm are not only numerically better but also more stable than TF-IDF. Therefore, considering ADF in the term weighting method can actually improve the performance of text classification considerably in unbalanced cases.Figure 9
The decline ratio of the performance with balance ratio changing.
(a)vpos: 38.5pt(b)(c)Table 9
Micro-F1.
Balance ratio (%)TF-IDFTF-IADFTF-IADFnormTF-IADF+TF-IADF+norm80.92920.93720.93720.92260.914070.92780.93540.93600.91790.909460.92500.93450.93520.91540.906350.92260.93380.93510.91290.903240.91890.93230.93380.90730.897730.91190.92790.93130.89600.885920.89940.91790.92220.87530.865810.85420.89460.90290.82580.8156Table 10
Macro-F1.
Balance ratio (%)TF-IDFTF-IADFTF-IADFnormTF-IADF+TF-IADF+norm80.92870.93660.93660.92250.914770.92730.93490.93550.91790.910260.92460.93410.93480.91540.907250.92230.93330.93460.91300.904240.91860.93190.93330.90740.898730.91150.92750.93090.89560.886820.89850.91740.92170.87260.864910.84430.89220.90140.80550.7987Figure 10
The decline ratio of the overall performance with balance ratio changing.
(a)vpos: 11.2pt(b)


## 5.1.3. Analysis

For this corpus, the TF-IADF+ and TF-IADF+norm methods are more suitable for the NB classifier, and TF-IADF+norm is the best. For the RF and SVM classifiers, TF-IADF+norm and TF-IADF are more suitable. In addition, TF-IADF is better in case of a balanced dataset, while TF-IADFnorm is better in case of an unbalanced dataset. This means that the processed TF-IADFnorm method is more sensitive and stable in the case of unbalanced datasets, while TF-IADF improves the classification effect more when the datasets are relatively balanced. For several mathematical models proposed by different algorithms, it can be concluded that the formula suitable for different algorithms may be different, and for the improvement of the corresponding algorithm effect, it can also be concluded that the ADF index has improved the effect of text classification, which confirms the conjecture; especially, when the dataset is not evenly distributed, the effect of text classification is more stable.


## 5.2. Results on the Fudan Corpus

### 5.2.1. Results’ Analysis

Table11 and Figure 11 show the micro-F1 and macro-F1 scores obtained on Fudan corpus using SVM, RF, and NB algorithms with different TWSs, while Tables 12–14 show the detailed results.Table 11
Overall performances on the Fudan corpus.
Fudan corpusTF-IDFTF-IADFTF-IADFnormTF-IADF+ (%)TF-IADF+norm (%)micro-F1 (SVM)50.0254.5754.9348.8547.90macro-F1 (SVM)90.3191.0490.9490.2489.82micro-F1 (RF)48.5751.4348.8049.7851.64macro-F1 (RF)80.3281.9580.8381.2079.85micro-F1 (NB)65.4462.2859.9466.3966.97macro-F1 (NB)88.9988.2087.5589.3189.38Figure 11
Overall performances of classification on the Fudan corpus.Table 12
Performance on the Fudan corpus using the SVM classifier.
SVMPrecisionF1 scoreRecallTF-IDFTF-IADFTF-IADFnormTF-IADF+TF-IADF+normTF-IDFTF-IADFTF-IADFnormTF-IADF+TF-IADF+normTF-IDFTF-IADFTF-IADFnormTF-IADF+TF-IADF+normC1196.1095.9896.1396.24%96.24%92.0692.9992.8491.75%91.75%94.0394.4694.4593.94%93.94%C15100.00100.00100.00100.00%100.00%9.099.099.099.09%9.09%16.6716.6716.6716.67%16.67%C16100.00100.00100.00100.00%100.00%3.573.573.573.57%3.57%6.906.906.906.90%6.90%C17100.00100.00100.00100.00%100.00%18.5222.2222.2218.52%18.52%31.2536.3636.3631.25%31.25%C1995.6495.9395.5395.50%95.56%98.5398.9799.1298.53%98.31%97.0697.4397.2996.99%96.92%C23100.00100.00100.00100.00%100.00%5.885.885.885.88%5.88%11.1111.1111.1111.11%11.11%C29100.0096.1595.65100.00%100.00%8.4842.3737.295.09%3.39%15.6358.8253.669.68%6.56%C387.1088.8589.1787.18%86.90%95.5595.5595.4295.28%94.74%91.1392.0892.1991.05%90.65%C3195.6196.9797.7795.80%95.86%96.6397.2196.9697.29%97.04%96.1297.0997.3696.54%96.45%C3294.1794.1093.4794.43%94.37%96.3896.6796.5896.18%95.21%95.2695.3795.0095.30%94.79%C3490.2090.1890.2389.67%87.03%94.8895.1994.6394.88%94.75%92.4892.6292.3892.20%90.73%C35100.0095.2492.31100.00%100.00%23.0838.4646.1511.54%7.69%37.5054.8061.5420.69%14.29%C36100.00100.00100.00100.00%100.00%1.8913.2115.091.89%1.89%3.7023.3326.233.70%3.70%C37100.00100.00100.00100.00%100.00%5.265.265.265.26%5.26%10.0010.0010.0010.00%10.00%C3882.6582.3682.7883.19%82.66%95.6194.6494.6495.52%95.22%88.6688.0788.3188.93%88.50%C3985.3688.6987.6884.86%85.93%96.7396.9796.4996.97%96.89%90.6992.6591.8890.51%91.08%C4100.00100.00100.00100.00%100.00%5.885.885.885.88%2.94%11.1111.1111.1111.11%5.71%C566.6766.6766.6766.67%66.67%3.283.283.283.28%3.28%6.256.256.256.25%6.25%C688.8988.8990.0088.89%88.89%17.7817.7820.0017.78%17.78%29.6329.6332.7329.63%29.63%C782.7881.3080.9982.43%81.70%68.8072.4473.7268.16%65.81%75.1576.6177.1874.62%72.90%Table 13
Performance on the Fudan corpus using the RF classifier.
RFPrecisionRecallF1 scoreTF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-IADF+norm (%)TF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-+norm (%)TF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-IADF+norm (%)C1182.9490.8585.5287.6186.5082.5685.0579.1378.1983.8082.7587.8582.2082.6385.13C1533.3380.0044.4466.6783.3315.1512.1212.1212.1215.1520.8321.0519.0520.5125.64C1640.0050.0057.1430.7757.147.1417.8614.2914.2914.2912.1226.3222.8619.5122.86C1755.5650.0038.7158.3361.9137.0437.0444.4425.9348.1544.4442.5541.3835.9054.17C1989.9090.5587.6186.3188.2496.3296.6196.3297.0596.6993.0093.4891.7691.3792.27C2343.7538.4633.3334.7850.0020.5914.718.8223.5314.7128.0021.2813.9528.0722.73C2943.5969.5750.0067.7460.8728.8127.1222.0335.5947.4634.6939.0230.5946.6753.33C375.7774.8973.4175.1471.7386.3990.4390.4389.6287.8780.7381.9381.0481.7578.98C3192.0493.1791.7792.8693.4990.2391.7991.5490.7288.4291.1392.4791.6691.7890.89C3284.1087.0186.3088.0881.9684.3486.5083.8686.7980.9284.2286.7585.0687.4381.44C3474.4975.8573.5975.0372.6188.2690.2689.8289.6988.0780.7982.4380.9081.7179.59C3566.6777.7856.5268.7560.007.6913.4625.0021.1517.3113.7922.9534.6732.3526.87C3657.1466.6751.8575.0051.5222.6430.1926.4228.3032.0832.4341.5635.0041.1039.54C3742.8642.8645.4613.6466.6711.8415.796.583.9510.5318.5623.0811.496.1218.18C3865.6666.4570.0066.8666.8176.2277.9780.5180.0276.5170.5571.7574.8972.8571.33C3984.7986.2987.5487.6586.6784.4585.3384.0582.6281.4284.6285.8185.7685.0683.96C4100.0075.00100.00100.00100.005.888.825.885.888.8211.1115.7911.1111.1116.22C555.5685.7150.0062.5077.788.209.8411.488.2011.4814.2917.6518.6714.4920.00C690.9184.6283.3363.6490.0022.2224.4422.2215.5620.0035.7137.9335.0925.0032.73C769.5470.4873.0477.9165.0525.8625.0017.9527.1425.8637.7036.9128.8240.2537.00Table 14
Performance on the Fudan corpus using the NB classifier.
NBPrecisionRecallF1 scoreTF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+TF-IADF+norm (%)TF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-IADF+norm (%)TF-IDF (%)TF-IADF (%)TF-IADFnormTF-IADF+ (%)TF-IADF+norm (%)C1194.9894.5794.5394.9994.8391.2889.5688.7991.5991.4393.0992.0091.5793.2693.10C1593.75100.00100.0093.7593.7545.4630.3027.2745.4645.4661.2246.5142.8661.2261.22C1666.67100.00100.0075.0080.007.143.573.5710.7114.2912.906.906.9018.7524.24C1787.5086.6783.3393.7594.1251.8548.1537.0455.5659.2665.1261.9151.2869.7772.73C1996.3896.0295.8796.4696.3896.1796.0295.8096.1796.1796.2896.0295.8496.3196.28C2393.3392.31100.0094.7490.0041.1835.2920.5952.9452.9457.1451.0634.1567.9366.67C2986.3687.8887.1086.3688.8964.4149.1545.7664.4167.8073.7963.0460.0073.7976.92C382.9481.8181.1983.4383.5194.3495.1595.4294.3494.2188.2787.9887.7388.5588.54C3197.5997.8997.7897.6997.6992.9491.3890.4893.7693.7695.2194.5293.9995.6995.69C3291.9290.7090.3591.6891.6090.2288.7587.9790.6190.7191.0689.7189.1491.1491.15C3485.0582.8681.3285.7686.2092.7692.6992.4492.9492.8888.7487.5086.5289.2189.42C3587.5086.9687.5086.9686.9640.3938.4640.3938.4638.4655.2653.3355.2653.3353.33C3690.9188.8993.3390.9190.9137.7430.1926.4237.7437.7453.3345.0741.1853.3353.33C3778.8584.0981.4078.4378.8553.9548.6846.0552.6353.9564.0661.6758.8262.9964.06C3884.3083.1282.1685.4685.5592.1192.5992.0192.2092.3088.0387.6086.8188.7088.80C3989.3088.2987.3089.3989.3993.8693.2292.6694.0294.0291.5290.6989.9091.6491.64C450.00100.00100.0050.0050.005.885.885.885.885.8810.5311.1111.1110.5310.53C580.0075.0071.4387.5077.7813.129.848.2011.4811.4822.5417.3914.7120.2920.00C690.0090.0090.0090.0090.0020.0020.0020.0020.0020.0032.7332.7332.7332.7332.73C769.1071.7672.4968.9769.1766.8866.2464.7468.3869.0267.9768.8968.4068.6769.09For the NB classifier, TF-IADF+norm has the best performance, which is 89.38%, an increase of 0.39% compared to TF-IDF. Meanwhile, TF-IADF+ also gets an increase of 0.32%. This means for Chinese text datasets, these two models are more suitable for a NB classifier and can achieve an improvement on the overall classification effect. When comparing the specific measurements which are shown in Figure 12 and more details can be seen in Table 14, TF-IADF+norm has achieved a lot of the highest performance-score items. Furthermore, the difference between performance scores of TF-IDF and those of TF-IADF+norm is relatively small in categories, where TF- IADF+norm is not as good as TF-IDF. For example, in C32 and C35, where TF-IDF achieves the best precision score, the difference between the precision score of TF-IDF and that of TF-IADF+norm is less than 1%. However, in categories such as C17, where TF-IADF+norm achieves the highest precision score, the difference between that and precision score of TF-IDF is more than 6% which is six times of the difference occurring in categories where TF-IDF achieves the higher score. In fact, in C17, when comparing TF-IADF+norm to TF-IDF, the precision is improved by 6.62%, and the recall score of C17 is also increased by 7.41%, which is obvious. In an overall view of the F1 scores, it can be noticed that, in all the 20 categories except C35 and C5, the scores of TF-IADF+norm are not lower than those of TF-IDF. Especially, in cases of TF-IADF+norm with higher elevation such as C16, the F1 score is increased by nearly 10%.Figure 12
Detailed performances on the Fudan corpus using the NB classifier.
(a)(b)(c)For the RF classifier, the micro-F1 score obtained by TF-IADF is 81.95%, which is 1.63% higher than that obtained by TF-IDF. Meanwhile, TF-IADF+ and TF-IADFnorm also have achieved improvements in different extents. It is known that RF algorithm has certain randomness, but our proposed TF-IADF method is more stable and has achieved better results which can be seen in Figure 13. When looking into detailed results as shown in Table 13, in some categories, such as C29 and C5, the precision score has been improved by 25.98% and 30.16%, respectively, and the recall score has remained basically unchanged. For the performance of F1 score, in some categories, such as C16 and C36, the score obtained by TF-IADF is about 10% higher than that obtained by TF-IDF. Furthermore, there are only three categories where F1 of TF-IADF is not as good as TF-IDF.Figure 13
Detailed performances on the Fudan corpus using the RF classifier.
(a)(b)(c)For the SVM classifier, it is still the case that TF-IADF and TF-IADFnorm have achieved improved performances. When comparing with TF-IDF in the micro-F1 score, TF-IADF has increased by 0.73%, while TF-IADFnorm has increased by 0.63%. It can be seen in Figure 14 that, for the SVM classifier, these two improved methods are more stable and have achieved some improvements. As shown in Table 12, there are many categories where the precision scores are very high, even up to 1. It can easily be seen in Figure 14 that the size (data amount) of those categories is very small. For example, there are only 32 training samples in C15. The reason is that due to the small eigenvalues, no other categories being considered as this category are responsible for the high precision score. However, it can also be concluded from the detailed results that, for most of those categories with a high precision but a small size, the recall score is significantly reduced; that is to say, many documents are assigned to the wrong categories, which may be due to the impact caused by a small number of features. For example, the recall score of C15 is only 0.09. TF-IADF and TF-IADFnorm perform better in most of the categories, especially in C29, where the size is small. Although the precision scores obtained by the two methods are decreased by about 4%, the recall scores are improved by about 30% which is significant. In addition, there are only two categories where F1 scores of TF-IADF are lower than those of TF-IDF, while for most categories, TF-IADF is better.Figure 14
Detailed performances on the Fudan corpus using the SVM classifier.
(a)(b)(c)


### 5.2.2. Further Discussion

Due to the unbalanced distribution of this corpus, all categories can be roughly divided into three groups by their size for specific analysis. The first group refers to categories with size in the range of 0 to 100, the second refers to categories with size in the range of 101 to 1000, and the third group refers to categories with size over 1000. First of all, for the first group, due to the insufficient size of training data, the same property is reflected in all term weighting methods, i.e., high precision and low recall scores, such as C16. Taking SVM combined with TF-IDF as an example, the precision score reaches 100%, while the recall score is only 3.57%, resulting in poor classification effect. In short, it assigns a few samples correctly, while a large number of test samples are assigned to the wrong categories. And the performance is similar in the RF and NB classifiers. This is the impact caused by the unbalanced distribution and the small size of the training data. For the second group, the number of training sets has been improved to a higher level. For example, in C11 when using the SVM classifier with TF-IDF, the precision score and recall score are 96.10% and 92.06%, respectively, which is a significant improvement compared to the first group. For the third category, such as C19, the precision score and recall score are 95.64% and 98.53%, respectively. It can be concluded that the precision score in categories with large training set size is relatively low, while the recall rate is relatively high. In categories with a small training set size, it will have better precision but very low recall score. In these kinds of conditions, our proposed methods will demonstrate a similar but more stable performance compared to TF-IDF. Taking C29 as an example, there are only 57 training samples. Comparing TF-IDF with TF-IADF using the SVM classifier, the precision scores are 100% to 96.15%, while the recall scores are 8.48% to 42.37%. It can be seen that although the accuracy rate of our algorithm is slightly reduced, the recall rate is greatly increased by nearly 500%, and theF1 score is also greatly improved, from 15.63% to 58.82%, showing a very obvious improvement.In the Fudan corpus, a similar phenomenon with the internet corpus can be seen. For example, the TF-IADF+norm method is the best one in the NB classifier, while TF-IADF+ is also better than TF-IDF. And with RF and SVM classifiers, both TF-IADF and TF-IADFnorm achieve a relatively stable performance. The difference is that TF-IADF achieves the best effect in the Fudan unbalanced dataset. It is also noticed that although the micro-F1 score of the SVM classifier is the best, the macro-F1 score, which is about 50%, is not as good as that of the NB classifier which is over 65%. That is to say, although the overall accuracy of the SVM classifier is high, the effect of the NB classifier is better when each category is regarded as equally important.


## 5.2.1. Results’ Analysis

Table11 and Figure 11 show the micro-F1 and macro-F1 scores obtained on Fudan corpus using SVM, RF, and NB algorithms with different TWSs, while Tables 12–14 show the detailed results.Table 11
Overall performances on the Fudan corpus.
Fudan corpusTF-IDFTF-IADFTF-IADFnormTF-IADF+ (%)TF-IADF+norm (%)micro-F1 (SVM)50.0254.5754.9348.8547.90macro-F1 (SVM)90.3191.0490.9490.2489.82micro-F1 (RF)48.5751.4348.8049.7851.64macro-F1 (RF)80.3281.9580.8381.2079.85micro-F1 (NB)65.4462.2859.9466.3966.97macro-F1 (NB)88.9988.2087.5589.3189.38Figure 11
Overall performances of classification on the Fudan corpus.Table 12
Performance on the Fudan corpus using the SVM classifier.
SVMPrecisionF1 scoreRecallTF-IDFTF-IADFTF-IADFnormTF-IADF+TF-IADF+normTF-IDFTF-IADFTF-IADFnormTF-IADF+TF-IADF+normTF-IDFTF-IADFTF-IADFnormTF-IADF+TF-IADF+normC1196.1095.9896.1396.24%96.24%92.0692.9992.8491.75%91.75%94.0394.4694.4593.94%93.94%C15100.00100.00100.00100.00%100.00%9.099.099.099.09%9.09%16.6716.6716.6716.67%16.67%C16100.00100.00100.00100.00%100.00%3.573.573.573.57%3.57%6.906.906.906.90%6.90%C17100.00100.00100.00100.00%100.00%18.5222.2222.2218.52%18.52%31.2536.3636.3631.25%31.25%C1995.6495.9395.5395.50%95.56%98.5398.9799.1298.53%98.31%97.0697.4397.2996.99%96.92%C23100.00100.00100.00100.00%100.00%5.885.885.885.88%5.88%11.1111.1111.1111.11%11.11%C29100.0096.1595.65100.00%100.00%8.4842.3737.295.09%3.39%15.6358.8253.669.68%6.56%C387.1088.8589.1787.18%86.90%95.5595.5595.4295.28%94.74%91.1392.0892.1991.05%90.65%C3195.6196.9797.7795.80%95.86%96.6397.2196.9697.29%97.04%96.1297.0997.3696.54%96.45%C3294.1794.1093.4794.43%94.37%96.3896.6796.5896.18%95.21%95.2695.3795.0095.30%94.79%C3490.2090.1890.2389.67%87.03%94.8895.1994.6394.88%94.75%92.4892.6292.3892.20%90.73%C35100.0095.2492.31100.00%100.00%23.0838.4646.1511.54%7.69%37.5054.8061.5420.69%14.29%C36100.00100.00100.00100.00%100.00%1.8913.2115.091.89%1.89%3.7023.3326.233.70%3.70%C37100.00100.00100.00100.00%100.00%5.265.265.265.26%5.26%10.0010.0010.0010.00%10.00%C3882.6582.3682.7883.19%82.66%95.6194.6494.6495.52%95.22%88.6688.0788.3188.93%88.50%C3985.3688.6987.6884.86%85.93%96.7396.9796.4996.97%96.89%90.6992.6591.8890.51%91.08%C4100.00100.00100.00100.00%100.00%5.885.885.885.88%2.94%11.1111.1111.1111.11%5.71%C566.6766.6766.6766.67%66.67%3.283.283.283.28%3.28%6.256.256.256.25%6.25%C688.8988.8990.0088.89%88.89%17.7817.7820.0017.78%17.78%29.6329.6332.7329.63%29.63%C782.7881.3080.9982.43%81.70%68.8072.4473.7268.16%65.81%75.1576.6177.1874.62%72.90%Table 13
Performance on the Fudan corpus using the RF classifier.
RFPrecisionRecallF1 scoreTF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-IADF+norm (%)TF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-+norm (%)TF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-IADF+norm (%)C1182.9490.8585.5287.6186.5082.5685.0579.1378.1983.8082.7587.8582.2082.6385.13C1533.3380.0044.4466.6783.3315.1512.1212.1212.1215.1520.8321.0519.0520.5125.64C1640.0050.0057.1430.7757.147.1417.8614.2914.2914.2912.1226.3222.8619.5122.86C1755.5650.0038.7158.3361.9137.0437.0444.4425.9348.1544.4442.5541.3835.9054.17C1989.9090.5587.6186.3188.2496.3296.6196.3297.0596.6993.0093.4891.7691.3792.27C2343.7538.4633.3334.7850.0020.5914.718.8223.5314.7128.0021.2813.9528.0722.73C2943.5969.5750.0067.7460.8728.8127.1222.0335.5947.4634.6939.0230.5946.6753.33C375.7774.8973.4175.1471.7386.3990.4390.4389.6287.8780.7381.9381.0481.7578.98C3192.0493.1791.7792.8693.4990.2391.7991.5490.7288.4291.1392.4791.6691.7890.89C3284.1087.0186.3088.0881.9684.3486.5083.8686.7980.9284.2286.7585.0687.4381.44C3474.4975.8573.5975.0372.6188.2690.2689.8289.6988.0780.7982.4380.9081.7179.59C3566.6777.7856.5268.7560.007.6913.4625.0021.1517.3113.7922.9534.6732.3526.87C3657.1466.6751.8575.0051.5222.6430.1926.4228.3032.0832.4341.5635.0041.1039.54C3742.8642.8645.4613.6466.6711.8415.796.583.9510.5318.5623.0811.496.1218.18C3865.6666.4570.0066.8666.8176.2277.9780.5180.0276.5170.5571.7574.8972.8571.33C3984.7986.2987.5487.6586.6784.4585.3384.0582.6281.4284.6285.8185.7685.0683.96C4100.0075.00100.00100.00100.005.888.825.885.888.8211.1115.7911.1111.1116.22C555.5685.7150.0062.5077.788.209.8411.488.2011.4814.2917.6518.6714.4920.00C690.9184.6283.3363.6490.0022.2224.4422.2215.5620.0035.7137.9335.0925.0032.73C769.5470.4873.0477.9165.0525.8625.0017.9527.1425.8637.7036.9128.8240.2537.00Table 14
Performance on the Fudan corpus using the NB classifier.
NBPrecisionRecallF1 scoreTF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+TF-IADF+norm (%)TF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-IADF+norm (%)TF-IDF (%)TF-IADF (%)TF-IADFnormTF-IADF+ (%)TF-IADF+norm (%)C1194.9894.5794.5394.9994.8391.2889.5688.7991.5991.4393.0992.0091.5793.2693.10C1593.75100.00100.0093.7593.7545.4630.3027.2745.4645.4661.2246.5142.8661.2261.22C1666.67100.00100.0075.0080.007.143.573.5710.7114.2912.906.906.9018.7524.24C1787.5086.6783.3393.7594.1251.8548.1537.0455.5659.2665.1261.9151.2869.7772.73C1996.3896.0295.8796.4696.3896.1796.0295.8096.1796.1796.2896.0295.8496.3196.28C2393.3392.31100.0094.7490.0041.1835.2920.5952.9452.9457.1451.0634.1567.9366.67C2986.3687.8887.1086.3688.8964.4149.1545.7664.4167.8073.7963.0460.0073.7976.92C382.9481.8181.1983.4383.5194.3495.1595.4294.3494.2188.2787.9887.7388.5588.54C3197.5997.8997.7897.6997.6992.9491.3890.4893.7693.7695.2194.5293.9995.6995.69C3291.9290.7090.3591.6891.6090.2288.7587.9790.6190.7191.0689.7189.1491.1491.15C3485.0582.8681.3285.7686.2092.7692.6992.4492.9492.8888.7487.5086.5289.2189.42C3587.5086.9687.5086.9686.9640.3938.4640.3938.4638.4655.2653.3355.2653.3353.33C3690.9188.8993.3390.9190.9137.7430.1926.4237.7437.7453.3345.0741.1853.3353.33C3778.8584.0981.4078.4378.8553.9548.6846.0552.6353.9564.0661.6758.8262.9964.06C3884.3083.1282.1685.4685.5592.1192.5992.0192.2092.3088.0387.6086.8188.7088.80C3989.3088.2987.3089.3989.3993.8693.2292.6694.0294.0291.5290.6989.9091.6491.64C450.00100.00100.0050.0050.005.885.885.885.885.8810.5311.1111.1110.5310.53C580.0075.0071.4387.5077.7813.129.848.2011.4811.4822.5417.3914.7120.2920.00C690.0090.0090.0090.0090.0020.0020.0020.0020.0020.0032.7332.7332.7332.7332.73C769.1071.7672.4968.9769.1766.8866.2464.7468.3869.0267.9768.8968.4068.6769.09For the NB classifier, TF-IADF+norm has the best performance, which is 89.38%, an increase of 0.39% compared to TF-IDF. Meanwhile, TF-IADF+ also gets an increase of 0.32%. This means for Chinese text datasets, these two models are more suitable for a NB classifier and can achieve an improvement on the overall classification effect. When comparing the specific measurements which are shown in Figure 12 and more details can be seen in Table 14, TF-IADF+norm has achieved a lot of the highest performance-score items. Furthermore, the difference between performance scores of TF-IDF and those of TF-IADF+norm is relatively small in categories, where TF- IADF+norm is not as good as TF-IDF. For example, in C32 and C35, where TF-IDF achieves the best precision score, the difference between the precision score of TF-IDF and that of TF-IADF+norm is less than 1%. However, in categories such as C17, where TF-IADF+norm achieves the highest precision score, the difference between that and precision score of TF-IDF is more than 6% which is six times of the difference occurring in categories where TF-IDF achieves the higher score. In fact, in C17, when comparing TF-IADF+norm to TF-IDF, the precision is improved by 6.62%, and the recall score of C17 is also increased by 7.41%, which is obvious. In an overall view of the F1 scores, it can be noticed that, in all the 20 categories except C35 and C5, the scores of TF-IADF+norm are not lower than those of TF-IDF. Especially, in cases of TF-IADF+norm with higher elevation such as C16, the F1 score is increased by nearly 10%.Figure 12
Detailed performances on the Fudan corpus using the NB classifier.
(a)(b)(c)For the RF classifier, the micro-F1 score obtained by TF-IADF is 81.95%, which is 1.63% higher than that obtained by TF-IDF. Meanwhile, TF-IADF+ and TF-IADFnorm also have achieved improvements in different extents. It is known that RF algorithm has certain randomness, but our proposed TF-IADF method is more stable and has achieved better results which can be seen in Figure 13. When looking into detailed results as shown in Table 13, in some categories, such as C29 and C5, the precision score has been improved by 25.98% and 30.16%, respectively, and the recall score has remained basically unchanged. For the performance of F1 score, in some categories, such as C16 and C36, the score obtained by TF-IADF is about 10% higher than that obtained by TF-IDF. Furthermore, there are only three categories where F1 of TF-IADF is not as good as TF-IDF.Figure 13
Detailed performances on the Fudan corpus using the RF classifier.
(a)(b)(c)For the SVM classifier, it is still the case that TF-IADF and TF-IADFnorm have achieved improved performances. When comparing with TF-IDF in the micro-F1 score, TF-IADF has increased by 0.73%, while TF-IADFnorm has increased by 0.63%. It can be seen in Figure 14 that, for the SVM classifier, these two improved methods are more stable and have achieved some improvements. As shown in Table 12, there are many categories where the precision scores are very high, even up to 1. It can easily be seen in Figure 14 that the size (data amount) of those categories is very small. For example, there are only 32 training samples in C15. The reason is that due to the small eigenvalues, no other categories being considered as this category are responsible for the high precision score. However, it can also be concluded from the detailed results that, for most of those categories with a high precision but a small size, the recall score is significantly reduced; that is to say, many documents are assigned to the wrong categories, which may be due to the impact caused by a small number of features. For example, the recall score of C15 is only 0.09. TF-IADF and TF-IADFnorm perform better in most of the categories, especially in C29, where the size is small. Although the precision scores obtained by the two methods are decreased by about 4%, the recall scores are improved by about 30% which is significant. In addition, there are only two categories where F1 scores of TF-IADF are lower than those of TF-IDF, while for most categories, TF-IADF is better.Figure 14
Detailed performances on the Fudan corpus using the SVM classifier.
(a)(b)(c)


## 5.2.2. Further Discussion

Due to the unbalanced distribution of this corpus, all categories can be roughly divided into three groups by their size for specific analysis. The first group refers to categories with size in the range of 0 to 100, the second refers to categories with size in the range of 101 to 1000, and the third group refers to categories with size over 1000. First of all, for the first group, due to the insufficient size of training data, the same property is reflected in all term weighting methods, i.e., high precision and low recall scores, such as C16. Taking SVM combined with TF-IDF as an example, the precision score reaches 100%, while the recall score is only 3.57%, resulting in poor classification effect. In short, it assigns a few samples correctly, while a large number of test samples are assigned to the wrong categories. And the performance is similar in the RF and NB classifiers. This is the impact caused by the unbalanced distribution and the small size of the training data. For the second group, the number of training sets has been improved to a higher level. For example, in C11 when using the SVM classifier with TF-IDF, the precision score and recall score are 96.10% and 92.06%, respectively, which is a significant improvement compared to the first group. For the third category, such as C19, the precision score and recall score are 95.64% and 98.53%, respectively. It can be concluded that the precision score in categories with large training set size is relatively low, while the recall rate is relatively high. In categories with a small training set size, it will have better precision but very low recall score. In these kinds of conditions, our proposed methods will demonstrate a similar but more stable performance compared to TF-IDF. Taking C29 as an example, there are only 57 training samples. Comparing TF-IDF with TF-IADF using the SVM classifier, the precision scores are 100% to 96.15%, while the recall scores are 8.48% to 42.37%. It can be seen that although the accuracy rate of our algorithm is slightly reduced, the recall rate is greatly increased by nearly 500%, and theF1 score is also greatly improved, from 15.63% to 58.82%, showing a very obvious improvement.In the Fudan corpus, a similar phenomenon with the internet corpus can be seen. For example, the TF-IADF+norm method is the best one in the NB classifier, while TF-IADF+ is also better than TF-IDF. And with RF and SVM classifiers, both TF-IADF and TF-IADFnorm achieve a relatively stable performance. The difference is that TF-IADF achieves the best effect in the Fudan unbalanced dataset. It is also noticed that although the micro-F1 score of the SVM classifier is the best, the macro-F1 score, which is about 50%, is not as good as that of the NB classifier which is over 65%. That is to say, although the overall accuracy of the SVM classifier is high, the effect of the NB classifier is better when each category is regarded as equally important.


## 5.3. Results on Reuters-21578

The overall performance on Reuters-21758 of all methods is shown in Table15, while Figure 15 shows a brief comparison between all proposed methods and the well-known TF-IDF method when combining with different classifiers. More detailed results are shown in Tables 16–18.Table 15
Overall performances on Reuters-21578.
TF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-IADF+norm (%)micro-F1 (SVM)91.0091.4191.8690.5990.23macro-F1 (SVM)79.6380.3381.9778.4275.77micro-F1 (RF)87.4585.2387.7787.4188.14macro-F1 (RF)69.9059.0365.4367.9167.76micro-F1 (NB)89.8290.4190.3289.4189.36macro-F1 (NB)76.2277.8677.7776.2175.95Figure 15
Performance of the NB classifier on Reuters-21578 (precision, recall, andF1 score).Table 16
Performance on Reuters-21758 using the SVM classifier.
SVMPrecisionRecallF1 scoreTF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-IADF+norm (%)TF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TFIADF+norm (%)TF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-IADF+norm (%)Acq90.1791.4393.0390.0690.4292.2491.9592.1092.3992.2491.1991.6992.5691.2191.32Crude90.3989.7289.9192.6392.5577.6979.3480.9972.7371.9083.5684.2185.2281.4880.93Earn92.0692.0092.4891.3590.2197.4297.6997.6997.5197.8894.6694.7695.0294.3393.89Grain100.00100.00100.00100.00100.0040.0040.0050.0040.0030.0057.1457.1466.6757.1446.15Interest91.3090.0088.2491.1190.4856.0060.0060.0054.6750.6769.4272.0071.4368.3364.96Moneyfx79.7184.0680.0079.1079.0363.2266.6773.5660.9256.3270.5174.3676.6568.8365.77Ship100.00100.0095.8395.6595.2463.8961.1163.8961.1155.5677.9775.8676.6774.5870.18Trade90.6389.8088.7889.5892.4794.5795.6594.5793.4893.4892.5592.6391.5891.4992.97Table 17
Performance on Reuters-21758 using the RF classifier.
RFPrecisionRecallF1 scoreTF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-IADF+norm (%)TF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-IADF+norm (%)TF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-IADF+norm (%)Acq86.5282.2585.4181.7788.0690.3790.5292.5394.1190.0988.4186.1888.8387.5189.06Crude92.6893.2484.8891.2586.8162.8157.0360.3360.3365.2974.8870.7770.5372.6474.53Earn89.8987.7890.4893.1189.7496.8696.8696.5894.8397.6993.2492.1093.4493.9693.55Grain100.00100.0033.33100.0050.0030.0010.0010.0020.0010.0046.1518.1815.3933.3316.67Interest75.5174.2978.0072.7386.0549.3334.6752.0053.3349.3359.6847.2762.4061.5462.71Moneyfx75.4773.1782.6972.4171.4345.9834.4849.4348.2845.9857.1446.8861.8757.9355.94Ship90.0090.0092.31100.0090.4850.0025.0033.3341.6752.7864.2939.1348.9858.8266.67Trade72.7378.2184.8880.2381.2578.2666.3079.3575.0084.7875.3971.7782.0277.5382.98Table 18
Performance on Reuters-21758 using the NB classifier.
NBPrecisionRecallF1 scoreTF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-IADF+norm (%)TF-IDF (%)TF-IADF (%)TF (%)TF-IADF+ (%)TF-IADF+norm (%)TF-IDF (%)TF-IADF (%)TF-IADFnorm (%)TF-IADF+ (%)TF-IADF+norm (%)Acq91.8391.7491.6091.1291.3492.1092.5392.3991.3890.9591.9792.1391.9991.2591.15Crude85.7188.0788.0786.4987.1679.3479.3479.3479.3478.5182.4083.4883.4882.7682.61Earn92.6592.6892.6792.1491.8495.4895.8595.7595.2095.5794.0494.2494.1993.6493.67Grain71.4371.4371.4371.4371.4350.0050.0050.0050.0050.0058.8258.8258.8258.8258.82Interest78.2684.4482.6178.2679.5548.0050.6750.6748.0046.6759.5063.3362.8159.5058.82Moneyfx76.0077.2277.2277.0377.0365.5270.1270.1265.5265.5270.3773.4973.4970.8170.81Ship85.1992.0092.0088.4685.1963.8963.8963.8963.8963.8973.0275.4175.4174.1973.02Trade70.5974.3474.3469.7569.7591.3091.3091.3090.2290.2279.6281.9581.9578.6778.67For the SVM classifier, TF-IADFnorm comes with the best performance which is 0.86% higher than TF-IDF in the micro-F1 score. According to the detailed results shown in Table 15 and Figure 16, the improvement of TF-IADFnorm is mainly contributed by the recall score, which is greatly improved in the category of crude, grain, interest, and money-fx. Especially in the category of money-fx, the recall score has increased by 10.35%. The other categories also improved in different extents, namely, crude increased by 3.31%, interest increased by 4%, and grain increased by 10%. From the perspective of the F1 score, in the total eight categories, the max F1 score of five categories is obtained by TF-IADFnorm, and the largest increase is obtained in the category of grain, where the F1 score increased from 57.14% of TF-IDF to 66.67% of TF-IADFnorm, close to 10%.Figure 16
Performance of the SVM classifier on Reuters-21578 (precision, recall, andF1 score).
(a)(b)(c)For the NB classifier, the results are different from those on the Chinese dataset, where TF-IADF+ and TF-IADF+norm achieved better performance, and performances of TF-IADF+ and TF-IADF+norm were worse on this corpus. However, the proposed TF-IADF and TF-IADFnorm showed an improvement, the micro-F1 score of which is 0.59% and 0.5% higher than that of TF-IDF, respectively. As shown in Table 16 and Figure 17, TF-IADF has achieved the highest F1 score among all methods in all categories of this corpus, and the maximum improvement, 3.83%, occurs in the category of interest. Specifically, in terms of precision, we see that the four categories of cloud, interest, ship, and trade have improved significantly, with an improvement of 2.36%, 4.35%, 6.82%, and 3.75%, respectively. In terms of the recall score, the increase in the money-fx category is 4.60%.Figure 17
Performance of the NB classifier on Reuters-21578 (precision, recall, andF1 score).
(a)(b)(c)For the RF algorithm, TF-IADF+norm and TF-IADFnorm have improved the effect, micro-F1 score of which is 0.69% and 0.32% higher than that of TF-IDF, respectively. TF-IADF, which has better performance in the Chinese dataset, has worse performance on this corpus. It can be seen in Table 17 and Figure 18 that the better performance of TF-IADF+norm is mainly due to the higher recall score. Comparing with TF-IDF, the recall score obtained in the trade category is 6.52% higher, and in the crude category, it is 2.48% higher. In terms of precision, it has achieved good performance in the categories of interest and trade, with an increase of 10.54% and 8.52%, respectively. On the side of F1 score, the trade category improved significantly, up to 7.59%. However, we also see a decrease in the performance on the categories of grain and money-fx. This may be due to the small size of the test dataset, which changes greatly, and so, it is not easy to draw more accurate conclusions.Figure 18
Performance of the RF classifier on Reuters-21578 (precision, recall, andF1 score).
(a)(b)(c)According to all experimental results on Reuters-21578, the proposed TF-IADFnorm outperformed TF-IDF in nearly all conditions except the micro-F1 score of the RF classifier. In fact, the RF classifier’s performance was worse among all classifiers, suggesting that the RF classifier may not be suitable for this corpus.


## 5.4. Discussion

The experimental results obtained on the English dataset (Reuters) are somehow different from those on the Chinese dataset, and the most effective combination of a term weighting method and classification algorithm is different. For example, TF-IADF+norm is suitable for the Chinese internet corpus using the NB algorithm, whereas TF-IADF performs best in the English unbalanced dataset. It can be inferred from the experimental results that all of the proposed algorithms can generally always be combined with a suitable mathematical model that shows a better performance than the original TF-IDF. For example, TF-IADF+norm performs better in the RF algorithm, while TF-IADFnorm performs better in the SVM. The best combinations concluded from the experiments are shown in Table 19. For the two Chinese datasets in the experiment, we draw the following conclusions: (1) for the NB algorithm, TF-IADF+ and TF-IADF+norm are more suitable, and TF-IADF+norm can achieve better performance whether in balanced or unbalanced datasets; (2) for the RF algorithm and SVM algorithm, TF-IADF and TF-IADFnorm can achieve a relatively stable improvement effect. In the experiments with the internet corpus, it can be concluded that TF-IADF has a better improvement effect when the dataset is relatively balanced, and TF-IADFnorm has a better classification effect when the dataset is unbalanced. However, in the Fudan corpus (unbalanced), TF-IADF has achieved better performance than TF-IADFnorm, while TF-IADFnorm has also improved. It is possible that TF-IADF is more suitable for RF and SVM algorithms when there are many categories which are unbalanced in the corpus.Table 19
The best combinations of TWSs with classifiers.
Dataset/classifierNBRFSVMBalanced Chinese datasetTF-IADF+normTF-IADFTF-IADFUnbalanced Chinese datasetTF-IADF+normTF-IADF, TF-IADFnormTF-IADF, TF-IADFnormUnbalanced English datasetTF-IADFTF-IADF+normTF-IADFnorm


## 6. Conclusions

In this paper, an improved TF-IDF with novel term weighting schemes is proposed to greatly reduce the impact from the unbalanced distribution of datasets. It is easy to be observed that all unbalanced corpuses that are categorized with a larger amount of data will always have an impact on the classification effect due to the larger amount of feature words. Meanwhile, the precision score decreases, while documents from other categories will be easily mistaken into this category. This is due to the increase in the training set; some feature words with strong performance capabilities in other categories have also appeared in this category which causes errors. Meanwhile, categories with smaller amount of data will be reflected in the significant reduction in the recall score due to the insufficient collection of feature words. The training process cannot classify these categories well without sufficient representation in the training process. As a result, many documents are assigned to wrong categories. In these cases, the proposed methods can increase the weight of those feature words which have a document frequency close to the average value, while reducing the weight of low-frequency and high-frequency words in order to obtain better results.The simulation results show that the proposed methods with ADF are more effective than the original TF-IDF, although different mathematical models may be needed for improvement when utilizing different classification algorithms. Especially in experiments specifically designed in which the size of data in sport decreases while keeping other conditions the same, the results proved that the proposed methods are with a better performance and more stable than the well-known TF-IDF on the unbalanced corpus. It can also be concluded that our proposed methods come with a better performance in the balanced dataset when compared with TF-IDF. Document frequency of specific words may vary across categories, even in cases where the training sets appear roughly the same. Methods with ADF can weight those words more reasonably and form a more representative model.However, for the purpose of TC on internet media reports, this paper just focuses on the term weighting scheme under unbalanced distribution but ignoring linguistic characteristics, which might be helpful in the term extraction process. Therefore, our next study will focus on enhancing the TC performance by combining the proposed methods with language characteristics.



---

*Source: 6619088-2021-03-05.xml*