# The Influence of Feature Representation of Text on the Performance of Document Classification
## 3 Materials and Methods
### Ranking Classification Models  
The comparative analysis and ranking of different model variants with regard to a single performance measure, such as accuracy and macro-averaged AUROC, precision, recall and F1-score is trivial: the larger the performance metrics, the better the classification model is. We assess the statistical significance of the differences between the performance of variants using the framework for comparison of classification methods over multiple data sets proposed by [PERSON] [73]. Following that framework, the comparison of methods is performed in two steps. In the first step, the Friedman test, a non-parametric alternative to the repeated measure ANOVA (analysis of variance), was used to validate the null hypothesis that the methods perform equally well. When the null hypothesis was rejected, the Nemenyi post-hoc test was used to calculate the critical distance (CD). The average ranks of two methods (averaged over the data sets) significantly differed if their difference was at least CD. The results of the comparison are visualized using an average-rank diagram (see Figure 2). The horizontal axis of the diagram corresponds to the rank score, and the top-ranked method is on the rightmost position on the axis. The groups of methods with average ranks that were not significantly different are connected with thick lines below the axis; the line above the axis depicts the critical distance. For calculating the critical distance, we used the significance level of 0.05. On the other hand, when we ranked models with regard to a per-class performance measure (recall, precision, F1-score or AUROC), we had to compare their performance along multiple dimensions corresponding to class labels. To obtain a joint ranking with respect to the performance measured on all class labels, we employed the non-dominated sorting algorithm used in multi-objective decision theory [74]. We first embedded the model variants into a \(|C|\)-dimensional space (recall that \(|C|\) denotes the number of class labels), where each dimension corresponds to a ranking of the variants with respect to one of the class labels. Each variant can then be depicted as a single point in that space, where each coordinate value was the rank of the method according to its performance on the particular class label. Figure 1 depicts the projection of the four-dimensional space for the Brown4 data set and AUROC in two dimensions, obtained by using multidimensional scaling [75]. The red, green, and blue labels in the graph correspond to the 16 variants of the document representation models. \begin{table}
\begin{tabular}{l c c c} \hline \hline  & **Number of** & **Number of** & **Total** \\  & **Train Documents** & **Test Documents** & **Total** \\ \hline Brown & 401 (80\%) & 99 (20\%) & 500 \\ Reuters8 & 6800 (72\%) & 2660 (28\%) & 9460 \\ WebKB & 6623 (80\%) & 1651 (20\%) & 8274 \\
20 News & 11,314 (60\%) & 7532 (40\%) & 18,846 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: The number of train and test documents for each benchmark; note that, 20 News and Reuters8 are already split into two sets of training and test documents, for the other two, we used stratified random sampling to split the corpora into training and test data sets. To identify the top-ranked variants, we searched for a set of non-dominated points in the \(|C|\)-dimensional space, i.e., we identified the Pareto front of the non-dominated points in the space. The points in the Pareto front corresponded to the methods that are the best performers on at least one class label. After we assigned the top rank to these variants, we removed the corresponding points from the multi-dimensional space and iteratively continued to identify Pareto fronts in the reduced sets of points until all the methods were ranked. The lines in Figure 1 depict the three Pareto fronts in the two-dimensional projection of the original three-dimensional space.