# Short Text Classification with Tolerance-Based Soft Computing Method
## 3 Materials and Methods
### Methods
This result leads us to conclude that perhaps balanced tolerance classes may not be a significant factor and may depend on the vector generation method. Table 11 gives the results of the Wilcoxon signed-ranks test based on the weighted F1-score of the classifiers using a two-sided test with \(\alpha=0.05\) to test the null hypothesis that there is no difference between the TSC-mean and other classifiers. The results are a pair-wise test on all datasets. Based on these results, the null hypothesis can be rejected (i.e., there is a difference between between the classifiers based on weighted F1-score). ## 5 Conclusions

In this paper, we present a tolerance near sets-based (TSC) algorithm to classify sentiment polarities as well as news categories from text. We have introduced formal definitions and illustrate our method using tolerance classes for generating representative vectors. The performance of the TSC algorithm was tested with ten well-researched text classification datasets including a manually labeled dataset from Twitter about opinions on COVID-19 across the globe. The curated datasets include long and short words, a mix of sentiment classes as well as balanced and imbalanced datasets. We have analyzed the effect of TF-IDF and the transformer vector-generation method (SBERT) on the performance of the TSC algorithm and five classical ML algorithms. We have demonstrated that the TSC algorithm performs well in seven out of ten datasets using weighted F1, Precision and Recall scores. The highest AUC-ROC score was obtained in two datasets and comparable in six other datasets. The highest AUC-PRC score was obtained in one dataset and comparable in four other datasets. Additionally, significant differences were observed in most comparisons when examining the statistical difference between the weighted F1-score of TSC and other classifiers using a Wilcoxon signed-rank test. As future work, optimization of the TSC algorithm in terms of computing tolerance classes is a natural extension of the current work by using parallel computing with CUDA to compute the distance matrix. This is \begin{table}
\begin{tabular}{l l l l l l l} \hline \hline
**Dataset** & **TSC-Mean** & **RF** & **ME** & **SVM** & SGD & **LGBM** \\ \hline COVID-Sentiment & 0.58 & 0.54 & 0.61 & **0.64** & 0.43 & 0.60 \\ U.S. Airline & 0.80 & 0.83 & **0.87** & **0.87** & 0.71 & 0.86 \\ IMDB & **0.84** & 0.78 & 0.81 & 0.82 & 0.76 & 0.81 \\ SST-2 & 0.90 & 0.90 & 0.93 & **0.94** & 0.85 & 0.92 \\ Sentiment140 & 0.75 & 0.74 & 0.79 & **0.79** & 0.53 & 0.77 \\ SemEval 2017 & 0.62 & 0.60 & 0.69 & **0.70** & 0.59 & 0.66 \\ Sanders corpus & 0.72 & 0.84 & 0.84 & 0.85 & 0.58 & **0.86** \\ UCI Sentence & 0.95 & 0.90 & 0.95 & **0.96** & 0.83 & 0.92 \\ AG-News & 0.85 & 0.86 & 0.90 & **0.91** & 0.75 & 0.90 \\
20-Newsgroups & 0.42 & 0.38 & 0.62 & **0.64** & 0.35 & 0.54 \\ \hline \hline \end{tabular}
\end{table} 
Table 10: SBERT vector-based AUC-PRC score for six classifiers. Best results are in bold face. \begin{table}
\begin{tabular}{l l l} \hline \hline
**Classifiers** & **Test-Statistic** & \(p\)**-Value** \\ \hline TSC-mean—RF & 1 & 0.017 \\ TSC-mean—ME & 15 & 0.6736 \\ TSC-mean—SVM & 21 & 0.8582 \\ TSC-mean—SGD & 21 & 0.8583 \\ TSC-Mean—LGBM & 7.5 & 0.5270 \\ \hline \hline \end{tabular}
\end{table} 
Table 11: Wilcoxon signed-rank test. necessary in order to extend this work to the original dataset, which is larger than the curated datasets, and to test whether balanced tolerance classes are indeed a definitive factor in classification performance. Another factor to consider is to use some filtering criteria to control the size of the vocabulary when computing TF-IDF vectors specific to the datasets. This could be important when comparing with TSC where a fixed size vector was used. Experiments with other word-embedding methods (e.g., Word2 Vec, GloVe, fastText) as well as examining the impact of the bag-of-words representation dimensionality will be explored. Conceptualization, Validation, Writing--original draft draft preparation, Supervision, S.R.