# Short Text Classification with Tolerance-Based Soft Computing Method
## 3 Materials and Methods
### Methods
best results in longer sentences or document-level classification tasks such as IMDB and 20-newsgroups datasets. Table 6 gives the experimental results with the transformer-based vectors. Here, our proposed TSC algorithm performs best in the U.S. Airline, IMDB, UCI SST-2 and 20-Newsgroup datasets and is comparable with the COVID-Sentiment, SST-2 and Sentiment 140 datasets. It can be observed that balanced tolerance classes are an indication of good semantic similarity between vectors generated by the transformer model. This can be seen with SST-2 and UCI sentence datasets that have approximately a similar number of tolerance classes with a weighted F1-score of over 85%. Another observation is that these two datasets contain only two sentiment classes. On the other hand, the TSC algorithm performs very well with the 20-Newsgroups datasets. Tables 7 and 8 give the weighted precision and recall scores for all the tested classifiers. Based on this score, the proposed TSC algorithm performs best in the UCI, SST-2, IMDB and 20-Newsgroups datasets. These results mirror the values obtained with the weighted F1-score except with the U.S. Airline dataset. Table 9 gives the AUC-ROC score for all the tested classifiers. A pair-wise comparison of all combinations (ovo) with the weighted average score parameter was used for obtaining the results. Based on the AUC-ROC score, the proposed TSC algorithm performs best in the UCI and SST-2 datasets that have an approximately similar number of tolerance classes and two classes. It is also noteworthy that the UCI and AGnews datasets have the best separability (90%), and IMDB (88%), 20-Newsgroups (87%) and SST-2 (85%) have scores above 84% based on the tested classifiers. Since SGD is a linear SVM, for most datasets, the weighted F1, Precision and Recall scores are either similar or identical. \begin{table}
\begin{tabular}{l l l l l l l} \hline
**Dataset** & **TSC-Mean** & **RF** & **ME** & **SVM** & **SGD** & **LGBM** \\ \hline COVID-Sentiment & 55 & 44 & 57 & 57 & **57** & 56 \\ U.S. Airline & **77** & 77 & 77 & 75 & 77 \\ IMDB & **76** & 69 & 73 & 73 & 72 & 72 \\ SST-2 & 85 & 85 & 85 & **86** & 85 & 85 \\ Sentiment140 & 70 & 68 & **72** & 72 & 66 & 70 \\ SemEval 2017 & 60 & 54 & **64** & 63 & 63 & 60 \\ Sanders corpus & 69 & 70 & **76** & 74 & 76 & 75 \\ UCI Sentence & **89** & 84 & 86 & 87 & 87 & 83 \\ AG-News & 82 & 79 & 88 & 81 & **88** & 83 \\
20-Newsgroups & **66** & 41 & 58 & 52 & 52 & 53 \\ \hline \end{tabular}
\end{table} 
Table 6: SBERT vector-based weighted F1-score (rounded) results for six classifiers. Best results are in bold face. \begin{table}
\begin{tabular}{l l l l l l} \hline
**Dataset** & **RF** & **ME** & **SVM** & **SGD** & **LGBM** \\ \hline COVID-Sentiment & **67** & 59 & 62 & 62 & 58 \\ U.S. Airline & 78 & 79 & 79 & 80 & **80** \\ IMDB & 83 & 87 & **89** & 89 & 86 \\ SST-2 & 82 & 82 & 82 & 82 & 74 \\ Sentiment140 & 71 & **74** & 72 & 72 & 73 \\ SemEval 2017 & 64 & 64 & 65 & 63 & **65** \\ Sanders corpus & 74 & 74 & 75 & **75** & 69 \\ UCI Sentence & 70 & 77 & 76 & 76 & 63 \\ AG-News & 78 & 84 & 84 & **85** & 82 \\
20-Newsgroups & 62 & 73 & 74 & **76** & 66 \\ \hline \end{tabular}
\end{table} 
Table 5: TF-IDF-based weighted F1-score (rounded) results for five classifiers. Best results are in bold face. Table 10, gives the AUC-PRC score for all the tested classifiers. The loss function for the SGD classifier was changed from _hinge_ (default) to _modified_huber_ to enable probabilistic outputs ([[https://scikit-learn.org/stable/modules/generated/sklearn.linear_model](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model)]([https://scikit-learn.org/stable/modules/generated/sklearn.linear_model](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model)). SGDClassifier.html accessed on 1 May 2022). Since all other metrics for SVD were based on the hinge loss function, the results for this classifier (indicated in blue) could be omitted in the overall analysis.