# Short Text Classification with Tolerance-Based Soft Computing Method
## 3 Materials and Methods
### Methods
Based on the AUC-PRC score, the proposed TSC algorithm performs best with the IMDB dataset and is comparable with the UCI, SST-2, Sentiment140 and COVID-Sentiment datasets. In terms of the three reported metrics (weighted F1, AUC-ROC, AUC-PRC) from Tables 6, 9 and 10 respectively, overall, the proposed TSC algorithm performs best with mostly balanced datasets having two sentiment classes (binary classification) with the U.S. Airline dataset being the exception. TSC performs poorly with two highly imbalanced datasets having more than sentiment classes (i.e., [PERSON] corpus and SemEval 2017). The weighted F1 score is computed only on predicted classes, whereas the AUC scores reflect the performance of a classifier over a range of values (prediction score). In comparison \begin{table}
\begin{tabular}{l l l l l l l} \hline \hline
**Dataset** & **TSC-Mean** & **RF** & **ME** & **SVM** & **SGD** & **LGBM** \\ \hline COVID-Sentiment & 0.58 & **0.60** & 0.59 & 0.59 & 0.59 & **0.60** \\ U.S. Airline & 0.77 & 0.75 & **0.79** & 0.76 & 0.78 & 0.77 \\ IMDB & **0.76** & 0.69 & 0.73 & 0.73 & 0.72 & 0.72 \\ SST-2 & **0.85** & 0.83 & **0.85** & **0.85** & **0.85** & **0.85** \\ Sentiment140 & 0.71 & 0.69 & **0.72** & **0.72** & 0.71 & 0.71 \\ SemEval 2017 & 0.59 & 0.58 & **0.65** & 0.64 & 0.63 & 0.62 \\ Sanders corpus & 0.70 & **0.78** & 0.76 & 0.74 & 0.77 & **0.78** \\ UCI Sentence & **0.90** & 0.87 & 0.86 & 0.87 & 0.87 & 0.85 \\ AG-News & 0.81 & 0.80 & **0.85** & 0.82 & 0.82 & 0.83 \\
20-Newsgroups & **0.67** & 0.42 & 0.59 & 0.53 & 0.60 & 0.51 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: SBERT vector-based Weighted Precision score for six classifiers. Best results are in bold face. \begin{table}
\begin{tabular}{l l l l l l l} \hline \hline
**Dataset** & **TSC-Mean** & **RF** & **ME** & **SVM** & **SGD** & **LGBM** \\ \hline COVID-Sentiment & 0.56 & 0.50 & **0.57** & **0.57** & **0.57** & **0.57** \\ U.S. Airline & 0.78 & 0.76 & **0.80** & 0.77 & 0.78 & 0.79 \\ IMDB & **0.76** & 0.69 & 0.73 & 0.73 & 0.72 & 0.72 \\ SST-2 & **0.85** & 0.83 & **0.85** & **0.85** & **0.85** & **0.85** \\ Sentiment140 & 0.70 & 0.68 & **0.72** & **0.72** & 0.67 & 0.70 \\ SemEval 2017 & 0.59 & 0.57 & **0.64** & 0.63 & 0.63 & 0.61 \\ Sanders corpus & 0.70 & 0.76 & 0.77 & 0.75 & 0.76 & **0.78** \\ UCI Sentence & **0.89** & 0.84 & 0.86 & 0.87 & 0.87 & 0.83 \\ AG-News & 0.81 & 0.80 & **0.85** & 0.82 & 0.82 & 0.83 \\
20-Newsgroups & **0.66** & 0.44 & 0.58 & 0.52 & 0.55 & 0.50 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: SBERT vector-based Weighted Recall score for six classifiers. Best results are in bold face. \begin{table}
\begin{tabular}{l l l l l l l} \hline \hline
**Dataset** & **TSC-Mean** & **RF** & **ME** & **SVM** & **SGD** & **LGBM** \\ \hline COVID-Sentiment & 0.65 & **0.71** & 0.68 & 0.70 & 0.70 & 0.67 \\ U.S. Airline & 0.75 & 0.77 & 0.76 & 0.77 & 0.77 & **0.79** \\ IMDB & 0.76 & 0.83 & 0.83 & 0.86 & **0.88** & 0.87 \\ SST-2 & **0.85** & 0.82 & 0.81 & 0.82 & 0.82 & 0.73 \\ Sentiment140 & 0.70 & 0.71 & **0.74** & 0.72 & 0.73 & 0.73 \\ SemEval 2017 & 0.66 & 0.71 & 0.71 & 0.71 & 0.70 & **0.72** \\ Sanders corpus & 0.77 & 0.77 & 0.78 & 0.79 & **0.80** & 0.76 \\ UCI Sentence & **0.90** & 0.72 & 0.78 & 0.77 & 0.76 & 0.67 \\ AG-News & 0.87 & 0.86 & **0.90** & **0.90** & **0.90** & 0.88 \\
20-Newsgroups & 0.82 & 0.81 & 0.86 & 0.86 & **0.87** & 0.82 \\ \hline \hline \end{tabular}
\end{table} 
Table 9: SBERT vector-based AUC-ROC score for six classifiers. Best results are in bold face. with other classifiers, the proposed TSC algorithm does better in more datasets with the AUC-ROC score. However, if we examine the overall AUC scores \(\geq\) 80, TSC gives better performance with five datasets using the AUC-PRC score as compared to four datasets with the AUC-ROC score. Another point to note is that the size of tolerance classes for negative sentiment is larger the size of tolerance classes for the positive sentiment in two of these datasets (i.e., IMDB and US Airline).