# Short Text Classification with Tolerance-Based Soft Computing Method
## 3 Materials and Methods
### Methods
It should be noted that \(\varepsilon\) values range from 0.16 to 0.32. Other datasets with three and four sentiment classes do not generate balanced tolerance classes. Random Forest (RF) [34], Maximum Entropy (ME), Support Vector Machine (SVM) [35], Stochastic Gradient Decent (SGD) [36] and Light Gradient Boosting Machine (LGBM) [37] classifier implementations in Scikit-learn ([[https://scikit-learn.org/stable/](https://scikit-learn.org/stable/)]([https://scikit-learn.org/stable/](https://scikit-learn.org/stable/))) (accessed on 1 May 2022) with the following parameters were used. For the RF classifier, 100 trees and gini index were used to determine the quality of split. The minimum and maximum samples were set to 2 and 1, respectively, to split an internal node of the tree, and the maximum number of features was set to 27 (square root of the size of the vector). Bootstrap samples were used when building trees and the random_state parameter was set to 42. For the ME (logistic regression) classifier, the l2 penalty term with the stopping criteria set to \(10^{-4}\) was used. The RBF kernel was used for the SVM classifier with a kernel cache size = 200 MB, gamma set to scale, C value set to 1 and l2 penalty term. The hinge loss function was used in the SGD classifier with default values for other parameters (penalty l2, alpha set to 0.0001, maximum iterations set to 1000 and learning rate set to optimal). Since the loss is hinge, this classifier is a linear SVM. For the LGBM classifier, the max_leaf_nodes was set to 31, and the learning_rate, n_estimators, min_child_weight, and min_child_samples were set to 0.1, 100, \(10^{-3}\), and 20, respectively. Table 4 shows the AG-News dataset with a well-balanced tolerance class distribution for the four categories (TC-World, TC-Sports, TC-Business and TC-Science) for \(\varepsilon=0.2\). It should be noted all algorithms perform (third best) on this dataset. The 20-Newsgroups dataset has the highest number of categories among all other datasets. It has twenty sentiment classes and a fairly balanced tolerance class distribution as shown in Table 2. Due to better semantic similarity of its vectors, the performance of TSC on this dataset is better than the COVID-sentiment and SemEval 2017 datasets. Table 5 gives the weighted F1-score for all datasets with the TF-IDF-based ML algorithms. The size of TF-IDF vectors depends on the vocabulary of that dataset, which means having longer sentences results in better vocabulary for the TF-IDF approach, which is an advantage over transformer vectors. While building vocabulary with TF-IDF, the frequency of the words was considered to compute the vectors. We considered default parameters of TF-IDF to build the vocabulary. The minimum and maximum document frequency was set to the default value of 1.0. The results show that TF-IDF-based ML algorithms give the \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Dataset** & \(\varepsilon\)**Value** & **TC-World** & **TC-Sports** & **TC-Business** & **TC-Science** \\ \hline AG-News & 0.2 & 1900 & 2607 & 2563 & 2398 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Best \(\varepsilon\) value for the TSC algorithm and tolerance class size for AG-News dataset. \begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Dataset** & \(\varepsilon\)**Value** & **TC-Positive** & **TC-Negative** & **TC-Neutral** & **TC-Irrelevant** \\ \hline COVID-Sentiment & 0.23 & 249 & 1136 & 2590 & - \\ \hline U.S. Airline & 0.32 & 2009 & 8202 & 1234 & - \\ \hline IMDB & 0.26 & 6979 & 12,956 & - & - \\ \hline SST-2 & 0.23 & 6501 & 5531 & - & - \\ \hline Sentiment140 & 0.16 & 926 & 1969 & - & - \\ \hline SemEval 2017 & 0.26 & 6583 & 1755 & 5797 & - \\ \hline Sanders corpus & 0.24 & 205 & 390 & 1415 & 1518 \\ \hline UCI Sentence & 0.17 & 576 & 557 & - & - \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Best tolerance \(\varepsilon\) value with sizes of tolerance classes for the positive, negative, neutral and irrelevant text polarities.