## 4 Proposed Hate Speech Detection Architecture
### Proposed Odia-English Text Preprocessing
#### 4.1.1 Removing (Cleaning) Irrelevant Character, Punctuations Symbol, and Emoji's
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Features**} & \multicolumn{5}{c}{**5-Folds Accuracy (\%) for RF Model**} & \multicolumn{1}{c}{**Average**} \\ \cline{2-7}  & **1 st** & **2 nd** & **3 rd** & **4 th** & **5 th** & **Accuracy** \\ \hline Word unigram & 74.44 & 72.12 & 72.80 & 73.97 & 72.87 & 73.24 \\ Word bigram & 73.82 & 71.62 & 71.80 & 74.17 & 72.57 & 72.80 \\ Word trigram & 73.52 & 71.92 & 72.10 & 73.67 & 73.47 & 72.92 \\ Combined n-gram & 73.32 & 72.02 & 72.40 & 74.97 & 71.87 & 72.92 \\



In addition to the result accuracy score obtained by the 5-fold CV, this paper also uses other models' performance evaluation metrics. These metrics are Precision (P), Recall(R), and F-score (F1). Table 10 shows the results of the evaluation metrics of each model based on the features extracted. It further uses the normalized confusion matrix of the models that use 5-fold prediction, with the results shown in Figures 6-8 below. Based on the F1-score, the SVM model with word2 vec obtains a higher score of 73% than the RF and NB models. However, the accuracy of the RF model is 75.39% which is higher than that of the SVM and NB models using the word2 vec features (Figure 5). F1-score metrics are selected in order to compare the models based on the features because this is more useful than the accuracy when the dataset contains uneven class distribution. Figure 6 illustrates the sampled confusion matrix for SVM models. SVM models based on bigram classify 79% of HS and 55% of NH correctly, but 21% of HS and 45% of NH are misclassified; SVM models based on TF-IDF classify 78% of HS and 50% of NH correctly, but 22% of HS and 50% of NH are misclassified; and SVM models based on word2 vec classify 73% of HS and 72% of NH correctly, but 27% of HS and 28% of NH are misclassified. Figure 7 illustrates the sampled confusion matrix for the NB models. NB models based on combined n-grams classify 91% of HS and 38% of NH correctly, but 9% of HS and 62% of NH are misclassified; NB models based on TF-IDF classify 86% of HS and 40% of NH correctly, but 14% of HS and 60% of NH are misclassified; and NB models based on word2 vec classify 73% of HS and 65% of NH correctly, but 27% of HS and 35% of NH are misclassified. Similarly, Figure 8 shows the confusion matrix of the RF model. RF models based on unigram classify 89% of HS and 34% of NH correctly, but 11% of HS and 66% of NH are misclassified; RF models based on TF-IDF classify 90% of HS and 33% of NH correctly, but 10% of HS and 67% of NH are misclassified; and RF models based on word2 vec classify 96% of HS and 28% of NH correctly, but 4% of HS and 72% of NH are misclassified. A normalized confusion matrix for binary classifier models based on the feature extracted using the prediction result of 5-fold CV for the models is shown in Figures 6-8 respectively. The actual class is the labels post or comment in the dataset and the predicted class is the prediction labels made by the models. The heatmap represents the predicted or classified instance of posts and comments in each class. Finally, the result of the binary models demonstrates that the NB model based on n-grams shows better accuracy than the RF and SVM models. However, the RF models based on word2 vec and TF-IDF show higher accuracy than the SVM and NB models. Furthermore, SVM models with word2 vec give better classification results than both models. Figure 6: Confusion matrix of sample binary SVM models based on extracted features. ### Ternary Classification Models Evaluation Results

The models were also trained based on the prepared three-class dataset. The trained models are tested using the 5-fold CV. Tables 11-13 show the accuracy scores of the models trained by SVM, NB, and RF, respectively, with various feature extraction vectors. Table 11 reveals that the SVM model using TF-IDF feature has the lower average accuracy of 48.51%, and the SVM model using word2 vec feature has the higher average accuracy of 53.35%.