## 4 Proposed Hate Speech Detection Architecture
### Proposed Odia-English Text Preprocessing
#### 4.1.1 Removing (Cleaning) Irrelevant Character, Punctuations Symbol, and Emoji's
Table 12 shows that the NB model using the unigram feature has the lower average accuracy of 41.89%, and the NB model using the word2 vec feature has the higher average accuracy of 49.57%. Table 13 shows that the EF model using the bigram feature has the lower average accuracy, at 50.08%, and the EF model using the word2 vec feature has the higher average accuracy, at 55.05%. The bar chart of Figure 9 visualizes the 5-fold CV average accuracy of each model based on the features in Tables 11-13 for the ternary classification experiments. It reveals that the NB models result in a lower score than the SVM and RF models. The \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Feature**} & \multicolumn{2}{c}{**SVM**} & \multicolumn{2}{c}{**NB**} & \multicolumn{2}{c}{**RF**} \\ \cline{2-10}  & **P** & **R** & **F1** & **P** & **R** & **F1** & **P** & **R** & **F1** \\ \hline Word unigram & 0.72 & 0.71 & 0.72 & 0.73 & 0.75 & 0.71 & 0.71 & 0.73 & 0.70 \\ Word bigrams & 0.72 & 0.72 & 0.72 & 0.73 & 0.75 & 0.72 & 0.71 & 0.73 & 0.70 \\ Word trigrams & 0.72 & 0.72 & 0.72 & 0.73 & 0.75 & 0.72 & 0.71 & 0.73 & 0.70 \\ Combined n-grams & 0.72 & 0.71 & 0.72 & 0.73 & 0.75 & 0.72 & 0.70 & 0.72 & 0.70 \\ TF-IDF & 0.70 & 0.70 & 0.70 & 0.70 & 0.72 & 0.71 & 0.71 & 0.73 & 0.70 \\ TF-IDF+ combined n-grams & 0.70 & 0.71 & 0.70 & 0.72 & 0.74 & 0.72 & 0.70 & 0.72 & 0.70 \\ Word2 vec & 0.76 & 0.73 & 0.73 & 0.73 & 0.71 & 0.72 & 0.75 & 0.75 & 0.72 \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Classification performance result of binary models. Figure 8: Confusion matrix of sample binary RF models based on extracted features. Figure 7: Confusion matrix of sample binary NB models based on extracted features. SVM model achieves a higher score on bigram, trigram and combined n-gram. However, the RF model has the highest score using TD-IDF and word2 vec based features. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Extracted Feature**} & \multicolumn{4}{c}{**5-Folds Accuracy Scores (\%) for the RF Model**} & \multicolumn{2}{c}{**Average Accuracy**} \\ \cline{2-7}  & **1 st** & **2 nd** & **3 rd** & **4 th** & **5 th** & **(\%)** \\ \hline Word unigram & 52.6 & 50.6 & 50.4 & 52.8 & 48.9 & 51.07 \\ Word bigrams & 52.9 & 50.4 & 51.1 & 51.0 & 48.7 & 50.08 \\ Word trigrams & 52.4 & 51.7 & 49.0 & 51.4 & 49.0 & 50.69 \\ Combined n-grams & 53.3 & 50.5 & 50.6 & 50.9 & 48.4 & 50.75 \\ TF-IDF & 52.2 & 50.0 & 48.2 & 52.2 & 49.3 & 50.38 \\ TF-IDF+ combined & 52.9 & 50.1 & 49.7 & 49.9 & 50.3 & 50.59 \\ Word2 vec & 56.0 & 56.7 & 53.0 & 54.3 & 55.2 & 55.05 \\ \hline \hline \end{tabular}
\end{table} 
Table 13: RF models’ accuracy scores on each features using three class dataset. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Feature Models**} & \multicolumn{4}{c}{**5-Folds Accuracy Scores (\%) for NB Model**} & \multicolumn{2}{c}{**Average Accuracy**} \\ \cline{2-7}  & **1 st** & **2 nd** & **3 rd** & **4 th** & **5 th** & **(\%)** \\ \hline Word unigram & 40.2 & 42.8 & 44.3 & 38.8 & 43.5 & 41.89 \\ Word bigrams & 41.8 & 44.1 & 47.2 & 42.6 & 43.8 & 43.87 \\ Word trigrams & 41.9 & 44.5 & 47.6 & 42.7 & 43.8 & 44.07 \\ Combined n-grams & 52.7 & 51.4 & 48.8 & 47.1 & 48.6 & 49.73 \\ TF-IDF & 48.4 & 48.3 & 47.4 & 45.9 & 47.4 & 47.45 \\ TF-IDF+ combined & 49.5 & 49.3 & 48.4 & 46.9 & 48.0 & 48.39 \\ Word2 vec & 54.0 & 51.0 & 49.5 & 46.0 & 47.3 & 49.57 \\ \hline \hline \end{tabular}
\end{table} 
Table 12: NB models’ accuracy scores on each features using three class dataset. Figure 9: Ternary models’ comparison using CV average accuracy. Table 14 shows the results of the evaluation metric of each model, based on the features extracted and the normalized confusion matrix of the models using 5-fold CV prediction results. Comparing the result of the F1-score, the SVM model that was based on word2 vec attained an F1-score of 53%; a higher score than the rest of the models. However, in terms of accuracy, the RF model (at 55.5%) was higher than the NB and SVM models with word2 vec features.