## 4 Proposed Method
### Data Cleaning (Pre-Processing)  
We target classifying short text on social media platforms, and in particular, tweets. Due to the informal nature of such short text messages, they are bound to contain characters and words that may not be essential for hate speech detection, such as mentions/usernames (e.g., @shivang), emoticons (e.g., @, @), special characters (e.g., @, +), punctuation marks (e.g.,?,!), and URLs (e.g., http://tco/hH50P5 pytX). Indeed, the experiments in Section 5  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Language** & **Class** & **Number of Tweets** \\ \hline \multirow{2}{*}{English} & Hate & 20,620 \\ \cline{2-3}  & Non-hate & 4163 \\ \hline \multirow{2}{*}{Hindi–English} & Hate & 4039 \\ \cline{2-3}  & Non-hate & 3729 \\ \hline \multicolumn{2}{c}{Total} & 32,551 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Dataset class label statistics.  
Figure 1: Flowchart of the proposed method.  
further corroborate this assumption. Therefore, the following pre-processing steps are carried out on all input tweets:  
1. **Decoding HTML**: The encoded HTML parts are decoded by replacing the HTML entity names or numbers with their original text representation. Some encoded HTML parts, such as \"&amp\", are not correctly decoded and, thus, are removed.
2. **Removal of emoticons**: All the emoticons encoded with UTF-8 BOM (Byte Order Marks) in the original text are first decoded and then removed.
3. **Removal of mentions (usernames)**: Since mentions are just other individuals' user-names tagged in the tweets, they do not contribute to any kind of sentiment in the tweet. Hence, they are removed.
4. **Removal of URL links**: All the URLs (http/https/www) and other links (including links of pictures/videos) in the tweets are removed, since they do not contribute to any kind of sentiment in the tweet.
5. **Conversion to lower case**: All the tweets are converted to lower case in order to have only lower-case letters for character-level representation/embedding.
6. **Expansion of negations**: As a special character, all apostrophes are removed. However, in doing so, the negative contractions lose their meaning. For example, words like \"can't\" end up as \"can t\", which is not an English word. Therefore, to avoid this situation, all negative contractions are appropriately expanded. Thus, a contraction such as \"can't\" is converted to \"cannot\".
7. **Removal of hashtags, punctuation marks, numbers, and special characters**: There are cases where the words/phrases used with the hashtag symbol (#) provide useful information about the tweets and their sentiment. Hence, only the \"#\" symbols are removed, in addition to other special characters, punctuation marks, and numbers.
8. **Removal of extra white spaces**: Any extra white space in the tweets is removed.
9. **Removal of duplicates**: A dataset may contain duplicate tweets (same tweet repeated multiple times). All such instances of duplications are removed, keeping only unique tweets to avoid these duplicates from falling in both the training and test set during the train-test split step and hence avoid overfitting. However, some duplicate tweets are mistakenly labeled as both hate and non-hate. With careful examination, all erroneously labeled duplicates are removed.
10. **Removal of null values**: Some tweets may be composed of entirely removable parts, such as mentions, links, and emoticons. Consequently, pre-processing such tweets renders them empty strings (null). Such tweets are removed entirely.  
After pre-processing our employed dataset, the final clean version consists of a total of 31,456 tweets with target class labels, where 7903 are classified as non-hate and 23,553 as hate. As a result, the dataset is imbalanced due to the unequal distribution of class labels.