# An Anomaly Detection Framework for Twitter Data
## 3 Social Media Data Analysis and Anomaly Detection
### Clustering Using K-Means  
Clustering is a method of automatically finding natural clusters of datasets and annotating input tweets to the final cluster in a specified feature space. This involves dividing tweets into groups based on their similarity. Two clusters, Cluster 0 and Cluster 1, are created using the most dominant Tweets in this section. Both groups have some keywords related to happiness, betterment, hope, success, love, good, morning, growth, and devel \begin{table}
\begin{tabular}{c c c c} \hline \hline good & thanks & think & beautiful \\ morning & best & always & blessed \\ India & future & love & friend \\ information & great & life & birthday \\ like & health & know & hope \\ happy & water & day & actually \\ photo & team & people & new \\ social & life & need & listen \\ well & future & everyone & dream \\ world & truth & nice & sorry \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Most frequent words using NMF and LDA. The dataset was collected from a Twitter developer account using a Twitter API key and an access token key. Tweets were mainly related to COVID-19 containing 10,586 tweets. Table 7 shows some sample datasets with date of creation and location. The retrieved tweets have different fields, namely, tweet_text, tweet_created_at, tweet_source, tweet_retweet, tweet_like, and tweet_location. The confusion matrix for this case study (Corona Virus Tweet Dataset) is depicted in Figure 6. The results are calculated by calculating the F1 score, precision, recall, and accuracy using Equations (2)-(9). The results show that the new framework detected anomalous data with 79% accuracy. \[\textit{Sensitivity or Recall}=\frac{\textit{Sum of all TP for each class}}{\textit{Sum of all TP}+\textit{Sum of all FN}} \tag{2}\]  
\[\textit{Sensitivity or Recall}=\frac{4816}{4816+1218}=0.7981 \tag{3}\]  
\[\textit{Precision}=\frac{\textit{Sum of all TP for each class}}{\textit{Sum of all TP}+\textit{Sum of all FP}} \tag{4}\]  
\[\textit{Precision}=\frac{4816}{4816+1050}=0.8210 \tag{5}\]  
\[\textit{F1 Score}=\frac{2(Precision*Recall)}{Precision+Recall} \tag{6}\]  
\[\textit{F1 Score}=\frac{2(0.8210*0.7981)}{0.8210+0.7981}=0.8093 \tag{7}\]  
\[\textit{Accuracy}=\frac{\textit{Sum of all TP for each class}+\textit{Sum of all TN}\textit{ for each class}}{\textit{Sum of all TP}+\textit{Sum of all TN}+\textit{Sum of all FP}+\textit{Sum of all FN}} \tag{8}\]  
\[\textit{Accuracy}=\frac{4816+3502}{4816+3502+1050+1218}=0.7857 \tag{9}\] \begin{table}
\begin{tabular}{l|c c} \hline \hline
**Tweets** & **Location** & **Created on** \\ \hline Hindu Pfizer is in another deadlock with the GOI They continue to insist they & Delhi, India & 22 May 2021 \\ will only provide their vaccine if the company wa & & \\ Pfizers COVID-19 vaccine can now can be stored at conventional refrigerator & Barcelona & 19 May 2021 \\ temperatures for up to a month as per Health & & \\ PublicHealth CUPHD is hosting COVID-19 walk-in vaccination clinics each & & \\ Friday from 830 AM400 PM wPfizer Moderna and Johnson amp & & \\ Can a COVID-19 vaccine make me sick with COVID-19 NO COVID-19 & & \\ vaccines contain the live virus that causes COVID-19 & & \\ My Grandommmy was very sick with pneumonia last month amp got over it & & \\ slowly I was so worried shed catch COVID-19 & & \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Sample tweets (Corona Virus Tweet Dataset) with the corresponding location and date. Figure 6: Confusion matrix of case study—Corona Virus Tweet Dataset. Here, _TP_ stands for true positive, _TN_ stands for true negative, _FP_ stands for false positive, and _FN_ stands for false negative. Based on cosine similarity, a tweet is defined as a normal or anomaly tweet. The average cosine similarity was 0.4549 for the considered dataset. The cosine similarity of each tweet was compared with the average cosine similarity. The tweet was treated as a normal tweet if the tweet had a greater cosine value than the average cosine value.