# An Anomaly Detection Framework for Twitter Data
## 3 Social Media Data Analysis and Anomaly Detection
### Topic Modeling  
Topic modeling is a popular approach for recognizing the top word from a corpus of the tweet dataset, and is used to create a query set to find a similarity score with each tweet of the corpus. A pre-processed tweet dataset is used for topic modeling methods. We used two topic modeling techniques, namely NMF and LDA. NMF is an unlabeled subject on which training is given. It factors a high dimensional vector into a low dimensional vector. This creates two matrices, W and H, for the subjects by term and by coefficients. LDA is used to map each document in the corpus with a set of topics and assign a probability to each document related to a topic. The first step is to vectorize each tweet using the TF-IDF vectorizer. Once the tweets are converted into vectors, the second step uses NMF and LDA methods to extract the hidden topics from the large tweet dataset. Figure 3: Highlighted words based on their frequency and relevancy in the tweets. Figure 2: Sixty most frequently appearing words in the considered datasets. * Topic modeling using NMF. This method is based on the dominant score of each topic, which is used to select topics with a higher score to form a query set. It involves finding the top 10 topics from the entire dataset, and each topic with the 15 most similar words. It obtains the leading topic for each tweet and finds the most relevant topic for each tweet. Based on the dominant score of each topic with each tweet, its finds the most dominant topics, which are ordinary tweets. These tweets are used to find anomalies in a new set of tweet datasets. The most dominant tweets from the entire dataset are selected that are most relevant to the above ten topics. Table 4 shows the top 10 topics; each has 15 words. Table 5 shows the sample's most dominant tweets. * Topic modeling using LDA. This method is based on the weightage of each word in each topic. The top 10 topics from the entire dataset are found; each topic has 15 of the most similar words with their weightage. The most weighted topics are used to create a query string. Here most frequently used topics identified by topic modeling were: know, food, happy, fan, like, thank you, people, nice, really, time, great, love, today, strong, day, good morning, always, think, look, want, best, help, never, please, and make. The third step is to collect the most frequently generated words using the NMF and LDA topic modeling methods. The most frequent words using NMF and LDA topic modeling are shown in Table 6. \begin{table}
\begin{tabular}{c c} \hline \hline
**S. No.** & **Topic** \\ \hline
1 & know time people want make think see even go still say take would thing never \\
2 & good morning good morning dawn good dawn drink beautiful luck ka nice happy today water look everyone \\
3 & like look like feel feel like would like sound act something looking lol unity seems really \\
4 & one person one best another also right ever give best made top many buy last two \\
5 & love much always thank listen radio would playing happy channel unity fall art give birthday \\
6 & day nice every happy got today morning last great ka hope another beautiful drink start \\
7 & get check shope get shope free airdrop mx mar mx get airdrop mar rm soon mexcglobal participate well \\
8 & project best great team hope happy future nft labor really crypto thank community strong amazing \\
9 & need really help please stop india learn programming go much freelance hope unity business engagement \\
10 & new nft come india year live check news season start design next song update token \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Top 10 topics having 15 most similar words.