# Exploring Pandemics Events on Twitter by Using Sentiment Analysis and Topic Modelling
## 3 Methodology  
Based on our research questions, we have defined a methodology (summarised in Figure 1) able to reflect on people's opinions with respect to 11 epidemics using tweets. Data have been collected from Twitter developer platform according to keywords related to 11 viruses: cholera, Ebola, HIV/AIDS, influenza, malaria, Spanish influenza, swine flu, tuberculosis, typhus, yellow fever, and Zika. We have employed the tweepy package ([PERSON] et al., 2018) to scrawl data every Thursday at 9 a.m. Just English tweets have been considered, and retweets and replies have been filtered out. The considered period goes from 3 March to 15 September 2022, allowing to collect 369,472 tweets. In the original dataset, there are the following columns: datetime is the time of tweet posting; tweet id and author id are the unique identifiers of a tweet and its author, respectively; original text is the content of the tweet; retweet count, replay count, and like count show the interaction with a specific tweet; geo is the geolocation information of the tweet (if it is reported in the tweet). After obtaining the dataset for each epidemic and merging them together (de-duplication according to tweet id), it is essential to preprocess the text. The original texts have been cleaned by removing various noises, such as emojis and special characters, through the usage of natural language processing tasks. The cleaning steps are described in the following list:  
* Removing user names, hashtags, URLs, non-ASCII characters, numbers, punctuation, and special characters by using regular expressions;
* Making lowercase;
* Removing stop words that are common words (such as the, is, at, which, and on) but have no real meaning by using the spacy package [26];
* Performing lemmatisation to recover a word to its original form (e.g., transforming ate into eat) by using WordNetLemmatizer package [27];
* Correcting spelling errors by using the autocorrect package [28]. Once data are preprocessed, we have started to explore the findings by using basic visualisations. We have, e.g., plotted the dataset distribution, the number of tweets over time, the word frequencies, and the tweets' geolocation. We have transformed text into numerical representation, because the computer is not able to understand text directly. NLP provides several vectorisation techniques: some are based on sentences or they produce sentence vector representation directly, e.g., bag of words (BOW) [29] and term frequency (TF)-inverse document frequency (IDF) [30]. A bag of words is a simple representation of text that describes the occurrence of words within a document. TF-IDF evaluates how relevant a word is to a document in a collection of documents. Figure 1: Methodology overview. Some others NLP techniques focus on words to build word vectors, e.g., Word2 Vec [31] and FastText [32]. Word2 Vec is an unsupervised learning technique that uses a shallow, two-layer neural network to train and reconstruct linguistic contexts of words: this technique can utilise continuous bag of words (CBOW) or continuous skip-gram, where the model uses the current word to predict the surrounding window of context words. FastText is a word embedding and text classification method sourced by Facebook in 2016 that often achieves comparable accuracy to deep networks. According to these vectorisations' characteristics, different machine learning and natural language processing techniques can be applied. Clustering based on word embedding identifies similarity or dissimilarity between observations according to their distances. We have considered Word2 Vec and FastText for word embedding and \(k\)-means to calculate the Euclidean distance between observations [33]. Furthermore, we have used Latent Dirichlet Allocation (LDA)-based topic modelling [34] with bag of words to find the latent topics. Sometimes one tweet discusses more than one topic. In natural language processing, the topic modelling approach represents an unsupervised learning method to find topic distribution in corpus.