## 3 Methodology
### NLP Pipeline
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Algorithm** & **Brief Description** & **Advantages** & **Disadvantages** \\ \hline \multirow{4}{*}{Whitespace} & Splits tokens based on whitespace. & Simple, fast. & Not suitable for languages where whitespace does not denote word boundaries. \\ \cline{2-4}  & Word & Splits text into words using NLTK’s word\_tokenize. & Robust, handles punctuation. & Slower compared to whitespace tokenizer. \\ \cline{2-4}  & Punke & Language-independent tokenizer. & Good for European languages, handles punctuation. & May not work well for languages with different sentence structures. \\ \cline{2-4}  & Regexp & Tokenization based on regular expression patterns. & Highly customizable. & Requires good understanding of regular expressions. \\ \cline{2-4}  & Sentence & Splits text into sentences. & Useful for document summarization and segmentation. & Not useful for word-level analysis. \\ \cline{2-4}  & SpaCy & Tokenizer provided by the SpaCy library. & Fast, handles multiple languages, robust. & Requires installing the SpaCy library and language models. \\ \hline \multirow{4}{*}{PorterStemmer} & A well-known stemming algorithm. & Effective for English, reduces words to base form. & Over-stemming or under-stemming possible. \\ \cline{2-4}  & Lancaster Stemmer & More aggressive than Porter. & Reduces words to a basic form. & Can produce stems that are not meaningful words. \\ \cline{2-4}  & Snowball Stemmer & Extends Porter to support multiple languages. & Language support, aggressive. & Over-stemming or under-stemming possible. \\ \cline{2-4}  & WordNet Lemmatizer & Lemmatizer based on WordNet lexical database. & Produces valid words, less aggressive than stemming. & Slower, may require POS tags for accurate lemmatization. \\ \cline{2-4}  & SpaCy Lemmatizer & Lemmatizer offered by the SpaCy library. & Fast, handles multiple languages, usually more accurate. & Requires installing the SpaCy library and language models. \\ \hline \multirow{4}{*}{Poorty} & CountVectorizer & Converts text to a matrix of token counts. & Simple, effective for most cases. & Does not consider semantic meaning, sensitive to frequent words. \\ \cline{2-4}  & TfidfVectorizer & Converts text to a matrix of TF-IDF features. & Considers global importance of words, less sensitive to frequent words. & Sensitive to the scale of the dataset, more computationally intensive than CountVectorizer. \\ \hline \multirow{4}{*}{Poorty} & Latent Dirichlet Allocation & Most used topic modeling algorithm. & Effective for a wide range of topics, easy to interpret. & Requires choosing the number of topics a priori, may not be suitable for all types of text data. \\ \cline{2-4}  & Non-negative Matrix Factorization & Useful for topic modeling and other types of clustering. & Fast, easier to interpret than LDA. & Assumes linear structure, may not work well for all types of data. \\ \cline{2-4}  & Latent Semantic Analysis & Also known as latent semantic indexing (LSI). & Good for capturing semantic meaning, less sensitive to word frequency. & High computational cost for large datasets. \\ \cline{2-4}  & Non-negative Matrix Factorization & Useful for topic modeling and other types of clustering. & Fast, easier to interpret than LDA. & Assumes linear structure, may not work well for all types of data. \\ \cline{2-4}  & Latent Semantic Analysis & Also known as latent semantic indexing (LSI). & Good for capturing semantic meaning, less sensitive to word frequency. & High computational cost for large datasets. \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Algorithm options for the NLP pipeline. The CountVectorizer calculated TF as the feature importance, which is the number of times that a term appeared in a document. The TfidVectorizer tampers the TF by an inverse document frequency (IDF), effectively attenuating the importance of a word based on its global frequency in the corpus. Hence, if a word appears very frequently in the target document but also frequently in other documents, its importance for that document will be lower.