STREAM: Simplified Topic Retrieval, Exploration, and Analysis Module (2024)

Abstract: Topic modeling is a widely used technique to analyze large document corpora. With the evergrowing emergence of scientific contributions in the field, non-technical users may often use the simplest available software module, independent of whether there are potentially better models available. We present a Simplified Topic Retrieval, Exploration, and Analysis Module (STREAM) for user-friendly topic modelling and especially subsequent interactive topic visualization and analysis. For better topic analysis, we implement multiple intruderword based topic evaluation metrics. Additionally, we publicize multiple new datasets that can extend the so far very limited number of publicly available benchmark datasets in topic modeling. We integrate downstream interpretable analysis modules to enable users to easily analyse the created topics in downstream tasks together with additional tabular information. The code is available at the following link: https://github.com/AnFreTh/ STREAM

heading: 1 Introduction
text: Identifying latent topics within extensive text corpora is a fundamental task in the field of Natural Language Processing (NLP) and has been of larger scientific interest since the early 2000s (Hofmann, 2001; Blei et al., 2003). Especially with the emergence of contextualized embeddings, extraction algorithms and topic models continue to evolve and achieve increasingly impressive results in terms of topic coherence (Larochelle and Lauly, 2012; Srivastava and Sutton, 2017; Chien et al., 2018; Wang et al., 2019; Dieng et al., 2020). Even, methodologically simpler methods achieve state-ofthe-art results by leveraging document and wordembeddings (Sia et al., 2020; Grootendorst, 2022; Angelov, 2020).
The publication of open source software like Gensim (Řehůřek and Sojka, 2010), the Natural
Language Tool Kit (nltk) (Bird et al., 2009) or SpaCy (Vasiliev, 2020) have enabled researchers to apply such models in various fields, including education (Granić and Marangunić, 2019), offsite construction (Liu et al., 2019), bioinformatics (Liu et al., 2016), communication sciences (Maier et al., 2018), finance (Thormann et al., 2021) and numerous other applications (e.g., (Hall et al., 2008; Daud et al., 2010; Boyd-Graber et al., 2017; Kant et al., 2022; Thielmann et al., 2021; Hannigan et al., 2019; Tillmann et al., 2022)).
The OCTIS (optimizing and comparing topic models is simple) (Terragni et al., 2021a) framework in particular has found favor in the scientific community and made fitting and evaluating sophisticated topic models easy and efficient. However, OCTIS lacks the methodologically simpler yet very performant models such as clustering based topic extraction (Sia et al., 2020; Angelov, 2020) and the user-centric implementation of BERTopic (Grootendorst, 2022). Especially the user-friendly implementation and visualization possibilities of BERTopic allow non-technical users to easily analyze their document corpora and visualize their results which has led to a variety of use cases especially in the social sciences (e.g. (Falkenberg et al., 2022; Jeon et al., 2023; Zankadi et al., 2023)).
We thus contribute the STREAM (Simplified Topic Retrieval Exploration and Analysis Module) software package. It gets its acronym not only from the easy to use, user-centric topic modelling, evaluation and exploration implementation but also from the integration of downSTREAM models to analyze topic contributions to regression or classification problems.
The core of the STREAM package is built on top of the OCTIS framework and allows seamless integration of all of OCTIS’ multitude of models, datasets, evaluation metrics and hyperparameter optimization techniques.
435

heading: 1.1 Contributions
text: The contributions of STREAM can be summarized as follows:
• STREAM integrates multiple clustering based topic models into the OCTIS framework (see the Appendix for a full list of all available models).
• Through interactive visualization methods, STREAM allows easy exploration and analysis of all models.
• We publicize multiple multi-modal datasets to enable researchers to compare their models beyond the standard topic modeling datasets, such as 20NewsGroups and Reuters (Mitchell, 1999; Lewis, 1997).
• STREAM integrates interpretable downstream modeling by introducing a Neural Additive Topic Model (NAM) (Agarwal et al., 2021) that incorporates the documents topicprevalences along further structural variables into an interpretable downstream regression or classification model.

heading: 2 Model Fitting and OCTIS Integration
text: STREAM is effectively built upon the core concepts of the OCTIS package and inherits from the AbstractModel, AbstractMetric and OctisDataset classes. Thus, all models, evaluation metrics, visualization functions, datasets and downstream models are perfectly integrable with all of OCTIS’ models and metrics.
Datasets Creating custom datasets including tabular data is as simple as running the following few lines of code: from stream .data_utils import TMDataset df = pd .read_csv ("your_data.csv" )
dataset = TMDataset ( ) dataset .create_load_save_dataset (
data=df , dataset_name="your_name" , save_dir="save directory" , doc_column="text" , #column name where documents are stored label_column="popularity" )
All textual data is preprocessed according to the users specifications of the preprocessing pipeline and therefore, e.g., lower cased, stopwords removed and lemmatized. In the specified directory, the necessary files and a .csv file storing the tabular data are saved.
Model fitting Fitting a model (here e.g. a simple Kmeans clustering topic model) can subsequently be done simply by running the following code: from stream .models import KmeansTM
model = KmeansTM (num_topics=20) model_output = model .train_model (dataset ) Depending on the model, the hyperparameters can easily be adjusted. Note, that all STREAM datasets are fully usable with all OCTIS models and users can thus easily fit e.g. a LDA (Blei et al., 2003) or ETM (Dieng et al., 2020) on the TMDataset class.
Evaluation STREAM offers multiple new, intruder-word based topic evaluation metrics (Thielmann et al., 2024b) alongside classical NPMI coherence scores (Lau et al., 2014), computed over the complete documents and not over sliding windows, and also Embedding based Coherence metrics (Terragni et al., 2021b). See the Appendix for an overview over all available metrics. The evaluation of a model can thus be done by simply running: from stream .metrics import ISIM metric = ISIM (dataset ) metric .score (model_output )

heading: 2.1 Available Datasets
text: In addition to the implemented models, metrics and downstream tasks, we publicize multiple datasets suited for topic model comparison.
• Multiple Spotify datasets comprised of the songs’ lyrics and various tabular features, such as the popularity, danceability or acousticness of the songs.
• A new Reddit dataset, which is filtered for "Gamestop" (GME) from the Subreddit "r/wallstreetbets". The data is taken from the thread "What are your moves tomorrow?". It is covering the time around the GME short squeeze of 2021.
• A new Stocktwits dataset also filtered for "Gamestop" (GME). It is covering the time around the GME short squeeze of 2021.
• In addition, we upload the preprocessed Reuters and Poliblogs (Roberts et al., 2018) datasets that are well suited for comparing topic model outputs.

heading: 2.2 Topic Analysis
text: One of the core concepts of topic modelling is the subsequent qualitative and visual analysis of the created topics. In addition to the available topicword-lists and matrices, STREAM implements multiple visualization methods to easily analyze the created topics. Besides classical wordclouds, the created topic clusters, topical distances, or top word distributions can be interactively visualized. from stream .visuals import visualize_topic_model visualize_topic_model (model , port=8050)

heading: 3 Downstream Tasks
text: While the visual analysis of topics is often very helpful in analyzing a large corpus, the contents of documents often also have effects on other variables. Roberts et al. (2018) e.g. introduced a model that captures the effects of additional tabular variables on topics. STREAM offers the possibility to analyze the effects of topics and additional tabular variables on any given target variable, via implementing a downstream NAM1. The general form of a NAM can be written as:
E(y) = h  β + J∑
j=1
fj(xj)
  , (1)
where h(·) is the activation function used in the output layer, e.g. linear activation for a simple regression task or softmax activation for a classification task. x ∈ Rj are the input features, β describes the intercept. The shape-functions are expressed as fj : R → R and represent the MultiLayer Perceptron (MLP) corresponding to the j-th feature. The model structure of a simple NAM is given in Figure 3.
1see an example in the appendix
Further, let x ≡ (xtab,xdoc) denote the categorical and numerical (continuous) structural features xtab and xdoc denote the documents. After fitting a topic model (see section 2), STREAM extracts the documents topical prevalences and thus "creates" z ≡ (xtab,xtop), a probability vector over the documents and topics. Note, that x(i)j(tab) denotes the jth tabular feature of the i-th observation and x(i)k(top) denotes document i-th topical prevalence for topic k. In order to preserve interpretability the available downstream model is given by:
h(E [y]) = β + J∑
j=1
fj(xj(tab)) + K∑
k=1
fk(xk(top)),
(2) Thus, the visualization of shape-function fk shows
the impact topic k has on a target variable y and the visualization of fj shows the impact of tabular feature j. With the given datasets and examples available in STREAM, this could represent the effect a topic created from the Spotify dataset and a songs duration have on a songs popularity. With a fitted topic model (see section 2), fitting a downstream model is straight forward leveraging the pytorch trainer class. Subsequently, all shape functions can easily be visualized similar to the plots introduced by Agarwal et al. (2021). from pytorch_lightning import Trainer from stream .NAM import DownstreamModel
# Instantiate the DownstreamModel downstreammodel = DownstreamModel (
trained_topic_model=topic_model , #your trained topic model target_column='day' , #specify your target column task='regression' , #or 'classification' dataset=dataset , batch_size=128 , lr=0.0005
) # Use PyTorch Lightning's Trainer to train and
validate the model trainer = Trainer (max_epochs=10) trainer .fit (downstreammodel )
# Plotting from stream .visuals import plot_downstream_model plot_downstream_model (downstream_model )

heading: 4 Conclusion
text: In this paper, we present the STREAM framework. A user-friendly topic modeling module for creating datasets, training and evaluating topic models, visualizing results and fitting interpretable downstream models. The proposed framework is a python library and closely interacts with the existing OCTIS framework from Terragni et al. (2021a).
Future adaptations could include the integration of further more performant or e.g. distributional downstream models (Chang et al., 2022; Luber et al., 2023; Thielmann et al., 2024a) to further allow researchers to analyze the effect a topic has on a regression or classification task.

heading: 5 Limitations
text: We present a python package for topic modeling. While all implemented models, visualizations and the downstream models are straightforward, the actual interpretation of the results and figures is still done by the user. Given that especially textual data might include a lot of noise or harmful language, we must therefore stress the users to be careful in their final assessment of their created results.
Additionally, while NAMs (Agarwal et al., 2021) offer visual interpretability, they do not allow for statistical significance as the more theoretical Generalized Additive Models (Wood, 2017) or direct causal inference.

heading: A Appendix
text: A.1 Available Models Multiple topic model/document clustering and subsequent topic extraction models are available in STREAM. Additionally, STREAM inherits from all models available in OCTIS. Thus, the following models are available:
The SomTM is described in Honkela (1997). WordCluTM follows the word clustering approach introduced by Sia et al. (2020). CEDC is described in Thielmann et al. (2024b). The KMeansTM is similar to Grootendorst (2022) and often used as a fast-compute benchmark model. DCTE is a semisupervised few-shot model introduced in Thielmann et al. (2024c). TNTM is introduced in Reuter et al. (2024). CTMneg is based on CTM (Bianchi et al., 2021) and introduced by Adhya et al. (2022). CBC is the only model of the STREAM models not based on document embeddings and focuses on coherence scores between documents, described in Thielmann et al. (2023) with adaptations from Luber et al. (2021). The neural topic models implemented in OCTIS and thus also available in STREAM are the CTM introduced by Bianchi et al. (2021), the ETM (Dieng et al., 2020), NeuralLDA and ProdLDA introduced by Srivastava and Sutton (2017). Further models are LDA (Blei et al., 2003), HDP (Teh et al., 2004), LSI (Landauer et al., 1998) and classical NMF (Lee and Seung, 2000).
A.2 Available Datasets The available datasets are described in the paper in section 2.1. Since most of STREAMs models are centered around Document embeddings (Reimers and Gurevych, 2019), STREAM comes along with
a set of pre-embedded datasets. Once a user fits a model that leverages document embeddings, the embeddings are saved and automatically loaded the next time the user wants to fit any model with the same set of embeddings, thus enabling very fast model fitting and comparison.
A.3 Available Metrics In addition to the metrics from OCTIS, STREAM offers the following available topic evaluation metrics: ISIM, INT and ISH are all intruder based metrics proposed by Thielmann et al. (2024b). Embedding Coherence is similarly implemented as by Terragni et al. (2021b) without the normalization of the embeddings. NPMI describes classical NPMI scores proposed by Lau et al. (2014) and Embedding Coherence is similar to the Coherence metrics from Terragni et al. (2021b). Expressivity and Embedding Topic Diversity are both diversity metrics calculated in the embedding space. Future developments could include e.g. metrics proposed by Rahimi et al. (2024) or Weisser et al. (2023).
• Intruder Metrics
– ISIM: Average cosine similarity of top words of a topic to an intruder word.
– INT: For a given topic and a given intruder word, Intruder Accuracy is the fraction of top words to which the intruder has the least similar embedding among all top words.
– ISH: Calculates the shift in the centroid of a topic when an intruder word is replaced.
• Diversity Metrics
– Expressivity: Cosine Distance of topics to meaningless (stopword) embedding centroid.
– Embedding Topic Diversity: Topic diversity in the embedding space.
• Coherence Metrics
– Embedding Coherence: Cosine similarity between the centroid of the embeddings of the stopwords and the centroid of the topic.
– NPMI: Classical NPMi coherence computed on the source corpus.
A.4 Downstream task As a demonstration of the downstream task, we have simulated some simple data. We have created three data generating topics, consisting of fruits, vehicles and animals. The documents are generated by having a random draw with 60% out of one specified topic and the remaining 40% out random topics. Additionally, we have generated two continuous variables and made the target variable a function of two effects of the continuous variables as well as an effect of the number of words
taken from each generated topic. We subsequently fit a CDEC (Thielmann et al., 2024b) model and extracted the topics. The animal topic is depicted in Figure 4.
The downstream model is then simply specified as defined in equation 2. The continuous feature effects are accurately detected and visualized in figures 6 and 7. The topic effects, one continuous 5 and one more complicated 8 are also accurately depicted. It is clearly recognizable, that the animal topic from figure 4 has a continuous positive effect
on the target variable whereas the effect of the second topic is more refined and roughly follows a squared function.

heading: Future Work
text: Future adaptations could include the integration of further more performant or e. g. distributional downstream models (Chang et al. , 2022; Luber et al. , 2023; Thielmann et al. , 2024a) to further allow researchers to analyze the effect a topic has on a regression or classification task