Guiding Sentiment Analysis with Hierarchical Text Clustering: Analyzing the German X/Twitter Discourse on Face Masks in the 2020 COVID-19 Pandemic (2024)

Abstract: Social media are a critical component of the information ecosystem during public health crises. Understanding the public discourse is essential for effective communication and misinformation mitigation. Computational methods can aid these efforts through online social listening. We combined hierarchical text clustering and sentiment analysis to examine the face mask-wearing discourse in Germany during the COVID-19 pandemic using a dataset of 353,420 German X (formerly Twitter) posts from 2020. For sentiment analysis, we annotated a subsample of the data to train a neural network for classifying the sentiments of posts (neutral, negative, or positive). In combination with clustering, this approach uncovered sentiment patterns of different topics and their subtopics, reflecting the online public response to mask mandates in Germany. We show that our approach can be used to examine long-term narratives and sentiment dynamics and to identify specific topics that explain peaks of interest in the social media discourse.

heading: 1 Introduction
text: Social media platforms play an essential role in the information ecosystem during public health emergencies such as disease outbreaks, as they are widely used (We Are Social et al., 2024a) and catalyze the dissemination of information (Vraga et al., 2023). The public turns to these platforms to look for information, share and access news,
express opinions, and exchange personal experiences (We Are Social et al., 2024b). When there is an overabundance of information available during health emergencies, this is called an infodemic (Briand et al., 2023). Infodemics may include any information, accurate or false, i.e., misinformation, regardless of the intention (Lewandowsky et al., 2020). Understanding the information ecosystem of infodemics is crucial for developing effective data-driven and human-centered public health communication that addresses concerns and mitigates harmful effects from misinformation (Borges do Nascimento et al., 2022; Briand et al., 2023) and for infodemic preparedness (Wilhelm et al., 2023).
In the context of social media, natural language processing can help to monitor the public discourse (Baclic et al., 2020). This monitoring is commonly referred to as social listening (Stewart and Arnold, 2018), a key research field in infodemic management (Calleja et al., 2021). While it is often used in digital marketing, social listening is relatively new to the public health domain (Boender et al., 2023). In social listening, the classification of social media data into topics is used to identify different aspects of online conversations (topic analysis) and to measure temporal relevance over time (Purnat et al., 2021). To this end, the World Health Organization’s Early Artificial Intelligence–Supported Response With Social Listening Platform (EARS, White et al. (2023)) used semi-supervised machine learning for classifying social media content into
153
topics, which offered real-time analytics to public health researchers during the COVID-19 pandemic. Other works have used unsupervised methods, in particular, topic modeling (Blei, 2012), which represents topics as word distributions through generative probabilistic modeling (e.g., Rowe et al. (2021)), and text clustering (Willett, 1988), which represents topics as groups of semantically similar texts (e.g., Santoro et al. (2023)). In addition, sentiment analysis (Liu, 2012) can improve the understanding of the public perception of healthrelated topics by classifying sentiments expressed in texts (Boender et al., 2023; Briand et al., 2023).
While some studies have combined these techniques (e.g., Rowe et al. (2021)), they have typically used a flat representation of the data, i.e., a fixed number of topics in one level. In contrast, text data can be represented hierarchically on multiple levels, i.e., subgroups within one topic, with varying cluster sizes and granularities (Aggarwal and Zhai, 2012). The representation as a hierarchy allows the structured exploration of large document collections (Cutting et al., 1992) and helps to identify online narratives on social media in the context of public health (White et al., 2023).
In this work, we combine sentiment analysis with hierarchical text clustering to analyze a German X (formerly Twitter) dataset on wearing face masks during the COVID-19 pandemic in 2020. In Germany, the mask requirement was introduced at the end of April 2020 for public transport and stores (Die Bundesregierung, 2020b). The introduction of the obligation was preceded by a lockdown from mid-March with contact restrictions and the closure of numerous facilities in public spaces, e.g., schools as a consequence of an increase in COVID-19 cases (Die Bundesregierung, 2020c). The first easing of restrictions was implemented in mid-April (MDR, 2020). In order to extract sentiments from this much debated time period, we annotated a subsample of the selected dataset for sentiment analysis and trained a neural network for sentiment classification. We analyze the combined results in the context of the COVID-19 pandemic in Germany. Based on the overview of high-level coarse clusters and corresponding sentiments, we identify topics of interest for an in-depth analysis. We demonstrate the ability of our approach to systematically analyze highly debated public health measures such as face masks (Deutschlandfunk, 2020; MDR, 2020), which significantly impacted daily life in Germany.

heading: 2 Related Work
text: In the following, we discuss related work, focusing on machine learning techniques and applications relevant to X data and the German language.

heading: 2.1 Sentiment Analysis
text: Regarding German language sentiment analysis, machine learning methods typically outperform lexicon-based methods, and neural network models typically outperform traditional machine learning (Borst et al., 2023; Schmidt et al., 2022; Struß et al., 2019; Zielinski et al., 2023).
Guhr et al. (2020) fine-tuned a neural network for classification using a broad range of German sentiment datasets (GBERTbroad), including two datasets with X posts. GBERTbroad builds on GBERT (Chan et al., 2020), a BERT transformerbased encoder (Devlin et al., 2019) pretrained exclusively on German text. GBERT is also used successfully for fine-tuning sentiment classifiers on other task-annotated data (e.g., Schmidt et al. (2022); Zielinski et al. (2023)).
XLM-T (Barbieri et al., 2022) is a multilingual sentiment classifier for X posts trained on eight languages, including German. It is based on the multilingual XLM-RoBERTa (Conneau et al., 2020), which also uses the BERT architecture. Notably, it benefits from additional pretraining on posts prior to supervised fine-tuning, which may improve the performance on supervised classification tasks (Gururangan et al., 2020).

heading: 2.2 Text Clustering
text: Xu et al. (2015) suggest that embeddings, i.e., highdimensional vector representations derived through language modeling (Vinokourov et al., 2002), yield better results as inputs for text clustering than the traditionally used bag of words, i.e., numeric representations based on word occurrences (Aggarwal and Zhai, 2012).
Embedding-based text clustering is proposed as an alternative to topic modeling for identifying topics in text data (e.g., Angelov (2020); Sia et al. (2020)). Unlike topic modeling, text clustering does not assign descriptive keywords to topics. These need to be extracted separately using techniques like term frequency–inverse document frequency (TF-IDF). This statistical measure calculates the relevance of words in a text collection (Ramos, 2003). We use embedding-based text clustering since it can be advantageous for social media
data, as it may work better with short texts (Egger and Yu, 2022).
Creating hierarchies for text collections, as opposed to flat clustering, can help to explore and understand the contextual relationships (Cutting et al., 1992). Hierarchical clustering algorithms are often computationally expensive (Aggarwal and Zhai, 2012), limiting their use on large datasets. In this work, we use the Sub-Cluster Component Algorithm by Monath et al. (2021), who address this issue through various conceptional improvements compared to the traditional hierarchical agglomerative clustering without sacrificing clustering quality. Through the use of this algorithm, the clustering is based entirely on text embeddings. This is in contrast to the text clustering framework BERTopic (Grootendorst, 2022), which enables hierarchical text clustering, but combines the clustering of text embeddings and bag of words.

heading: 2.3 COVID-19-specific X Analysis
text: Various studies used sentiment analysis to analyze the online debate on X around COVID-19 during the pandemic in Germany. Reiter-Haas et al. (2023) analyzed the debate on contact tracing, vaccination, and face masks and contrasted the results with survey results. Schmidt et al. (2022) focused on the 2021 federal election in Germany. They analyzed the change in sentiments of the political parties’ posts in the election. Rowe et al. (2021) used topic modeling and sentiment analysis to analyze X data from Germany, other European countries, and the United States to understand the sentiment towards immigration during the early stage of the COVID19 pandemic in 2020. None of these studies used text clustering.
Santoro et al. (2023) used flat text clustering to analyze the different aspects of the online debate about vaccination in different countries over time. However, they did not consider sentiments.
In non-German analyses, Sanders et al. (2021) combined text clustering and sentiment analysis to study English face mask-related posts. They represented topics in a two-level hierarchy. Purnat et al. (2021) developed a more fine-grained hierarchy of five levels for classifying COVID-19 online conversations in English and French. However, posts were classified into topics with manually defined keywords. This taxonomy then served as the basis for the semi-supervised topic classification in EARS (White et al., 2023).
This work combines and expands on the ideas
of Sanders et al. (2021) and Purnat et al. (2021) and presents a social listening approach for public health that unifies topic and sentiment analysis. Our approach allows a flexible representation of the hierarchy with an adjustable number of levels. Additionally, our work contributes to the analysis of the social media discourse during the COVID-19 pandemic in Germany, surpassing the time period of data considered in previous work (Reiter-Haas et al., 2023).

heading: 3 Data
text: In this section, we describe the collection of X data and the dataset construction for sentiment analysis.

heading: 3.1 X (formerly Twitter) Data
text: Collecting German posts Between November 2022 and April 2023, we collected 50% of all original posts (i.e., excluding replies, comments, or quotes) in the German language for the year 2020 using the Academic Research API (X, 2023a). We used the post counts API (X, 2024) to estimate the number of original posts in 2020 per minute. For each minute (the smallest possible sampling time period for API queries), the API returned 50% of posts starting from the end of each minute. As true random sampling is not possible with the research API, we used the smallest possible time period, aiming to retrieve a representative sample of the entire stream that reflects its temporal characteristics, e.g., day/night shifts, and discussed topics for posts in the German language. We used the language tag provided by the API and refined the data through language identification with FastText (Joulin et al., 2017). We limited the posts to 2020 because X terminated our API access in April 2023. The final dataset contained 38 million posts.
Face mask dataset We filtered posts by words used to describe face masks: ‘Maske’ (‘mask’), ‘Mundschutz’ and ‘Mund-Nasen-Schutz’ (‘face mask’ or ‘surgical mask’), and ‘FFP2’, resulting in a face mask dataset with 353,420 posts.1 Using a sample of 1,000 included and excluded posts, we calculated a precision of 97.4% and a recall of 100% (cf. Limitations). We note that this dataset is not limited to posts originating from Germany but includes any posts in German. In line with
1For ‘Mund-Nasen-Schutz’, we included ‘Mundnasenschutz’ and ‘Mund-Nasenschutz’ as variations. We excluded posts containing ‘#maskedsinger’, ‘#themaskedsinger’, ‘maskedsinger’, or ‘masked singer’ (relating to a German TV show). We used lowercasing for filtering.
our Ethics Statement, we exclude any location data from our analysis.
For context, we included the 7-day incidence rate, i.e., the sum of COVID-19 cases in Germany with a reporting date within the last seven days, based on 100,000 inhabitants (Robert Koch-Institut, 2024).

heading: 3.2 Sentiment Analysis Data
text: Data selection We sampled 2,200 posts from the face mask dataset, using weights based on hourly relative frequency to maintain a similar temporal distribution. We adjusted the weights using the square root of relative frequencies to avoid oversampling periods with high post volume. This ensured that events related to the face mask requirement in Germany, which may have led to an increased post volume in the face mask dataset, were proportionally represented in the sample.
Annotation We asked seven annotators to label the general sentiments expressed in posts. Every post was labeled by three different annotators (at least 500 posts per annotator). Annotators were instructed to classify posts into four distinct categories: neutral, negative, positive, and mixed (containing negative and positive sentiments). We provided annotators with instructions and examples (not included in the final dataset). We used the majority rule to decide on the final label of posts (Table 1) and excluded samples without a majority from the final dataset. To measure the agreement between annotators, we report a Fleiss’ κ (Fleiss and Cohen, 1973) of 0.60, calculated using statsmodels (Seabold and Perktold, 2010). This score indicates moderate agreement (Landis and Koch, 1977) and is comparable to Schmidt et al. (2022), who annotated X posts by German politicians in a similar setting.
Data splitting Finally, we split the annotated dataset into training, validation, and test sets using a 7:1:2 stratified random split, i.e., maintaining
the original class distribution (Table 1) across splits. We only used posts with neutral, negative, or positive labels. We removed mixed posts to establish a stronger baseline for distinguishing between the primary sentiment classes.

heading: 4 Methodology
text: In the follwoing, we outline the training of the sentiment classifier, the application of hierarchical clustering, and the analytical approach.

heading: 4.1 Sentiment Analysis
text: For training the sentiment classifier, the base version of GBERT (Chan et al., 2020) serves as a starting point. GBERT has shown competitive results in German sentiment analysis (Schmidt et al., 2022; Zielinski et al., 2023). We consider two scenarios: First, using GBERT out-of-the-box for initializing a classifier. Second, we continue pretraining on the face mask dataset using whole word ‘masking’ similar to GBERT. This excludes the sentiment analysis data. For continued pretraining, we use the hyper-parameter setup for task-adaptive pretraining (TAPT) as suggested by Gururangan et al. (2020). For the supervised fine-tuning, we use the hyper-parameter setup suggested by Devlin et al. (2019). We base model selection on validation set performance, with an evaluation carried out every 10 steps. The training is performed on a single NVIDIA A100 GPU with the PyTorch (Ansel et al., 2024) and Transformers (Wolf et al., 2020) frameworks.

heading: 4.2 Hierarchical Clustering
text: The Sub-Cluster Component Algorithm (SCC, Monath et al. (2021)) is used for hierarchical text clustering thanks to its competitive performance on large datasets. SCC operates on nearest-neighbor similarity, repeatedly merging clusters to build a tree with multiple partition levels. The user controls these levels and the minimum similarity threshold for cluster merging. In this work, we represent posts as embeddings, using cosine similarity to define similarity. Cosine similarity is a standard metric to measure semantic similarity for vectorbased text representations (Chandrasekaran and Mago, 2021) and is used by Monath et al. (2021).2
In the first step, posts are embedded using German BERT large paraphrase cosine (May et al.,
2We provide code on GitHub: https://github. com/ClimSocAna/sentiments-with-hierarchicalclustering.
2023), a GBERT-large model fine-tuned for representing text similarity using cosine similarity. GBERT models show competitive results for German language text clustering (Wehrli et al., 2023), making German BERT large paraphrase cosine well-suited for the use with the SCC algorithm.
In the second step, the dataset is transformed into a nearest neighbors graph using Faiss (Douze et al., 2024). This graph is then used as an input to the SCC algorithm, with the state-of-the-art parameter setup (Monath et al., 2021), which includes 200 rounds of geometrically increasing thresholds and average linkage clustering. To improve computational efficiency, Monath et al. (2021) use a highly sparsified nearest neighbors graph (considering the 25 closest neighbors) to approximate cluster similarity. However, in this work, the number of neighbors considered is increased as much as possible (10,000 neighbors) to obtain the most accurate clustering possible. Given our computational resources, this results in RAM use of roughly 300 GB and a runtime of under 10 hours in a multi-CPU setup.

heading: 4.3 Analytical Approach
text: Topic size To measure a topic’s importance, we use the number of posts it contains to represent the range of discussed content. Depending on the research question, however, this metric could be adapted, for example, by including the popularity of posts to give greater weight to social resonance.
Sentiment score We use a sentiment score to analyze the sentiment of topics. This score is defined as
scoret,p = |postst,p,pos| − |postst,p,neg|
|postst,p,all| , (1)
where posts denotes the set of posts for a topic t, a period of time p, and for specific sentiments (neg for negative, pos for positive, all for all categories). The sentiment score calculates the average sentiment for a set of posts, considering positive and negative posts as polar values (+1, respectively, -1). We use this metric to highlight differences in topics based on their sentiment composition.
Cluster selection The SCC’s multi-level output allows us to analyze topics in varying detail. To demonstrate the flexibility of the hierarchical clustering, we select three levels of increasing topic granularity with 20, 104, and 1,051 clusters, respectively.
Cluster labeling and validation We first extract descriptive keywords for each cluster using classbased TF-IDF (Grootendorst, 2022). This is a variation of the traditional TF-IDF, which emphasizes the distinctiveness of keywords between clusters. We limit the set of candidate words to lemmatized content words to increase the information value of keywords, using tokenization (Proisl and Uhrig, 2016), part-of-speech tagging (Proisl, 2018) and lemmatization (Schmid, 1999). We select components optimized for German social media text based on Ortmann et al. (2019).3 We validate clustering quality through keyword analysis and random sampling of 200 posts per cluster to ensure the analyzed clusters represent distinct topics. The selection process could be supplemented by measures that quantify the quality of individual levels through intra- and inter-cluster (dis)similarity. We manually extract labels for each analyzed cluster based on the extracted keywords and the sampled posts.
Cluster visualization Hierarchical clustering organizes data into a tree-like structure called a dendrogram, where each node represents a cluster, and the branches show the relationships between them. The dendrogram’s structure is often used to visualize the hierarchical relationships between the clusters. It is important to note that the clusters are, fundamentally, sets of data points, not tree structures. However, visualizations based on dendrograms are limited by the quantification of the underlying clustering metrics and by their lack of flexibility and customizability.
Given these limitations, we use a treemap idiom to provide a compact and intuitive way (Hattab et al., 2020) to navigate and explore the resulting hierarchies. The treemaps visualize the hierarchical relationships between clusters by making the size of each node proportional to the relative importance or size of the cluster. The hierarchical structure is conveyed by the nested layout of the treemap, where child nodes are contained within their parent nodes. This allows the visualization of complex topic hierarchies and identifies dominant topics and their relationships.
Two use cases are considered to illustrate the treemap idiom. They address the relationships of subclusters to their parent cluster and the temporal
3We provide a spaCy-based (Honnibal et al., 2020) implementation on GitHub: https://github.com/slvnwhrl/ GerSoMeTokenExtractor.
changes of the relative importance of one subcluster. This relies on the overview detail and the small multiple idioms (Shneiderman, 2003) and corresponds to Figures 2 and 3, respectively.

heading: 5 Results & Analysis
text: In this section, we evaluate the results of the sentiment classification training, followed by the analysis of the face mask dataset.

heading: 5.1 Evaluation of the Sentiment Classification
text: Results Table 2 reports the overall results for the classification of neutral, negative, and positive sentiments of posts on the test set for the fine-tuned GBERT models (as outlined in Subsection 4.1). Additionally, we evaluated GBERTbroad and XLM-T as baselines. GBERTSFT, fine-tuned on the annotated face mask posts, achieved an average weighted F1-score of 77.90%. GBERTbroad fell significantly behind with a more than 20 percentage points lower F1weighted. The performance gap of XLM-T is much smaller at less than four percentage points. GBERTTAPT+SFT delivered the best results, achieving an 3.06 percentage points higher average weighted F1-score than GBERTSFT through additional pretraining. Based on these results, we used the best-performing GBERTTAPT+SFT to analyze the face mask dataset.4
Error analysis The most common errors of GBERTTAPT+SFT were neutral posts misclassified as negative, vice versa, and positive posts misclassified as neutral (cf. Table 3, Appendix A). Predictions for the neutral and negative classes showed relatively balanced precision and recall compared to the positive class (cf. Table 4, Appendix A). The
4We released the best-performing model on Hugging Face: https://huggingface.co/slvnwhrl/gbert-facemask-sentiment.
lower overall F1-score (68.97%) and comparatively lower recall (62.50%) of the positive class is likely, to some degree, a result of the class imbalance (Table 1, Johnson and Khoshgoftaar (2019)).
The inspection of misclassified samples showed that the model sometimes struggled with implicit sentiment and sarcasm, likely contributing to the lower performance of the positive class (Riloff et al., 2013).

heading: 5.2 Results and Discussion of the Face Mask Dataset
text: The face mask dataset represents the dynamic social media discourse about face masks during the COVID-19 pandemic (Figure 1). The rise in notified COVID-19 cases, i.e., the 7-day incidence rate, and introductions of public health interventions were often associated with increased online conversation on this topic. Some events stand out, especially the initial introduction of the mask requirement in April 2020 (Figure 1). The falling number of cases in the summer of 2020 fueled the debate about the necessity of wearing face masks, for example, at the beginning of July. With the ris-
ing case numbers in autumn, new public health and social measures were implemented, which included the restriction of social contacts and the temporary closure of restaurants or cultural institutions, but less restrictive than during the first lockdown in March 2020 (Die Bundesregierung, 2020a,c). Finally, the federal government imposed a second lockdown before Christmas (Die Bundesregierung,
2020d). These dates showed a lower number of posts than during the first introduction of mandatory face mask wearing.
Topics overview The dynamic nature of the debate can also be seen in the overarching topics of the discourse on X, resulting from the hierarchical clustering (Figure 2). We provide an interactive
visualization on GitHub.5 The most prominently discussed topics are areas of life that were often the target of public health measures, namely stores ( 4⃝), public transport ( 8⃝), or schools ( 9⃝). When reviewing samples, we observed that many posts contain descriptions of everyday life, such as other people’s behavior when shopping, e.g., when neglecting the face mask mandate. The data show that a large part of posts are about sharing experiences ( 1⃝) and include a large variety of topics.
Sharing news also plays an important role in the COVID-19 discourse on face masks; on the one hand, news specific to the introduction of the mask requirement ( 2⃝), but also more general news on the subject of masks, such as reports on the number of COVID-19 cases ( 6⃝). It is noticeable that posts with shared news (e.g., with a link) often also contain the user’s opinion about the content. We identified similar tendencies for topics that mainly revolve around the benefits of masks ( 5⃝) and the political and social debate surrounding the obligation to wear masks ( 3⃝).
Finally, some topics deal primarily with the production of do-it-yourself (DIY) masks (10⃝) and the procurement of masks ( 7⃝).
Overall, posts expressing negative sentiment dominated the discourse (Figure 2, left). In fact, more negative than positive posts were published at all times during the investigated year (cf. Figure 4, Appendix A). Some topics are particularly negative ( 3⃝, 4⃝, 8⃝, 9⃝), with the topic on DIY masks (10⃝) standing out as the only topic with a slightly positive sentiment. During the analysis, we often noticed posts in which users shared their DIY masks. In Figure 2, we show the subtopics, i.e., deeper level, of the topic of politics and demonstrations ( 3⃝), to present a more differentiated view of this particularly negative topic. Within this topic, users discuss German or international politicians (e.g., Angela Merkel ( 3⃝(d)) or Donald Trump ( 3⃝(g))), Germany in international comparison ( 3⃝(h)), or topics related explicitly to Berlin ( 3⃝(f)) or the climate ( 3⃝(i)). This view reveals differences in the relative importance and sentiment of subtopics that make up the overall negative sentiment of the higher-level topic and shows that mask wearing is discussed in very different political contexts.
Demonstrations The topic of demonstrations in German cities is particularly striking due to its size
5https://github.com/ClimSocAna/sentimentswith-hierarchical-clustering.
and negativity (Figure 2, right). Thus, we chose this topic for a more detailed analysis, exemplifying the capabilities of hierarchical text clustering to guide sentiment analysis. In Germany, the first demonstrations against public measures took place at the beginning of May, with larger demonstrations occurring repeatedly throughout the year, for example, in Berlin and Leipzig (MDR, 2020). Figure 3 shows the size and sentiment of the subtopic cluster in the parent topic of politics and demonstrations (top) and the number of posts on demonstrations in German cities (bottom) in the three quadrimesters of 2020. We found that demonstrations are more frequently discussed over the course of 2020 and contribute more to the parent topic without significant changes in negativity (Figure 3, top). Pointwise increases in posts occurred at the time of specific demonstrations (Figure 3, bottom), e.g., two large demonstrations with tens of thousands of participants in Berlin at the beginning and end of August 2020 (MDR, 2020).
Comparing results Reiter-Haas et al. (2023) also investigated the sentiments of face maskrelated German posts from X. They did not find a clear tendency towards positive or negative sentiments. Compared to our study, they used a smaller X sample for sentiment analysis (15,425 versus 353,420 posts), only considered data from January to August 2020 (as opposed to data from the whole year), and used a different method for sentiment analysis (based on a sentiment lexicon). These factors may explain the different outcome of our study, as we find that the majority of posts are negative.
Sanders et al. (2021) used two-level hierarchical text clustering to analyze English X posts on face masks from March to July 2020. They identified topic clusters of varying sentiments, similar to our findings. However, posts with negative sentiments did not dominate the overall discourse. Additionally, topical parallels can be drawn between the discourse on political actors (such as Donald Trump) or on public measures such as the requirement to wear face masks in stores, and the sharing of personal experiences from everyday life. Similar to Reiter-Haas et al. (2023), they used lexiconbased sentiment analysis, albeit on a larger sample (1,013,039 posts).

heading: 6 Conclusion and Outlook
text: This study employed hierarchical text clustering and sentiment analysis to examine the public dis-
course surrounding face masks in Germany during the COVID-19 pandemic 2020. Our analysis of 353,420 face mask-related posts reveals the dynamics of the German public’s online response on X (formerly Twitter) to mask mandates. Our findings indicate an overall negative sentiment dominating face mask-related posts. Analyzing specific topics revealed nuanced sentiment patterns. For instance, the topic of DIY masks was slightly positive, while topics linked to COVID-19 demonstrations and general political discourse on face mask policies showed stronger negativity. We show that the combination of clustering, sentiment classification, and suitable visualization helps analyze complex social media discourse in a structured manner. This study thereby advances the methods of social listening in public health. Furthermore, our analysis contributes to the understanding of the online information ecosystem during the COVID-19 pandemic in Germany. This is a prerequisite to better understand the (harmful) impact of the infodemic during the COVID-19 pandemic on the public. Ultimately, this enables knowledge-based preparation for a future pandemic that will likely be accompanied by an infodemic again (Briand et al., 2023; Wilhelm et al., 2023). In this context, the proposed approach enables structured analyses of social media data for topics that are relevant for the review of the COVID-19 pandemic, respectively, infodemic (such as mental health).
Our results offer a starting point for further research. The presented approach could be adapted for real-time infodemic surveillance (‘infoveillance’) to track sentiment dynamics and emerging topics during future health-related social media discourses, e.g., based on the online version of the SCC algorithm (Monath et al., 2023). To operationalize this approach during the next pandemic or public health crisis, developing a framework for interactive data exploration is required to generate insights that can inform public health action. Finally, specific investigations into the role of misinformation within negative clusters are needed to illuminate public health communication challenges.
Limitations
Keyword filtering We noticed that a few posts were filtered incorrectly as relevant because ‘Maske’ (‘mask’) was used as a homonym for a beauty product or as part of a costume. Further-
more, other terms may be relevant to the online discussion about face masks, even if our analysis showed a high recall. For example, there are colloquial or dialect words to consider, such as ‘Schnutenpulli’ (NORD24, 2020), originally from Low German. Finally, we did not consider misspellings.
Clustering While the chosen clustering algorithm and language model have proven effective, a key limitation is their singularity. Exploring multiple clustering algorithms and language models could reveal different data structures and provide more nuanced insights. Comparing the results from diverse methods would help assess the robustness of the observed clusters and sentiment patterns.
Social media data Finally, we note that accessing data from social media platforms remains challenging, which currently limits the application of our approach to X data. The European Union’s Digital Services Act is likely to improve this situation for infodemic research and practice in the future, as it will legally enable researchers to access data on large platforms (Wehrli et al., 2024).
Ethics Statement
The X (formerly Twitter) data collected as part of this work is subject to X’s Developer Agreement and Policy (X, 2023b) and the European Union’s General Data Protection Regulation (GDPR, European Commission), which we comply with. We only process post texts and timestamps, remove user mentions and URLs from the post texts, and do not use any post metadata that allows the identification of individuals (such as user names or location data). In addition, we only present results at an aggregated level, i.e., for groups of posts. We do not publish or share any of the data or results in a way that does not align with X’s Developer Agreement and Policy and the GDPR.

heading: Acknowledgments
text: We would like to thank the annotators of the sentiment analysis dataset for their help in realizing this work. We would also like to thank the reviewers for their time and valuable input.

heading: A Additional Classification Results of the Sentiment Classification

heading: Future Work
text: Ultimately, this enables knowledge-based preparation for a future pandemic that will likely be accompanied by an infodemic again (Briand et al. , 2023; Wilhelm et al. , 2023). In this context, the proposed approach enables structured analyses of social media data for topics that are relevant for the review of the COVID-19 pandemic, respectively, infodemic (such as mental health). Our results offer a starting point for further research. The presented approach could be adapted for real-time infodemic surveillance (‘infoveillance’) to track sentiment dynamics and emerging topics during future health-related social media discourses, e. g. , based on the online version of the SCC algorithm (Monath et al. , 2023). To operationalize this approach during the next pandemic or public health crisis, developing a framework for interactive data exploration is required to generate insights that can inform public health action. Finally, specific investigations into the role of misinformation within negative clusters are needed to illuminate public health communication challenges. Limitations
Keyword filtering We noticed that a few posts were filtered incorrectly as relevant because ‘Maske’ (‘mask’) was used as a homonym for a beauty product or as part of a costume