The PGNSC Benchmark: How Do We Predict Where Information Spreads? (2024)

Abstract: Social networks have become ideal vehicles for news dissemination because posted content is easily able to reach users beyond a news outlet’s direct audience. Understanding how information is transmitted among communities of users is a critical step towards understanding the impact social networks have on realworld events. Two significant barriers in this vein of work are identifying user clusters and meaningfully characterizing these communities. Thus, we propose the PGNSC benchmark, which builds information pathways based on the audiences of influential news sources and uses their content to characterize the communities. We present methods of aggregating these news-source-centric communities and for constructing the community feature representations that are used sequentially to construct information pathway prediction pipelines. Lastly, we perform extensive experiments to demonstrate the performance of baseline pipeline constructions and to highlight the possibilities for future work. Our code and data can be found here: https://github.com/ataylor24/PGNSC.

heading: 1 Introduction
text: Social media platforms have become a crucial part of the information dissemination ecosystem. By allowing users to choose whom they share their socalled “content feeds" with, these platforms have created an environment in which the reach of information is amplified.
Pew Research (Forman-Katz, 2022) reported that roughly half of American adults regularly consume news through social media and that 13% prefer to get their news through social media, which increases to 33% for adults under 30. Further surveying suggests that adults under 30 place as much trust in news gathered from social media as from
†Corresponding author: Alexander K. Taylor {ataylor2@cs.ucla.edu}
traditional news outlets (Liedke, 2022). Because of this, traditionally "offline" news sources now have dedicated social media accounts that seek to propagate their content to the growing demographic of adults using social media for news consumption.
Given the relevance of this problem, we seek to establish a benchmark that will establish a foundation which future work may use to develop methods to identify the audiences of influential news sources and predict the flow of information. Because social media users rarely explicitly disclose which news sources they get information from, we must identify the audience, or community, of each news source we consider. We will focus on two benefits of aggregating users into communities (Zhang et al., 2019a; Lancichinetti et al., 2008).
The first benefit of community aggregation is that it allows us to leverage the structure present in the data to predict the behavior of a set of closely connected users (Lancichinetti et al., 2008). Using this principle, we are able to circumvent the lowresource setting of user-level prediction caused by inconsistent user posting schedules by predicting the behavior of communities of users as demonstrated in prior work (Li et al., 2022; Taylor et al., 2023). The second benefit is that we can leverage inductive bias to improve the representation of the community; for instance, members of communities centered around a given news outlet will likely share that outlet’s political leaning (Liedke, 2022) and are likely to have their views influenced by that outlet’s content. This motivates the construction of community representations that incorporate as much additional information as possible (to inform community representation).
We seek to take the first step in establishing a benchmark for information pathway prediction at the community level, which we title Prominent Global News Sources for Covid-19, hereafter referred to as PGNSC. PGNSC is a novel, humanvalidated dataset built using the most influential
15787
global news sources for COVID-19 (PGNSC). The dataset consists of instances of news articles being posted to social media and the resulting interactions between news organization communities. We also provide a general sequential framework for building pipelines to perform information pathway prediction and define baseline methods for each pipeline stage.
For the community aggregation stage, we include several methods of aggregating communities based on prior work (Taylor et al., 2023; Komorowski et al., 2018; Romero et al., 2010) and show their impact on information pathway prediction performance. To construct community feature representations, we seek to leverage the recent advances in large language models (LLMs) and their use in enhancing graph representations by using LLMs to summarize and encode of each organization’s content (He et al., 2023; Chen et al., 2023b). Because news content often includes images, we also incorporate a jointly-trained image-text encoder into the set of community node feature generation pipelines (Radford et al., 2021).
The appeal of PGNSC goes beyond providing data that can be used for analysis of patterns of information propagation as well as graph representation tasks. To the best of our knowledge, this is the first work to establish benchmark for information pathway prediction using heterogeneous graphs and and to use SOTA LLMs to enhance node feature representations to improve predicitons. We believe PGNSC is well positioned to serve as a vehicle for exploring how LLMs and graph data can be used in concert to make predictions.

heading: 2 Dynamic Graph Benchmarks
text: The development of graph representations has recently undergone a renaissance with the development and application of graph neural networks that benefit from data richness and complexity (Huang et al., 2023; noa; Gravina and Bacciu, 2023). This evolution has underscored the critical need for robust benchmarks in both static and dynamic graph domains, divided into real-world and synthetic datasets. Our focus herein is on dynamic graphs, which are pivotal for modeling time-evolving relationships in numerous applications.

heading: 2.1 Real-world Dynamic Graph Benchmarks
text: High-quality, real-world datasets are considered the gold standard for benchmarking because they
closely simulate the application of models in practical scenarios. However, such data can be costly to munge and often presents issues related to missing values or other quality control measures. There are many available datasets built from real-world data and described in prior works as shown in 1 (please see Appendix 8 for a full list) (Poursafaei et al.; Huang et al., 2023; Gravina and Bacciu, 2023; Horawalavithana et al., 2022). While these benchmarks encompass a broad range of spatial attributes and temporal granularity, they are overwhelmingly skewed towards homogeneous graphs with some exceptions (Poursafaei et al.; Huang et al., 2023; Gravina and Bacciu, 2023). These works, however, have not fully exploited the potential of dynamic, heterogeneous datasets nor embraced the advancements in large language models (LLMs) for multimodal representation. Our benchmark aims to fill this gap by integrating LLM-enhanced multimodal data, setting a new stage for information pathway prediction research.

heading: 2.2 Synthetic Dynamic Graph Benchmarks
text: To bypass the limitations of real-world data, some work has investigated synthetic datasets designed to replicate real-world data distributions (Greene et al., 2010; Ammar and Özsu, 2014; Lancichinetti et al., 2008; Rossetti, 2017). Synthetic benchmarks, while valuable for their controllability and reproducibility, often lack the unpredictability and intricate variability found in natural datasets, especially in domains like information propagation. They may not fully capture the complexities and nuances of real-world scenarios, and thus cannot provide the same value as empirical data. Our benchmark contributes a significant, large-scale real-world dataset, aiming to provide a comprehensive tool for developing more generalizable and robust GNNs.

heading: 3 PGNSC Benchmark
text: In this section, we discuss the construction of the PGNSC benchmark. To our knowledge, this is the first publicly available benchmark in which multimodal information cascades (structure, text, and image data) have been aggregated into communitybased information pathways.

heading: 3.1 News Data Collection
text: Prior to selecting the news organizations whose pathways are included in the PGNSC dataset, we identified countries from which to choose news or-
ganizations. We identified the 15 countries with the highest aggregate number of COVID-19 cases from May 15, 2020 to April 11, 2021. We then used the Digital News Report from the Reuters Institute as our primary source in determining the most prominent online news organizations for each country, and we supplemented with additional sources when the report did not cover a given country or required further justification. A full list of the countries and news sources identified for each country is available in the appendix. We then used the aforementioned URLs to query Google News for the data associated with the article link using the open-source package GNews. The data retrieved contained the article title, article text content, and the URLs for images embedded in the article, as shown in Table 4.

heading: 3.2 Twitter Data Collection
text: We utilized the COVID-19 Twitter API to retrieve data from May 15, 2020 to April 11, 2021
(331 days) using COVID-19 keywords. From this window, we identified roughly 53 million tweets containing article URLs (hereafter referred to as source tweets) from the selected news sources (see Appendix for data breakdown). We performed weighted random sampling based on the number of article links present in each day, which yielded roughly 304k source tweets. We then scraped all retweets and reply tweets (hereafter referred to as response tweets) for each source tweet as well as the metadata and content of each tweet (both source and reply)0. Each source tweet and the subsequent response tweets were used to construct a User-Level Information Cascade.

heading: 3.3 Information Pathway Mapping
text: We define communities as sets of users aggregated around the news sources defined in PGNSC (Taylor et al., 2023). Community aggregations were performed using the engagement metrics and user interactions drawn from the user-level information pathways according to methods described in detail in a later section. Each user-level information pathway was mapped to a communitylevel information pathway by replacing each user with the community it was assigned to (no selfedges) as shown in Figure 2.
We define a set of node and edge types that are used to construct the graph representation in Table 5, following the format of prior work (Taylor
0We want to note that while engagement metrics (number of favorites, retweets, and replies) were present for all source tweets, each value was defaulted to 0 for many response tweets.
et al., 2023). We generalize the naming conventions for connections at the community level beyond Twitter for ease of future application to other social media platforms.

heading: 3.4 Community Aggregation Methods
text: In this section, we describe the aggregation methods that we used to assign users to communities centered around the aforementioned news sources. While there are many methods for unstructured community identification, we found that the majority of methods do not scale to the size of our graph for the user-level network we have constructed (Zhang et al., 2019a; Zeng and Yu, 2018; Sattar and Arifuzzaman, 2022). We have selected three heuristic-based methods. Each community aggregation method was performed using the same set of community centers and applied to the same set of user-level information cascades. The general format for the aggregation methods presented in this work is generating community scores s for each
user, and then assigning the user to the community for which it has the highest score, as shown in Figure 3.

heading: 3.4.1 Engagement
text: Following prior work on information pathway prediction (Taylor et al., 2023), we compute the Engagement score for each user with respect to each community center. This community aggregation method is designed to assign users to the commu-
nity for which their posts have the most engagement and uses collected engagement metrics: the number of favorites, retweets, replies, user IDs mentioned in the tweet, and the most recent Following/Follower ratio. Users are assigned to only the community for which they have the highest Engagement score with ties broken randomly.

heading: 3.4.2 Interaction
text: Following prior work in community aggregation in Twitter data (Komorowski et al., 2018), we compute the Interaction score for each user with respect to each community center (Our formulation is modified to include the number of replies). This community aggregation method is designed to assign each user to the community for which it has the highest relative importance, as determined by the impact of its interactions with other users. We first construct super-graphs of the User-Level Information Pathway Instances associated with each community center. The Interaction score is used as the edge weight for an interaction from user p and user q.
The PageRank algorithm (Page et al., 1999) is then applied to each community center’s supergraph and produces a ranking of the relative importance of each user with respect to that community center. u represents the user whose rank we wish to calculate and Eu represents the set of users that have interactions with u. d is a damping factor, and L(p) is the number of links from page p. Users are assigned to only the community for which they have the highest Interaction score, with ties broken randomly.

heading: 3.4.3 Influence-Passivity
text: The last community aggregation method that we consider follows prior work that measured the relative influence of nodes in social networks (Romero et al., 2010). This community aggregation method is designed to measure the influence that a given node exerts on its neighborhood (influence) and vice versa (passivity). For each user in a given community, we retrieve the number of URLs they posted and their one-hop neighborhood of in and outgoing interactions, which results a directed super-graph for each community. This information allows us to perform the InfluencePassivity algorithm (Romero et al., 2010) which yields a 2-dimensional influence-passivity vector, [influence_score, passivity_score]. We assign each user to the community for which they have the
highest l2-norm of the influence-passivity vector.

heading: 3.5 Random
text: To evaluate the effectiveness of the community aggregation methods, we include two random settings, biased-random and unbiased-random.
Biased-random. The biased-random community selection method samples from the communities whose articles a given user has interacted with. We include this setting to compare randomly selecting from a user’s engagement history with using the bias described by the methods described in previous sections.
Unbiased-random. The unbiased-random community selection method randomly assigns each user to one of any community. This is the traditional random baseline and is included to evaluate the information compression properties of community assignment.

heading: 3.6 Comparison to Similar Benchmarks
text: We include a selected list of several dynamic heterogeneous graph benchmarks in Table 1. FBForum, FB-Covid19, Reddit, Reddit Hyperlink Network, Twitter-Tennis and tgbn-reddit, while rich structurally, do not incorporate node attributes beyond structural embeddings, and only FB-Covid19 and Reddit incorporate edge attributes in the form of LIWC encodings (Opsahl, 2011; Kumar et al., 2018; Panagopoulos et al., 2021; Béres et al., 2018; Huang et al., 2023). As mentioned in 1, to our knowledge, Expert is not publicly available for comparison. PGNSC is a publicly available, heterogeneous graph benchmark that incorporates LLM-enhanced multimodal graph attributes.

heading: 4 Feature Initialization Pipelines
text: In this section, we describe the pipelines used to generate node feature embeddings for each community. We explore the efficacy of three approaches to pipeline construction: single-stage, two-stage, and zero-shot.

heading: 4.1 Single-Stage Pipeline (Lightning)
text: The single-stage pipeline formulation (Lightning) illustrated in (Taylor et al., 2023) applies an encoder-only model to text that represents each community center. Lightning applies the encoderonly model, Longformer, to the concatenation of text from sampled articles to construct a representation of each community center (Beltagy et al., 2020; Taylor et al., 2023). The text we use consists
of the concatenations of article titles and article contents (illustrated in Figure 4). The embeddings generated through this process are used directly as the node feature representation of the community center to which they pertain.

heading: 4.2 Two-Stage Pipelines
text: Recent works have made strides in multi-document Summarization (MDS) (Xiao et al., 2022; Zhang et al., 2019b), and we seek to incorporate this into our baseline pipeline formulation. We chose to incorporate the state-of-the-art multi-document summary model, PRIMERA (Xiao et al., 2022), into the node feature embeddings generation process. This is accomplished by randomly sampling k articles from the aggregated articles available for each community to use as input to PRIMERA, which produces a text-based summary. We treat this summary as an encapsulation of the community’s content and encode it using BERT to serve as the community’s node features (Devlin et al., 2019; Taylor et al., 2023). We title this node feature generation pipeline PRIMERA-BERT. We also seek to explore the impacts of incorporating multi-modal data into the information pathway prediction task. Thus, we also include a pipeline that encodes the PRIMERA-generated text summary with the CLIP model (Radford et al., 2021), which is a joint textimage encoder. The process is similar to that of the PRIMERA-BERT pipeline: we randomly sample k articles and their accompanying images and simultaneously encode the PRIMERA-generated summary and relevant images using CLIP. We title this
node feature generation pipeline PRIMERA-CLIP. We include samples of the generated summaries in the Appendix.

heading: 4.3 Zero-Shot Pipelines
text: Modern LLMs have made dramatic progress in the recent past and are often able to outperform finetuned models in low-resource settings. Thus, we also test pipelines in which we leverage the vast context of LLMs to generate zero-shot text summaries of each community using manual prompttuning to achieve high-quality results. The generated text summaries are encoded using both BERT and CLIP, similar to the pipelines described above. We use two of the standard LLMs, Llama-2 and GPT-4, to perform these tasks and follow the naming convention established above for their respective pipelines.

heading: 5 Experimental Results
text: In this section, we discuss the logistical settings of our experiments and discuss the performance of our baseline models on PGNSC.

heading: 5.1 Experimental Setup
text: Data splits. We divide the data according to an 80:10:10 split for the training, validation, and testing sets, respectively. Each experimental number shown is the averaged result of 10 experiments.
Evaluation metrics. We use the GMAUC metric (geometric mean of AUC and AUPR) as the primary evaluation metric in our work to more aptly capture performance on both positive and negative edge predictions, as shown in prior work (Poursafaei et al.). We also follow the conventions set by prior link prediction tasks and include AUPR and AUC as our evaluation metric (You et al., 2020; Kumar et al., 2020; Wu et al., 2022), which are available in the Appendix.
Experiments. We perform a grid search across the performances of five community aggregation methods and seven node embedding methods to investigate the impact of our proposed node representation methods and its interaction with community formulation. The five community aggregation methods are Biased-Random assignment, Unbiased-Random assignment, Engagement (Taylor et al., 2023), Interaction (Komorowski et al., 2018), and Influence-Passivity (Romero
et al., 2010). The seven node feature representation pipelines are Longformer, PRIMERABERT, GPT4-BERT, Llama2-BERT, PRIMERACLIP, GPT4-CLIP, and Llama2-CLIP.

heading: 5.2 Link Prediction Model
text: For the dynamic link prediction module, we use HTGNN (Heterogeneous Temporal Graph Neural Network) (Fan et al., 2021) as it is a state-of-the-art model for dynamic link prediction in a heterogeneous setting. HTGNN consists of multiple heterogeneous temporal aggregation layers, each employing hierarchical aggregation mechanisms that are combined to yield the spatio-temporal embedding for each node: intra-relation aggregation, interrelation aggregation, and across-time aggregation. Intra-relation Aggregation For each node v with type ϕ(v) at timestamp t, the feature vector xtv is projected into a unified feature space:
ht,lv,r =
intra⊕
u∈N tr (v)
( ht,l−1u ;Θintra ) , (1)
ht,lv,R = inter⊕
r∈R(v)
( ht,lv,r;Θinter ) , (2)
ht,lv,ST =
across⊕
1≤t′≤T
( ht
′,l v,R;Θacross
) . (3)
where ⊕
denotes the aggregation function, and the terms are defined as follows: N tr (v) represents the set of neighbors of node v at timestamp t for relation type r, R(v) denotes the set of relation types connected to node v, T is the total number of timestamps considered in the model. The final embedding for each node is the sum of its embeddings across all timestamps:
hv =
T∑
t=1
ht,Lv .
Finally, HTGNN is trained with cross entropy loss: L = ∑
v∈VL J(yv, ŷv)+λ∥Θ∥22, ŷv = σ(MLP(hv)),

heading: 5.3 Overall Information Pathways Prediction Results
text: In this section, we will discuss the observations made and conclusions drawn from the experimental results shown in Figure 5.
Community Aggregation Method Comparison results. We observe that feature initialization pipelines (FI) show improvements of 6.55 to 10.98 on pathways mapped to communities constructed using the Influence-Passivity (IP) heuristic. We believe that IP benefits from factoring in both the influence exerted on outgoing edges and receptivity of users to incoming edges as opposed to Interaction and Engagement scores, which simply compare the relative influence exertion of users.
We also find that the Engagement community aggregation methods yields similar results to Biased Random community assignment, which indicates that heuristics relying on user-level network structure (Interaction and IP) are able to more meaningfully characterize communities than can engagement metrics. These results support the assertion that relying on the frequency of interactions provides more insight into user behavior than aggregated user engagement metrics (Romero et al., 2010).
Feature Initialization Method Comparison results. We observe that FI pipelines encoded by BERT offer an average 1.1 point improvement over those using CLIP across all community aggregation methods; this further increases to 1.6 point average improvement when considering the non-random aggregation heuristics. The two-stage FI pipelines provide an average improvement of 5.4 points over one-stage FI pipelines across all community heuristics, which increases to 6.2 points when excluding randomized community heuristics. We believe the reason behind the discrepancy between the effectiveness of BERT encodings and multimodal CLIP encodings is two-fold. First, CLIP embeddings are smaller due to model output parameters. Second, a manual investigation yields that, while images enhance the representation of articles they accompany, they may not aid in differentiating one community from another (See Appendix for example).
We also observe a slight trend in Llama-2 summary modules outperforming those using Primera and GPT4-based summary modules, and Llama2 has an average 1.2 point improvement over PRIMERA under non-random community aggregation heuristics. However, this difference shrinks to 0.8 points when random settings are included, and the average relative differences in performance between Llama-2 and GPT4 and between GPT4 and PRIMERA are similarly insignificant.

heading: 6 Related Works

heading: 6.1 Community-based Information Propagation Prediction on Social Networks
text: There are many existing methods designed to improve the performance of link prediction over social media network representations (Daud et al., 2020; Kumar et al., 2020; Wu et al., 2022; Haghani and Keyvanpour, 2019; Toprak et al., 2023). Several recent works have framed the link prediction problem as predictions of information propagation (Taylor et al., 2023; Jin et al., 2023). These works both define information pathways as the trajectory of individual pieces of information through communities of users.
Prior work made use of Reddit data, equating user participation in Subreddits with community membership and predicting information propagation (Jin et al., 2023). Many social networks that generate information cascades do not have explicit communities, thus Taylor et al. developed the Engagement heuristic to identify implicit communities using engagement metrics (Favorites, Retweets, etc.) on Twitter COVID-19 data (Taylor et al.,
2023). Using this heuristic, Taylor et al. showed that using collections of articles to construct the feature representations of these communities yielded better static link prediction results over using tweet histories to represent individual users.
Our work continues in this vein and uses four distinct community construction methodologies (including the Engagement heuristic) and novel LLMenhanced node feature generation methods for community feature representations (Taylor et al., 2023).

heading: 6.2 Multi-Modal Knowledge Graph Representations
text: Multi-modal graph representations have recently received increased attention because they more fully represent problems and lead to improved performance on downstream tasks (Peng et al., 2023; Liang et al., 2021; Zheng et al., 2022; Wei et al., 2019; Ektefaie et al., 2023). These improvements have been observed in diverse applications, from misinformation detection to transport systems (Abdali, 2022; Zhang et al., 2023; Tian et al., 2022; Rahmani et al., 2023; Jin et al., 2023; Chen et al., 2023a).
The recent revolution of LLMs has inspired re-
cent work to investigate the potential benefits of incorporating LLMs into graph neural networks (He et al., 2023; Chen et al., 2023b). Further works has shown that it is possible to use zero-shot LLM summarization in concert with graph embeddings to predict the news consumption habits of individual users (Chen et al., 2023a; Liu et al., 2023). These successes inspired prior work to use a longform text encoder to construct the node feature representations of communities based on collections of articles to predict information propagation (Taylor et al., 2023; Beltagy et al., 2020).
Our work furthers this direction by using SOTA LLMs to generate and encode community embeddings of a highly-curated, novel dataset. We demonstrate that downstream link-prediction performance benefits significantly from LLM-enhanced community summarization and outperforms prior approaches on our dataset.

heading: 7 Conclusion
text: In this work, we present a novel benchmark for dynamic link prediction on heterogeneous graphs in the novel application domain of information pathway prediction. We consider 5 community aggregation heuristics and 7 different feature initialization pipeline constructions, all of which show the improvement that LLM-enhanced feature representations can have on downstram link prediction. We also show that the community construction method significantly impacts overall performance, and our results support prior assertions of the benefits of considering interaction network structure (Romero et al., 2010).
It is our hope that PGNSC will make LLMenhanced graph tasks more accessible, fill the necessity for a large dynamic heterogeneous graph dataset, and last, but not least, offer a challenging setting for incorporating community aggregation into information propagation prediction tasks.
Limitations
This work presents a novel benchmark, 5 community aggregation heuristics, and 7 feature initialization pipelines. While we mentioned scalability issues of existing methodologies, we acknowledge the simplicity of the community aggregation heuristics included in this work. We also acknowledge the limitations of our data: as mentioned in the main paper, many response tweets are missing engagement metrics, meaning that the Engagement and
Interactions scores are being computed over incomplete data. Lastly, we have included the sources used to determine prominent global news sources, but acknowledge the bias inherent in the process of deciding whether to include news sources as communities or when excluding countries from data collection.
Ethics Statement
Our benchmark is intended to encourage exploration in the information propagation domain while avoiding any leakage of private information. The data we use in this work has been fully de-identified and only uses numerical codes to refer to users. Though it is very difficult to reconstruct protected personal information from such data, there remains some small risk that future models may be able to do so. The dataset we intend to release contains only de-identified user identification numbers to denote community membership and edges contain the engagement metrics of the associated tweet; no other metadata has not been retrieved. We have taken all known steps to prevent private information leakage.

heading: Acknowledgements
text: This research was supported in part by the Defense Advanced Research Projects Agency (DARPA) under grant numbers HR00112290103 and HR0011260656.
We extend our heartfelt gratitude to Maximillian Sloan, Alvyn Wang, Xiaofu Ding, Nuan Wen, and Jiaying Li for their contributions to the implementation of our methods. Special thanks are due to Arjun Pawar, Xinran Feng, and Jacob Boughter for their diligent efforts in collecting and organizing background information on the various news sources referenced in this study. Additionally, we are immensely thankful to Dr. Dana Watson Cairns for her expert advice and constructive feedback on our manuscript.
The contributions of everyone involved have been pivotal in the advancement of this research, and we are deeply appreciative of their efforts.

heading: 8 Appendices

heading: 8.1 Full Selected List of Dynamic Graph Benchmarks
text: Please see Table 8 for the full list of graph benchmarks used to inform the decisions made in this work.
Training and evaluation. We select the best epoch based on the highest GMAUC score of the link prediction task on the validation set.
Frameworks. Our entire codebase is implemented in PyTorch.1 The implementations of the transformer-based models are extended from the Huggingface2 codebase (?).

heading: 8.2 Experiment Details
text: We report the median result for ten runs with different random seeds (by default). All the models in this work are trained on NVIDIA A100 GPUs on a Ubuntu 20.04.2 operating system.

heading: 8.3 LLM Community Summaries and Prompt
text: Justification
We performed a manual tuning of 20 distinct prompt candidates and selected the prompt template that yielded the best results across the set of community summaries (150), which was a small enough set to manually validate.
Below is an example of the GPT4 response to the following prompt :"Summarize past content of NPR with respect to COVID-19 (Do not tell me you’re an AI)"
1https://pytorch.org/ 2https://github.com/huggingface/transformers
"NPR has provided comprehensive coverage of COVID-19 since the early beginnings of the pandemic. Its content has ranged from daily updates on the number of COVID-19 cases, deaths and progress on vaccines to in-depth features and analyses. They’ve frequently reported on the impact of the virus on businesses and the economy, as well as on daily life, education, and mental health issues.
NPR has regularly reported on the response of federal, state, and local governments to the pandemic, including policies put in place and financial relief measures. This has also included coverage of controversy around stay-at-home orders, mask mandates, and businesses re-openings.
They’ve conducted numerous interviews with public health officials, researchers, doctors, and individuals affected by the pandemic. They’ve also widely covered the development of COVID-19 vaccines, including the process, distribution issues, and public perception.
Specifically, there have been stories on the effect of the pandemic on marginalized communities, including racial disparities in healthcare and its impact. NPR also looked at international aspects of the pandemic, from the initial outbreak in Wuhan, China to comparisons with other countries’ responses and strategies.
NPR’s COVID-19 coverage tends to focus on factual reporting and sharing human stories and includes input from experts in multiple fields. They’ve encouraged listeners to practice safety measures and have made an attempt to dispel myths around COVID-19. The tone of the coverage, though serious, often brings in elements of empathy, humanity, and sometimes humor to lighten the heavy subject matter."

heading: 8.4 Overall Model Results Table
text: Please see Table 10 for the full tabular representation of the experiments shown in Figure 5.

heading: 8.5 Image Relevance Discussion
text: To provide an example included in our manual error analysis, we consider the following article from Hindustan Times. The article concerns beach crowds during a time of heightened infections, but the images are only of ’beach crowds’. Further work in gauging image relevance is necessary to explore the full benefits of incorporating multimodal article information into the feature initialization pipeline.
Table 10: Model Pipeline Performances
Pipeline Configuration Performance Metrics
Comm. Agg.
LLM Encoder AUC Var AUPR Var GMAUC Var
UnbiasedRandom
- Longformer 57.66 1.79E-06 24.18 1.39E-04 37.34 1.22E-04 PRIMERA
CLIP 58.56 3.52E-06 29.58 1.43E-04 41.62 5.36E-05
GPT4 60.82 1.13E-05 27.49 1.05E-04 40.89 1.03E-04 Llama-2 61.88 3.67E-06 23.72 5.15E-05 38.31 1.03E-04 PRIMERA
BERT 60.94 6.84E-05 26.14 2.41E-04 39.91 4.95E-04
GPT4 62.65 2.04E-05 23.66 1.43E-04 38.50 2.11E-04 Llama-2 52.82 6.84E-05 27.54 2.05E-04 38.14 4.95E-04
BiasedRandom
- Longformer 57.86 2.58E-03 32.70 1.075E-02 42.63 4.68E-03 PRIMERA
CLIP 61.77 1.65E-05 28.89 4.07E-06 42.24 6.24E-06
GPT4 61.79 3.81E-06 30.64 8.65E-07 43.51 5.13E-08 Llama-2 62.12 5.62E-07 32.38 7.43E-07 44.84 6.50E-07 PRIMERA
BERT 63.34 2.70E-05 31.28 1.07E-05 44.51 5.60E-06
GPT4 64.96 1.59E-05 32.43 2.19E-05 45.90 1.31E-05 Llama-2 64.26 5.83E-06 33.66 5.84E-06 46.50 6.26E-06
Engagement
- Longformer 65.17 1.16E-04 26.86 1.87E-07 41.84 1.09E-05 PRIMERA
CLIP 64.37 7.23E-05 30.56 4.20E-05 44.34 5.82E-06
GPT4 64.67 4.30E-06 26.38 3.77E-04 41.28 2.42E-04 Llama-2 65.74 6.84E-05 29.26 1.86E-04 43.85 9.72E-05 PRIMERA
BERT 64.20 1.13E-05 29.13 1.89E-06 43.24 2.02E-06
GPT4 66.61 4.42E-05 33.06 8.62E-06 46.93 1.75E-05 Llama-2 66.58 6.48E-06 33.02 9.08E-06 46.89 5.10E-06
Interaction
- Longformer 63.43 2.38E-04 32.90 2.947E-02 44.11 1.247E-02 PRIMERA
CLIP 62.52 2.84E-05 36.94 9.92E-05 48.05 7.48E-05
GPT4 62.17 6.13E-05 36.75 4.34E-05 47.80 3.62E-05 Llama-2 62.22 2.18E-05 36.83 1.19E-04 47.87 3.72E-05 PRIMERA
BERT 63.52 2.02E-05 36.71 6.82E-05 48.29 5.46E-05
GPT4 63.16 2.75E-05 36.83 6.19E-06 48.23 2.60E-06 Llama-2 62.98 4.01E-05 37.06 7.62E-05 48.32 3.48E-05
InfluencePassivity
- Longformer 66.85 3.99E-06 38.42 4.23E-04 50.66 1.69E-04 PRIMERA
CLIP 81.06 2.20E-05 53.12 3.66E-04 65.61 1.87E-04
GPT4 78.89 8.06E-05 55.25 3.07E-04 66.01 1.58E-04 Llama-2 79.33 2.31E-04 54.25 3.93E-04 65.59 2.52E-04 PRIMERA
BERT 78.95 7.15E-06 54.63 1.13E-04 65.67 3.22E-05
GPT4 82.43 1.47E-05 56.92 2.43E-05 68.50 1.98E-06 Llama-2 82.54 4.26E-06 58.17 2.02E-06 69.30 2.73E-07

heading: 8.6 List of countries and News Sources
text: We include a list of the news sources included in this work in Table 11, as well as the resources that we used to compile the list of news sources used in this work shown in Table 12.
Table 11 – continued from previous page
Country News Source Selected URLs
India Press Trust of India, CNN News 18, Times of India, The Hindu, Indian Express, Republic TV/Republic World, Hindustan Times, NDTV, Dainik Jagran, Dainik Bhaskar, Hindustan, Amar Ujala, Malayala Manorama, Sakal, Dina Thaanthi, Eenadu, Lokmat https://www. ptinews.com, https: //timesofindia. indiatimes.com
Brazil Globo News, UOL online, Record News, O Globo, Band News, CNN Brazil, O dia, Folha de Sao Paulo, O Estado de Sao Paulo, Rio Times https://g1. globo.com, https: //www.uol.com.br
Turkey Sozcu, CNN turk, Haberturk, TRT news, Sondakika, Mynet, NTV, Hurriyet, Cumhuriyet https://www. sozcu.com.tr, https://www. cnnturk.com
Italy La Repubblica, Corriere della Sera, Il Sole 24 Ore, TGCom 24, ANSA, Notizie Libero, Il Fatto Quotidiano https://www. repubblica.it, https://www. corriere.it
Spain El Pais, OKDiario, El Mundo, La vanguardia, ABC Spain, La Razon, Antena 3, Telecinco, RTVE https://elpais. com, https: //www.elmundo.es
Argentina Telefe News, Todo Noticias, Canal 13 News, C5N, A24, Infobae, Clarin online, La Nacion online https: //noticias. mitelefe.com, https://www. infobae.com
Mexico TV Azteca News, Televisa News, El Universal, Imagen News, Milenio Noticias, UnoTV news online, Aristegui News https://www. tvazteca.com, https://www. televisa.com
Israel Israel Hayom, Maariv, Haaretz, Yedioth Ahronoth https://www. israelhayom.com, https://haaretz. com
Iran Tehran Times, Iran Daily, Iranian Students News Agency, Financial tribune, Resalat, Hamshahri, Kayhan, Ettelaat https://www. tehrantimes.com, https://en.isna. ir
Germany ARD News, ZDF News, RTL News, t-online, Spiegel online, Bild.de https://www. tagesschau.de, https://www.zdf. de
Poland TVN News, RMF FM / RMF24, Polsat News, Radio Zet, Fakt, Gazeta Wyborcza, Onet, WP, Interia https://tvn24. pl, https: //www.rmf24.pl
Source Name Link
Pew Research Center Reports (pew) N/A News Agencies - Russia (new) N/A Mass media in Russia (Wikipedia) (mas) https://wikipedia. org China Central Television (CCTV) Reports (cct) N/A Oxford Studies (oxf) N/A Digital News Report (Reuters Institute) (reu)
https:// reutersinstitute. politics.ox.ac.uk/ digital-news-report
BBC Guide for India (bbc, a) N/A Audit Bureau of Circulations India (aud) N/A Potential Political Leanings (Twitter, Reddit) (?) Twitter, Reddit Media Bias/Fact Check (med)
https:// mediabiasfactcheck. com
BBC Media Guide Brazil (bbc, b) N/A Eurotopics (eur) https://www. eurotopics.net

heading: Future Work
text: Though it is very difficult to reconstruct protected personal information from such data, there remains some small risk that future models may be able to do so. The dataset we intend to release contains only de-identified user identification numbers to denote community membership and edges contain the engagement metrics of the associated tweet; no other metadata has not been retrieved. We have taken all known steps to prevent private information leakage