SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As per the discussion, users are always involved in the diffusion of trending topics that their social circle are talking about. To investigate such influence, we propose social circle influence representation after topic propagation process. In social circle influence representation, a user node will be imposed influence from their neighbors who have paid attention to the some trending topic. In PON  $G$ , every user node in  $U$  will receive different amounts of topic information from its user neighbors. We use the graph attention mechanism,  $\rho_{i,j} = \mathrm{softmax}(\mathrm{att}(\pmb{h}_i',\pmb{h}_j'))$ , to measure the influence weight for node  $v_i$  from its neighbor  $v_j$ , where  $h_i'$  and  $h_j'$  are the topic propagation representation of node  $v_i$  and  $v_j$  in Section 3.1. Therefore, the total influence on user  $v_i \in U$  from its social circle (i.e.,  $N_i$ ) can be computed as follows:
$$
\operatorname {P o o l} \left(\boldsymbol {h} _ {j} ^ {\prime} \mid v _ {j} \in N _ {i}\right) = \sum_ {v _ {j} \in N _ {i}} \rho_ {i, j} \cdot \boldsymbol {h} _ {j} ^ {\prime} \tag {9}
$$
Note that even though  $v_{i}$  can be influenced by its neighbors, its own representations learned from topic propagation representation are fundamental and important. Thus the final representation  $\mathbf{z}_{i}$  of a user node  $v_{i} \in U$  is as follows:
$$
\boldsymbol {z} _ {i} = \sigma \left(\boldsymbol {W} \cdot \left(\boldsymbol {h} _ {i} ^ {\prime} \oplus \operatorname {P o o l} \left(\boldsymbol {h} _ {j} ^ {\prime} \mid v _ {j} \in N _ {i}\right)\right)\right) \tag {10}
$$
Note that after topic propagation representation and social circle influence representation, the features of all the nodes in  $G$  are updated to  $\mathbf{Z} \in \mathbb{R}^{|V| \times f}$ .
Unsupervised Proximity Loss. In real world, proximity exists extensively in a public opinion field. The users within the same pof always have similar features such as background, career, interest and so on. For example, users who actively discuss ChatGPT usually have the same background or interest in computer science. Moreover, the features of the nodes within a pof should be correlated with feature of poc node in this pof, which is the reason that they prefer to pay attention to this poc node. In this paper, we utilize such proximity property, by enhancing the correlation of embeddings between poc node and all the nodes within the same pof, to obtain higher-quality embeddings from
our representation learning model. Specifically, we first use the readout function to aggregate the features of all the nodes within  $\mathsf{pof} P_i$  to generate their common embedding  $\boldsymbol{e}_i \in \mathbb{R}^f$ :
$$
\boldsymbol {e} _ {i} = \operatorname {R e a d o u t} \left(\boldsymbol {H} _ {i}\right) \tag {11}
$$
where  $H_{i} \in \mathbb{R}^{|N_{i} \cup \{v_{i}\}| \times f}$  denotes the embedding of all the nodes in the pof  $P_{i}$ . Readout( $\cdot$ ) denotes a simple permutation-invariant function (e.g., sum and mean). Let  $E_{c} \in \mathbb{R}^{|C| \times f}$  and  $Z_{c} \in \mathbb{R}^{|C| \times f}$  denote the readout embedding of all pofs and the embedding of all poc nodes, respectively. In order to maximize the correlation between the poc node and all the nodes within the same poc, the mutual information (MI) between  $E_{c}$  and  $Z_{c}$  is maximized. Specifically, given two random variables  $X$  and  $Y$ , we adopt MINE method proposed in [3] to maximize the lower bound of  $\mathrm{MI}(X, Y)$  as follows:
$$
\max  \operatorname {M I} (\boldsymbol {X}, \boldsymbol {Y}) \geq \mathbb {E} _ {P (\boldsymbol {X}, \boldsymbol {Y})} [ D (\boldsymbol {X}, \boldsymbol {Y}) ] - \log \mathbb {E} _ {P (\boldsymbol {X}) P (\boldsymbol {Y})} \left[ e ^ {D (\boldsymbol {X}, \boldsymbol {Y})} \right] \tag {12}
$$
where  $D(\mathbf{X},\mathbf{Y})$  is a binary discriminator. Therefore, to maximize  $\mathrm{MI}(\pmb {E}_c,\pmb {Z}_c)$ , the unsupervised proximity loss  $L_{p}$  is defined below:
$$
L _ {p} \left(\boldsymbol {E} _ {c}, \boldsymbol {Z} _ {c}\right) = - \mathbb {E} _ {P \left(\boldsymbol {e} _ {i}, \boldsymbol {z} _ {i}\right)} \left[ D \left(\boldsymbol {e} _ {i}, \boldsymbol {z} _ {i}\right) \right] + \log \mathbb {E} _ {P (\boldsymbol {e}) P (\boldsymbol {z})} \left[ e ^ {D \left(\boldsymbol {e} _ {i}, \boldsymbol {z} _ {i}\right)} \right] \tag {13}
$$
where  $e_i$  and  $z_i$  denote the readout representation and the original representation of the poc node  $v_i$ . The discriminator  $D(\cdot, \cdot)$  is defined as a simple neural network,  $D(e_i, z_i) = \sigma(e_i^\top W z_i)$ . In the experiment, for each batch, we estimate the first term in Eq. (13) by selecting a set of  $\{(e_i, z_i)\}_{i=1}^B$  from the joint distribution  $P(e_i, z_i)$ , and then we shuffle the  $z_i$  in the batch and generate "negative pairs" in order to estimate the second term.
Supervised Loss. For downstream tasks, we can minimize the cross-entropy loss function between ground truth and prediction:
$$
L _ {s} = - \sum_ {l \in L} \boldsymbol {Y} ^ {l} \ln (\hat {\boldsymbol {C}} \cdot \boldsymbol {Z} ^ {l}) \tag {14}
$$
where  $\hat{C}$  is the parameter of the classifier,  $L$  is the set of labeled nodes/edges,  $\mathbf{Y}^l$  and  $\mathbf{Z}^l$  are the label vector and embeddings of the labeled nodes/edges respectively. Finally, We define the total loss by linearly combining these two component losses:  $L = L_{s} + \mu L_{p} + \varphi L_{reg}$ , where  $\mu$  is the hyperparameter that controls the proportion of unsupervised proximity loss and  $\varphi L_{reg}$  is the regularization term to alleviate overfitting.
Datasets. Detailed statistical information of the datasets is shown in Table 1. (1) BuzzFeed and PolitiFact. The two datasets are from a comprehensive fake news detection benchmark dataset, FakeNewsNet [47, 48]. They are collected from two platforms with fact-checking: BuzzFeed and PolitiFact, both containing news content with labels and social context information. (2) Twitter. This dataset [31, 47] contains multiple articles, with each article corresponding to one event. For each event, the dataset includes its source, a list of participating users, and their tweets. This dataset also includes Twitter profile description and the list of Twitter profiles each
Table 1: Statistics of datasets.
Dataset	Relation (A-B)	# A	# B	# A-B	Average Degree	Average Clustering Coefficient	Connectivity
BuzzFeed	user-user	15257	15257	634750
user-event	15257	182	22779	88.01	0.087	True
source-event	28	182	174
PolitiFact	user-user	23865	23865	574744
user-event	23865	240	32791	53.13	0.066	True
source-event	17	240	236
Twitter	user-user	54461	54461	3172794
user-event	54461	1054	75809	118.25	0.094	True
source-event	442	1054	1054
source-source	442	442	3707
user follows. More information on the three datasets can be found in Appendix A.1 and Appendix A.2.
Baselines. For simplicity, our model is denoted as POFD (Public Opinion Field Effect Fusion in Representation Learning for Trending Topics Diffusion). We compare our model against three categories of graph representation learning methods: (1) Homogeneous-node2vec [13], GCN [60] and GAT [57]. (2) Heterogeneous-mp2vec [8], HAN [58], HetGNN [63], HGT [17] and HALO [1]. (3) Misinformation detection-FANG [35] and MetaHG [40].
Table 2: Experimental results (%) for the public opinion concern prediction task.
Dataset	Metrics	node2vec	GCN	GAT	mp2vec	HAN	HetGNN	HGT	HALO	FANG	MetaHG	POFD
BuzzFeed	AUC	76.12±0.37	92.87±0.58	93.29±0.53	86.13±0.33	93.43±0.51	93.77±0.68	93.88±0.73	94.32±0.48	90.92±0.61	93.07±0.43	96.36±0.33
AP	71.46±0.27	92.44±0.49	92.06±0.39	85.93±0.34	91.95±0.57	92.39±0.58	92.49±0.62	92.98±0.36	90.12±0.41	91.68±0.29	96.09±0.36
PolitiFact	AUC	74.22±0.35	93.02±0.24	93.61±0.64	88.16±0.39	93.88±0.24	93.29±0.35	92.39±0.47	93.57±0.32	91.31±0.29	92.85±0.43	95.44±0.37
AP	69.19±0.37	92.49±0.33	92.72±0.32	89.24±0.43	93.21±0.27	92.43±0.31	92.03±0.23	92.75±0.27	91.49±0.41	92.37±0.38	95.57±0.33
Twitter	AUC	79.14±0.41	93.12±0.41	93.68±0.45	88.30±0.33	93.90±0.62	92.41±0.43	93.12±0.26	94.25±0.34	92.03±0.39	93.12±0.44	96.18±0.44
AP	78.67±0.24	92.50±0.34	92.74±0.42	87.84±0.45	92.82±0.51	92.01±0.38	92.50±0.34	92.41±0.27	91.66±0.38	92.50±0.33	94.99±0.25
Parameter Settings. We set the embedding dimension to 256, return parameter to 1, in-out parameter to 1, path length to 40, and window size to 10 in node2vec and mp2vec. All GNN models are two-layer structures with a hidden layer dimension of 256 and an output dimension of 128 and are implemented through the PyTorch Geometric (PyG) package [10]. The models that require setting the number of attentional heads are set to 4. In POFD, we select the top 500 user nodes in degree ranking as  $U_{h}$ . The energy amplification factor of the fake news event is set to 3. For all neural network models, the learning rate is 0.001, the number of training epochs is 200. For all methods, we run 10 times with the same partition and report the average results. The dataset and code with running instructions are available at https://github.com/ki-1jl/POFD.
4.2 Public Opinion Concern Prediction
Setting. In this paper, we define public opinion concern prediction as two types of link prediction: user-user and user-event. Specifically, we randomly sample  $10\%$  of the two types of links from the dataset to generate test edges. In the test edges, the positive and negative edges are each  $50\%$ . Then, we mask the positive test edges from the dataset to prevent information leakage. We report the AUC and average precision (AP) of all models in Table 2.
Analysis. It is clear that POFD consistently performs better than all baselines on three datasets. Compared to the best performance of baselines, POFD achieves a high improvement in both AUC and AP, which indicates the effectiveness of considering public opinion field effect and social circle influence in representation learning. We also observe that GNNs generally achieve better performance than random walk-based methods, due to the fact that random walk-based methods can only simply exploit structural information in the PON without considering feature information that is crucial in topic diffusion.
4.3 Event Classification
Setting. We also conduct the event classification task to evaluate the embeddings learned from POFD. Specifically, since the properties of the event (positive or negative) are unknown, we first set the energy amplification factor of all poc nodes to 1. Then for each dataset, we randomly split event nodes into  $70\%$ ,  $10\%$ , and  $20\%$  for training, validation and testing. We report the accuracy and F1 scores of all models in Table 3.
Analysis. Based on the results reported in Table 3, we observe that POFD achieves the best performance on all metrics for all three datasets. By capturing the interactions between pofs of different events, POFD can learn more meaningful event node embeddings and thus achieves the best performance on the event classification task.
4.4 Universality Analysis
In addition to pof in PON, there are various fields in real life, such as academic field in DBLP [12]. Specifically, DBLP is a well-known heterogeneous academic network containing four types of nodes (author, paper, term and conference) and three types of links (author-paper, paper-term and paper-conference). More information about DBLP can be found in Appendix A.1. Like events attract users in PON, conference nodes in DBLP attract the attention of authors and their papers (and terms). Specifically, authors may be attracted to multiple conferences (this attraction is linked through author-paper and paper-conference).
Setting. In this experiment, we let the conference nodes be poc nodes and then conduct the research domain classification task for the author nodes. We use the DBLP dataset provided in PyG [10], which
Table 3: Experimental results (%) for the event classification task.
Dataset	Metrics	node2vec	GCN	GAT	mp2vec	HAN	HetGNN	HGT	HALO	FANG	MetaHG	POFD
BuzzFeed	ACC	77.78±0.38	86.11±0.56	88.89±0.32	80.56±0.37	86.11±0.37	83.33±0.48	86.11±0.51	86.11±0.28	88.89±0.34	88.89±0.41	94.44±0.34
F1	69.23±0.51	82.76±0.34	85.71±0.27	72.00±0.46	82.76±0.31	78.57±0.26	82.54±0.43	83.33±0.48	83.33±0.41	84.62±0.35	92.31±0.47
PolitiFact	ACC	70.83±0.31	85.42±0.34	85.42±0.44	72.92±0.47	87.25±0.25	87.50±0.34	89.58±0.38	91.67±0.35	89.58±0.33	93.75±0.25	95.83±0.38
F1	66.67±0.37	84.44±0.38	84.44±0.18	72.35±0.44	86.06±0.48	87.50±0.38	88.37±0.34	90.48±0.29	87.18±0.33	92.68±0.39	95.24±0.32
Twitter	ACC	61.14±0.49	68.24±0.34	68.72±0.27	66.82±0.35	72.35±0.43	72.22±0.43	77.78±0.38	82.93±0.33	78.67±0.33	83.33±0.24	85.31±0.48
F1	61.32±0.34	59.88±0.24	62.06±0.54	64.29±0.41	74.15±0.17	66.67±0.27	71.43±0.52	74.33±0.31	72.45±0.43	77.34±0.32	80.57±0.34
Table 4: Experiment results (%) on the DBLP for the node classification task.
Metrics	node2vec	GCN	GAT	mp2vec	HAN	HetGNN	HGT	HALO	FANG	MetaHG	POFD
Macro-F1	83.67±0.34	91.89±0.32	90.80±0.42	87.42±0.24	92.43±0.38	92.64±0.45	92.01±0.37	92.81±0.23	90.06±0.30	91.13±0.37	95.03±0.24
Micro-F1	85.27±0.41	92.62±0.24	91.84±0.37	87.92±0.34	92.85±0.34	92.09±0.37	92.60±0.24	93.34±0.24	90.75±0.25	91.74±0.25	95.33±0.42
(a) Public opinion concern prediction.
Figure 3: Ablation study on BuzzFeed dataset.
(b) Event classification.
contains a predefined training, validation, and testing set. We report the Macro-F1 and Micro-F1 of all models in Table 4.
Analysis. As shown in Table 4, POFD achieves the best performance on DBLP. The experimental results indicate that the academic field formed by conference nodes can effectively capture the potential influence of conference (poc) nodes on authors and their published papers, which in turn can indirectly guide the classification task of author nodes through conference information (authors always tend to publish in conferences related to their research fields).
4.5 Multi-hop Public Opinion Field
In this section, we experimentally demonstrate the reasonableness of our choice of 1-hop neighbors in Definitions 2 and 3. We give experimental results for 1-hop to 4-hop neighbors in Appendix A.3. The experimental results show that the use of multi-hop neighbors does not improve the model performance, probably because multi-hop neighbors introduce more redundancy. This indicates that pof energy is mainly determined by the 1-hop neighbors of poc nodes.
4.6 Centrality
In Eq. (1), we define  $w_{i,j} = \lambda_j \cdot \operatorname{att}(h_i, h_j)$ , where  $\lambda_j$  is used to measure the influence of node  $v_j$  in the PON. In this paper, we let  $\lambda_j$  be the degree centrality of  $v_j$ . It is well known that there are various other centrality metrics such as Eigenvector Centrality. However, as we stated earlier, there have been many studies on how to measure the influence of nodes in a network.  $\lambda_j$  is only used as an input for the study in this paper, and how to choose  $\lambda_j$  is not part of the study in this paper. In this section, we investigate the impact of different centrality metrics on the model performance. The experimental results and analysis can be found in Appendix A.4.
4.7 Ablation Experiment
Setting. In this section, we investigate how the three components of POFD affect its performance. Specifically, we create the following ablations: (1)  $\mathbf{POFD}_{tp}$ : keep topic propagation but remove social circle influence. (2)  $\mathbf{POFD}_{sci}$ : keep social circle influence but remove topic propagation (we use GAT [57] to replace topic propagation). (3)  $\mathbf{POFD}_{loss}$ : remove unsupervised proximity loss. We conduct ablation experiments on BuzzFeed, and the experiment results are shown in Fig. 3.
Analysis. Fig. 3 reports the results of the ablation experiments. We observe that POFD outperforms the three variants mentioned above, which indicates that all three modules of POFD can individually improve the model performance.
Content-Based. The content-based methods focus on the diffusion of information from blogs or news content. Mahdikhani et al. [32] collected over 1.25 million tweets containing timestamps to predict the popularity of tweets (based on the number of retweets) by extracting the features of the tweets. Tsagkias et al. [54] used a two-stage binary classification to predict the number of comments on online news reports. Kaltenbrunner et al. [21] proposed a growth model for predicting the popularity of slashdot stories. Lee et al. [26] investigated the probability that a web content will continue to be followed by online readers after a period of time. In general, this type of approach often requires the processing of content to produce manual features, often with significant workload.
Homogeneous Network-Based. With the rise of graph mining techniques [37, 13, 60, 15, 57], a large number of people have started to study how to model information diffusion using graph structures. Bourigault et al. [5] embedded nodes in a continuous latent space and calculate the contamination scores of nodes activated by source nodes through a diffusion kernel function in the latent space. Inf2vec [9] studied the problem of influence embedding of nodes in social networks. RNe2Vec [46] estimated the information diffusion probability based on the random wandering method. Based on the state of the nodes' network and local neighborhoods, DeepInf [41] aimed to predict the activation probability of the nodes. CasSeqGCN [59] combined network structure and time series to predict the information cascade. None of these models can model complex social contexts such as user-event relationships, and thus cannot extract high-quality embedding representations.
Heterogeneous Network-Based. The heterogeneous network-based methods learn high-quality embedding representations through diverse neighborhood relationships. Although earlier studies such as [19, 38, 39, 49] also considered heterogeneous scenarios, they all optimized a specific objective (e.g., fake news detection) rather than extracting abundant node representations, and thus were not very generalizable. FANG [35] improved the quality of node representation learning by capturing the rich social interactions between users, news and media to improve the accuracy of fake news detection. Such approaches mainly improve the embedding representation by considering heterogeneous information in the network, but they do not make use of some deeper semantics such as public opinion field effects.
In this paper, we propose for the first time to fuse the public opinion field effect into the representation learning of trending topics diffusion. Then, we propose a representation learning framework POFD that includes both topic propagation and social circle influence. Experimental results including public opinion concern prediction and event classification demonstrate the effectiveness of POFD. We also show that POFD is also effective for other types of fields (e.g., academic fields in DBLP).
Ethics Statement. An important application of our study is fake news controlling. For example, wiping out $130 billion in stock value after a false tweet claimed that Barack Obama was injured in an explosion [42]. Our intention behind this work is that studying trending topic diffusion can help timely discover potential crises caused by fake news and clarify the truth with evidence as soon as possible to avoid unnecessary losses.
Limitations and Future Work. We note that POFD does not consider the dynamics of PON, such as the disappearance of old events and the emergence of new events. Therefore, in the future we will investigate representation learning for dynamic PONs.
This work was supported by the State Key Laboratory of Communication Content Cognition Funded Project No. A32003, National Natural Science Foundation of China No. U22A2025 and No. 61972275.
A.1 Datasets Processing Details
In this section, we describe how the dataset is processed to obtain the inputs for the model.
- BuzzFeed and PolitiFact. For the news nodes, we use the BERT [7] to process the textual content of the news and generate an embedding of dimension 768 for each news node. For user nodes and source nodes, we use metapath2vec [8] to generate embeddings of dimension 128 for both types of nodes since there is not enough information as input.
- Twitter. We follow the settings in FANG [35] in this paper. For the news nodes, we construct a TF-IDF [43] vector from the text body of the article. We enrich the representation of news by weighting the pre-trained embeddings from Glove [36] of each word with its TF-IDF value forming a semantic vector. Finally, we concatenate the TF-IDF and semantic vector to form the news article feature vector. We calculate the user vector as the concatenation of a pair consisting of a TF-IDF vector and a semantic vector derived from the user profile's text description. Similarly to article representations, for each source node, we construct the source feature vector as the concatenation of its TF-IDF vector and its semantic vector derived from the words in the homepage and the "About Us" section, as some fake news websites openly declare their content to be satirical or sarcastic.
- DBLP. In the universality analysis experiment, we directly use the DBLP dataset provided in PyG [10]. The dataset contains 4057 authors, 14328 papers, 7723 terms, and 20 publication venues. The authors are divided into four research areas (Database, Data Mining, Artificial Intelligence, and Information Retrieval). Each author node is described by a bag-of-words representation of their paper keywords. For semi-supervised learning models, the author nodes are divided into training, validation, and testing sets of 400 (9.86%), 400 (9.86%), and 3257 (80.28%) nodes, respectively. Since there are no initial features for the publication venue nodes in this dataset, we use the torch.randn function to generate initial features of dimension 128 for each publication venue node.
A.2 Dataset Attributes
We describe some basic properties of the three datasets in Table 1, such as the number of nodes and the number of edges. In addition, we also give the Average Degree, Connectivity and Average Clustering Coefficient of the three networks. Furthermore, we give the degree distribution of the three networks in Fig. 4.
(a) BuzzFeed.
(b) PolitiFact.
(c) Twitter.
Figure 4: The degree distribution of the datasets.
A.3 Multi-hop Public Opinion Field
In this section, we investigate the impact on model performance when constructing pof and pof energy using multi-hop neighbors of poc node. We give experimental results for 1-hop to 4-hop neighbors in Table 5. Experimental results show that using multi-hop neighbors only degrades the performance of the model. We speculate that using multi-hop neighbors may bring additional redundant information.
A.4 Degree Centrality and Eigenvector Centrality
We describe the effect of degree centrality and eigenvector centrality on model performance in Table 6, where  $\mathrm{POFD}_d$  and  $\mathrm{POFD}_e$  denote the POFD using degree centrality and eigenvector centrality, respectively. Based on the experimental results, we find that the performance of the models implemented with the two centrality metrics is approximately equivalent. We hope that there will be follow-up work in the future to investigate how to assess the importance of nodes in the public opinion space.
Table 5: Multi-hop neighborhood experiment.
Table 6: Centrality experiment.
Dataset	Metrics	1-hop	2-hop	3-hop	4-hop	Dataset	Metrics	POFDd	POFDe
BuzzFeed	AUC	96.36±0.33	93.18±0.73	89.41±0.33	82.72±0.62	BuzzFeed	AUC	96.36±0.33	96.89±0.33
AP	96.09±0.36	93.38±0.66	90.32±0.36	79.22±0.43	AP	96.09±0.36	96.75±0.28
PolitiFact	AUC	95.44±0.37	93.64±0.51	88.98±0.37	79.83±0.62	PolitiFact	AUC	95.44±0.37	95.72±0.41
AP	95.57±0.33	94.43±0.23	90.55±0.34	78.89±0.34	AP	95.57±0.33	95.51±0.28
Twitter	AUC	96.18±0.44	91.22±0.26	87.34±0.51	79.28±0.28	Twitter	AUC	96.18±0.44	95.86±0.51
AP	94.99±0.25	91.53±0.34	88.21±0.25	80.40±0.33	AP	94.99±0.25	94.13±0.25