One-Model-Connects-All: A Unified Graph Pre-Training Model for Online Community Modeling
Authors: Ruoxue Ma
Jiarong Xu
Xinnong Zhang
Haozhe Zhang
Zuyu Zhao
Qi Zhang
Xuanjing Huang
zhongyu wei
Venue: EMNLP 2023 Findings (2023)
Keywords: community
pre-train
graph

Abstract: Online community is composed of communities, users, and user-generated textual content, with rich information that can help us solve social problems. Previous research hasn't fully utilized these three components and the relationship among them. What's more, they can't adapt to a wide range of downstream tasks. To solve these problems, we focus on a framework that simultaneously considers communities, users, and texts. And it can easily connect with a variety of downstream tasks related to social media. Specifically, we use a ternary heterogeneous graph to model online communities. Text reconstruction and edge generation are used to learn structural and semantic knowledge among communities, users, and texts. By leveraging this pre-trained model, we achieve promising results across multiple downstream tasks, such as violation detection, sentiment analysis, and community recommendation. Our exploration will improve online community modeling.

One-Modei-Connettt-All: A Unified Graph Pre-Training Model for Online Community Modeling
Ruoxue Ma', Jiarong Xu²; Xinnong Zhang', Haozhe Zhang', Zuyu Zhao',
Qi Zhang', Xuanjing Huang', Zhongyu Wei
Sccooo of Data Science, Fudan University, China
School of Management, Fudan University, China
"School of Computer Science, Fudan University, China
*Research Institute of Intelligent Complex Systems, Fudan University, China 'Huawei Technologies Co.,Ltd, China {rxma21, xnzhang23 fudan. edu. cn
{jiarongxu, qz, xjhuang, j@fudan. edu. cn (zhanghaozhe7, zhaozuyu1 yehuawei Jeauaweel com
Abstract
Texts
Communitaes
Users
Online community is composed of communi- ties, users, and user-generated textual content, with rich information that can help us solve so- cial problems. Previous research hasn't fully utilized these three components and the rela- tionship among them. What's more, they can't adapt to a wide range of downstream tasks. To solve these problems, work that simultancously considers communi- ties, users, and texts. And it can easily con- nect with variety of downstream tasks lated to social media. Specifically, we use ternary heterogeneous graph to model online communities. Text reconstruction and edge generation are used to learn structural and se- mantic knowledge among communities, users, and texts. By leveraging this pre-trained model, we achieve promising results across multiple downstream tasks, such as violation detection, sentiment analysis, and community recommen- dation. Our exploration will improve online community modeling.
cilati Sod
Yoate it
Senacio Duaoor Solertt
focus frame-
Figure I: An illustration of the connection among com- munities, users, and texts. Take Reddit, which has many online communities called subreddit, Similar subreddit r/fantasyfootball and r/detroitlions share more common users and have more similar se- mantics, while r/fantasyfootball and r/BgesBurgers do not. We slightly process the text for privacy protection.
example.
topics and discussions (Haythornthwaite, 2007; Baym, 2000).
There are many studies focused on integrating online community to solve social problems such as abusement detection (Park et al., 2021). conflict prediction (Kumar et al., 2018), and bias study (Strzalkowski et al., 2020). Previous research mainly aims specific social problem. Although they achieve positive results on target tasks, their methods require designing different models for dif- ferent tasks, which is very costly. The training pro- cess also requires high-quality, annotated datasets, which is very time-consuming and labor-intensive In addition, there is a strong connection among communities, users, and texts shown in Figure 1. But previous research did not take into account all these three components simultaneously. Some work only modeled the relationship between com- munities and users (Kumar et al., 2018; Martin, 2017; Waller and Anderson, 2019), while others
1 Introduction
An online community refers to a group of individ- uals who share similar interests or purposes and utilize the internet as a means to communicate with one another (Haythortthwaite, 2007; Baym, 2000). It serves as vital component of social networks, including platforms like NextDoor and Facebook groups. Typically, an online community is com- posed of users, user-generated textual content, and communities. Members within the same commu- nity often share common languages, behavior rules, and social cognition, and actively contribute to the community by posting or commenting on various
*corespponding author

---

only utilize the community and their textual con- tent (Strzalkowski al., 2020; Trujillo et al., 2021). What's more, these studies model the online com- munities' relationships in a simple way such as appending community information to the text as an additional contextual input (Park et al., 2021).
2 Proposed Framework
The proposed framework, as illustrated in Figure 2, consists of three components: Graph Construction, Pre-training, and Downstream Tasks.
2.1 Graph Construction
In this paper, we target unified pre-training framework that simultaneously models community, textual content, and user along with their informa- tion flow, such that one pre-trained model can sup- port a wide range of downstream tasks for online community.
2.1.1 Data Collection
Reddit is a social media website that is divided into different online communities called subred- dit. Users can post in topic-specific subreddit and comment posts in their interested subreddit. Sub- reddit inherently possesses the characteristics of online community, which makes Reddit an ideal data source for studying community profiling.
First, achieve this, we begin by constructing heterogeneous graph that captures the complex relationships among communities, users, and tex- tual content. The constructed graph consists of three types of nodes (community, user, and text) and three types of edges (post in, post by and ac- tive). This facilitates the subsequent pre-training tasks can capture community-level, user-level, and text-level information and the connection among different levels.
In 2021, the total number of subreddits has ceeded 100,000. Although there are large number of subreddits, we find that only 2% of subreddits can cover 75% the posts. So we only use a sub- reddit collection, which consists of the top 2% subreddits by number of posts per month, as our research object.
There are two types of text in Reddit: submis- sions and comments. Submissions are posts in sub- reddit and comments comments. Comments are threaded: Comments can be made directly on sub- missions or appear as reply to another comment. After deleting empty texts, deleted texts, removed texts, texts published by deleted users, too long or too short text, and all signed text, we sample 1.5% of submissions and 1%cof comments as re- search subjects. The small sampling proportion of comments is due to the vast quantity of com- ments, coupled with their lower semantic meaning compared to submissions.
Next, we introduce two self-supervised tasks such that the pre-trained model built upon the constructed hetcrogeneous graph can encode rich knowledge of communities, users, and texts. Text reconstruction task is designed to enable the model to learn the language style of communities and users, while the edge generation task aims to focus the connection among texts, communities, and users.
Third, to support different downstream tasks within online community, we fine-tune the pre- trained model on various downstream tasks such text classification and community recommen- dation. By simply adding a fully connected layer, model can be easily applied to community- level, user-level, and text-level tasks. Extensive experiments have been completed confirm the effectiveness our framework.
We define figuenre users as those whose number of submissions and comments falls in the top 25% of the subreddit annually. Due to the huge amount of data, we only focus on the high- frequency users among the authors of above sam- pled texts. Table 3 in A. presents the basic sum- mary statistics of our dataset.
2.1.2 Heterogeneous Graph Construction Heterogeneous graph consists of three types of nodes.
Our main contributions are as follows: Firstly, we propose unified framework containing com- munities, users, and textual content simultancously. Secondly, we employ graph-based approaches to model the relationship among these three elements. Lastly, we use two self-supervised tasks to pre- train this ternary graph, making it casy to adapt to multiple downstream tasks related to social media.
(2,, ]N.} is the set of community nodes, where ci 1, 2.... is the i-th com- munity node and N the number of commu- nity nodes.
'htps//wwww.rediir.com

---

a) Graph Construction
b) Pre-training
Downstream Tasks
Select Texts
Select Communities
Select Authors
Mask Nodes
sieazand
AVG
AVG
XLNET
AVG
AVG
XLNET
LEGT
Content- based Info
nStrwathh rigit
I[G1
Connection -based Info
10S8Am
Mask Edges
Construct Graph
Community
Text
FC
PC
PC
AVG
AVG
XLNET
Post
Active
HGTT
Community Recommendation
Text Sourve
Text- Related Tacks
Des
by
LosS
Figure 2: The proposed framework architecture. a) Graph Construction. We collect data and construct ternary heterogeneous graph incorporating community, user, and text nodes. b) Pre-training. Two self self-supervised tasks are used to effectively capture and leverage the rich knowledge the graph. c) Downstream Tasks. We transfer the learned parameters downstream tasks and add fully connected layers to adapt different objectives.
T = [ tt,,,, the set of text (sub- mission and comment) nodes, where each ti - 2.... is the i-th text and N₃ is the number of text nodes.
is in the top 25% of the subreddit annually. By connecting the community node and user node, user node can obtain supplementary in- formation from their highly engaged commu- nitics and community node can aggregate the information from their high-frequent users.
U {u1, UN. is the set of users, where each u;(i 2, is the set of user nodes and N,, is the number user nodes.
Relations between texts and users (post by). This type of edge represents the affiliation be- tween the text and its author. We construct an adjacency matrix B. Each element bij in- dicates whether the user ut writes this text tj.
Three types of edges can be constructed by con- necting three types of nodes.
Relations between communitics and texts (post in). In Reddit, each text is posted in corresponding community, so is easy to get the relationship between text nodes and com- munity nodes. By connecting the community node and text node, text node can obtain sup- plementary information from their community and community node can aggregate the infor- mation from all texts posted in it to get better representation. We construct an adjacency matrix A. Element aij indicate whether the community e; contains the text tj.
2.2 Pre-training
2.2.1 Initialization of Nodes
Large-scale pre-trained language models have learned rich semantics and can provide strong sup- port for various natural language processing tasks (Devlin et al., 2018; Liu et al., 2019; Zhang et al.. 2020) Due to the rich textual information of text nodes, we use XLNet (Yang et al., 2019) to initial- ize the text nodes.
X₁, XLNet(clean(tt))
(1)
Relations between communities and users (ac- tive). We define high-fir fettufrreunccc users as those whose number of submissions and comments
clean is text preprocessing function, which in- cludes removing symbols, modifying "hyperlinks"

---

and "mentioning users" [URL] and [USER]. Other nodes are initialized by averaging the vectors of text nodes connected them.
Edge Generation. Some edges are masked this stage. After 2.2.2. the output of the i-th node can be obtained as h/ E.ge We denote the set of nodes connected to the above i-th node but their edge is masked as S . C St. We also denote the decoder of edge generation as DeCEdge(). For each i and j+, we sample some unconnected nodes of the same type j+ and denote this set as S Then, contrastive loss can be calculated via
X AX)((
(2)
x AVG(X.))
AVG represents the average operation, and t represents the text nodes connected subreddit nodes and user nodes. After initializing, we obtain X E Nexd Xu E RNxxd, Xx, E Mid
expl DDecrgge (h. Edge h ()
Lt Edge
Σ
2.2.2 Graph Neural Network
st stes 5ES, -0-} wwpp Decgdge (h.
(hidde Edge Bedge))
We then feed the output of initializing into HGT (Hu et al., 2020b) module to update their representa- tions. HGT is graph neural network model which introduces attention mechanisms related node and edge types solve graph heterogeneity. con- sists of three parts: Heterogeneous Mutual Atten- tion, Heterogeneous Message Passing, and Target Specific Aggregation. After HGT module, we ob- tain: he = (G)) hy (Xu) he HGT X1), where he E RNNexd, hy E IR Nux, h4 E -.
(5)
1. Edge AVG(I (dgg)) 1,2,
(6)
Optimizing I, Fidge is equivalent to maximizing the likelihood of generating edges. In this way, the pre-trained model can capture the intrinsic struc- tures of graph, which means it can better learn the connections among users, texts, and communities. Overall Pre-training, Final loss is linearly com- bined by above two factors:
2.2.3 Pre-training Process
L XL Attr + (1 --) Edge
(7)
There two self-supervised tasks in GPT-GNN (Hu al., 2020a): node generation and edge gen- eration. Inspired by GPT-GNN, we construct text reconstruction and edge generation on our dataset. Same as GPT-GNN, we need to separate the text nodes into texl reconstruction nodes and edge gen- eration nodes to prevent information leakage dur- ing edge generation.
where À is hyperparameter controlling the weight of two losses.
2.3 Downstream Tasks
2.3.1 Methodology
We select several Reddit-related datasets and build them into the same graph structure as the above process. We use pre-trained model parameters initialize the model of downstream tasks and add a fully connected layer to adapt to different objec- tives. Our downstream tasks can be divided into node-related and edge-related.
Text Reconstruction. We mask the text recon- struction nodes with a dummy token and define the output of the dummy token generated by 2.2.2 as Attr We also denote the decoder of attribute generation as DECAur Dec () takes Att as input and generates the predicted attributes. A distance function is used to calculate the distance between the generated attributes and the original attributes. This distance function is also the loss function of this task.
2.3.2 Datasets
Experiments are conducted on five Reddit-related datasets. We perform additional crawls for text without community and author details. Preserving the structure of original datasets, we remove sam- ples whose text has been deleted or whose author has been deleted. Therefore, the composition of the dataset differs from the original. We report the size and partitioning of each dataset in the appendix.
L Attr AVG(DistanceeDee..a (M Att),, X1,)),
1,
(4)
Minimizing the distance function equivalent to maximizing the likelihood of observing the text. this way. the pre-trained model can capture the semantic of communities and users.
Node-related Downstream Tasks

---

Node-related Tasks
Edge-related Tasks
Anxiety-
Community
NORMVIO
Ruddit
Dreaddit
on on-Reddit
Recommendation
ACC
F1
MSE
ACC
F1
ACC
F1
NDCG
MRR
Weak
Majority
0.611
0.379
0.516
0.340
0.530
0.346
0.139
0.025
Baselines
Random
0.486
0,480
0.014
0.439
0.492
0.492
0.503
0.502
0.101
0.003
Neural
LSTM
0.737
0.702
0.825
0.036
0.767
0.763
0.779
0.773
Networks
Pretrained
BERT
0.754
0.720
0.854
0.027
0.791
0.790
0.808
0.806
Language
XLNet
0.758
0.736
0.853
0.030
0.785
0.784
0.814
0.808
Models
Hate
0.754
0.739
0.865
0.026
BERT
GAT
0.758
0.731
0.838
0.032
0.784
0.784
0.796
0.790
0.184
0.076
Graph
HAN
0.762
0.740
0,843
0.030
0.785
0.784
0.807
0.803
0.193
0.087
Models
HGT
0.760)
0.744
0.841
0.031
0.793
0.791
0.800
0.798
0. 196
0.087
ours
ours
0.773
0.755
0.854
0.027
0.809
0.807
0.826
0.825
0.205
0.095
Table 1: Comparison results of our models and baselines. (average of runs). We report ACC(aceuracy) and Fl (mmcro-ff)) for NORMVIO, Ruddit and Rnxiety-on-Reddit, r(Pearson's R) and MSE(Mcan Squared Error) for Ruddit, NDCG(Normalized Discounted Cumulative Gain) and MRR(Mean Reciprocal Rank) for Community Recommendation.
NORMVIO (Park et al., 2021) 1) is a new dataset focusing on more complete spectrum of com- munity norms and their violations. It includes Reddit comments which are removed for vio- lating community norms. We obtain the origi- nal comments based on IDs provided by the author and reconstruct the task as a binary classification to determine whether a vio- lation comment.
Anxiety-on-eeddit (Shen and Rudzicz, 2017) is a dataset to study anxiety disorders through personal narratives collected through Reddit. In this dataset, anxicty posts come from sev- eral anxiety-related subreddits. Due the unavailability of original data information. we rebuild a dataset similar to Anxiety-on- Reddit. Concretely, we crawl the data fol- lowing the data collection process described in paper. The time period is from January 2018 to March 2018. The label of text is deter- mined by the community to which they belong. Therefore, model this task community- text edge prediction task. When making pre- dictions, we block the text community edge prevent label leakage.
Ruddit (Hada et al., 2021) is the first dataset of English language Reddit comments that has fine-grained, real-valued scores between -1 (maximally supportive) and (maximally offensive). It is annotated with Best-Worst Scaling, a form comparative anmotation that has been shown to alleviate known biases of using rating scales.
Recommend community for users. The ciological theory of homophily asserts that individuals are usually similar to their friends (McPherson et al., 2001). People are more likely to share with their friends rather than random individuals (Yang and Eisenstein, 2017). So we design downstream task for recommending communities to users to help users find friends with similar interests or pur- poses. Concretely,
Dreaddit (Turcan and McKeown, 2019) is Reddit dataset for stress analysis in social me- dia. Stress is common in daily life, partic- ularly in online world. Too much stress is related to physical and mental health (Lupien et al., 2009; Calcia et al., 2016). Therefore it is very important to recognize stress. Dreaddit consists of posts from five different categories of Reddit communities.
crawl the data in Jan- uary 2018 and built them into graph de- scribed in 2.1. Recommending communities task modeled to predict which community will have an edge connection with the targeted
Edge-related Downstream Tasks

---

user. When making predictions, block the text community edge prevent label leakage.
which is designed to address the problems of long- term dependencies in sequential data. Pre-trained Language Models. BERT (Devlin et al., 2018) and XLNet (Yang et al., 2019) are two pre-trained language models that leverage transformer-basee architectures and are pre-trained in general corpus. HateBERT (Caselli et al., 2020) is re-trained BERT model for abusive language detection in English. We only test HateBERT in NORMVIO and Ruddit because only these two datasets are related to abusive language. Base- line pre-trained language models are continued pre- training on the collected Reddit data. We don't ' test pre pre-trained language models in community recom- mendation task because this task is not related to texts.
3 Experiment
3.1 Data preprocessing
Our pre-training is built on Reddit data of the whole year 2017. After data sampling introduced in 2.1.1, we perform data preprocessing on the text nodes. We first remove the special symbols contained in text and then replace hyperlinks and user mentions with [URL] and [ @ USERJ. Finally, we keep only the first 510 words for too long text.
There are too many nodes in this graph, so is difficult to train the entire graph. We sample subgraphs for training by HGSampling algorithm (Hu et al., 2020b). The above sampling methods guarantee a high connectivity between sampled nodes and preserve maximal structural information. Following GPT-GNN's approach, Adaptive Queue (Hu et al., 2020a) is used to alleviate the problem that r, Edge only has access to the sampled nodes in a subgraph.
Graph-based Models. GAT (Veličković al., 2017), HAN (Wang al., 2019) and HGT (Hu et al., 2020b) are three graph-basek models which can capture structural information in data. XLNet is used to initialize the node representation similar to 2.2.1.
3.4 Experiment Result
3.2 Implementation Details
Node-related Tasks. Our model shows very good performance on node-related tasks as shown in Ta- ble 1, which indicates combining the structure of social media can promote text comprehension. Our model performs worse than HateBERT on Ruddit. we think it is due to the small size of Ruddit and the lack of concentration of communities and authors, which means there are only a few texts coming from the same user or community.
We set sample depth to 6 and sample width to 256 for sampling subgraphs. The head number and layer number of HGT are set as and 3. Dur- ing pre-training, we set the decoder Dec and Decgdge() as multi-layer perception and distance function as cosine similarity. The size of adaptive queue is set as 256. The hidden dimension d = 768 (same as the output dimension of pre-trained lan- guage model) is set for all modules. We optimize the model via the AdamW optimizer (Loshchilov and Hutter, 2017) with the Cosine Annealing Learn- ing Rate Scheduler (Loshchilov and Hutter, 2016). We pre-train the model with 5 epochs to get the final parameters of graph network. The codes are implemented by Pytorch and PyTorch Geomet- ric(PyG). The entire pre-training procedure con- sumes a lot of memory, so it is conducted on only GeForce RTX 3090 GPU.
Edge-related Tasks. Our model also performs well on edge-related tasks as shown in Table 1. This means our model can effectively learn the structural information contained in the graph during the pre- training stage.
Further Analysis
4.1 Ablation Study
We conduct ablation studies to explore the effects of different components. Results are reported in Ta- ble 2. We divide datasets into text-related and user- related, and we remove user information and text in- formation respectively. We also analyze the impact of pre-training module. As shown in Table 2, struc- tural information is useful to better model texts, and semanties can also assist in structural learning. Additionally, pre- mee-trinnig has been proven to be useful.
3.3 Compared Models
Weak Baselines. Majority generates the label with the highest proportion and Random generates ran- dom labels. Majority is not used in Ruddit because the label in Ruddit continuous.
Neutral Models. LSTM (Hochreiter and Schmid- huber, 1997) is a type of recurrent neural network

---

Text-related Tasks
User-related Tasks
Anxicty-
Community
NORMVIO
Ruddit
Dreaddit
on-Reddit
Recommendation
ACC
F1
r
MSE
ACC
F1
ACC
NDCG
MRR
ours
0.773
0.755
0.854
0.027
0.809
0.807
0.826
0.825
ours
0.205
0.095
w/o user
0.757
0.731
0.841
0.033
0.791
0.790
0.798
0.796
w/o text
0.181
0.072
w/o
w/o pre-training
0.760
0.744
0.841
0.031
0.793
0.791
0.800
0.798
0.196
0.087
pre-training
Table 2: Results of ablation studies on text-related tasks and user-related Tasks
4.2 Result Understanding
Why community information help? As Table 1 shows, graph- models which utilize commu- nity insights perform better than language models on some datasets. Take NORMVIO as an exam- ple, NORMVIO is a dataset composed of com- ments which violate their community's 's norm and are deleted by moderators of community. In Reddit, the rules moderators enforce vary widely both in their formulation and interpretation across commu- nities, making one-size-fits-al approach increas- ingly brittle.
of
CAN
6.0
(a)
(b)
Canmar memide miltio
Protuabllty by Demity
We analyze the subreddits that show improve- ment after introducing community information and find that samples misclassified by language models in these subreddits usually due to vi- olating community-specific norms which are dif- ferent from common norms as shown in Fig- ure ((a). For example, the graph-based model has learned that hyperlinks are likely to be considered norm-violating in rlgirlsinyogapants. rCCanadaPolitiess has stricter civilization require- ments, and r/changemyview has stricter formatting requirements.
detro Hong
Velue
ROA Diw
(C)
(d)
Figure 3: (a) Subreddits with improved results after adding community information (b) Statistics users and their text labels (c) Probability density distribution of cosine similarity between users and their interested not interested subreddits (d) Subreddit visualization
Why user nodes help? User nodes can provide structural information by connecting two commu- nity nodes or two text nodes, and they are also asso- ciated with ground truth. By analyzing two classifi- cation text-related datasets NORMVIO and Dread- dit, we find that among users whose text quantity is greater than 1, the majority of their texts have only one text label. For example, as shown in Figure 3(b). in NORMVIO, there are 4541 users who have more than at least two comments in the dataset. 3300 of them have only one text label. Among the remaining 1241 users with two text labels, the proportion of the major label averaged by users is 56.05%. This proportion is shown in Figure 3(b) denoted by #
believe that active users have closer semantic rep- resentation the communities they are interested in. We get the semantic representation of commu- nity and user by averaging their texts' embedding produced by Xetet. Then we draw the probability density distribution of cosine similarity between users and their interested & not interested subred- dits. As shown in Figure 3(c), active users have closer semantic representation to the communities they are interested in. This can partially explain why text nodes work.
Why pretrain help? We analyze the changes in evaluation metrics of validation set during the first five epochs when incorporating pre-training and not incorporating pre-training. As shown in Figure 4, pre- pre-training allows the model to obtain better ini-
Why text nodes help? Tabel 2 has shown the im- provement brought by text nodes. Intuitively, we

---

NRRW10
Rudiit
Preudiit
Ave clyaanReelii
Comsnity Roommindaaion datiion
5.50
$1.7
5.00
Figure 4: Improvement brought by the pre-trained model during the first epochs on datasets
tialization, leading improved performance and faster convergence during the initial stages of train- ing.
of connection-based solutions is text in social media.
Our research uses both textual and user informa- tion, which allows for more comprehensive and complete modeling of online community.
4.3 Community Visualization
We perform Principle Component Analysis (PCA) subreddits' embedding generated in pre-training process(Figure 3(d)) Back to the example in 1. we can intuitively see that r/fantasyfootball and r/BobsBurgers show higher cosine similarity than r/fantasyfootball and B/rgers
Conclusion
In this paper, we propose unified graph pre- training model for online community modeling. contains communities, users, and textual content simultancously and can be casily used in various downstream tasks related to social media. Exten- sive experimental results validate the effectiveness and generalization capability of our approach.
5 Related Work
5.1 Pretrained Language Model in Social Network
Limitations
2017, Google's team completely abandoned the traditional network structure, and only used the at- tention mechanism for machine translation tasks (Vaswani al., 2017). In 2018, BERT performed well many tasks with its self-supervised pre- training tasks and deep bidirectional models (De- vlin et al., 2018). Subsequently, BERT and its vari- ants have also been widely used in social network problems such as BERTweet (Nguyen et al., 2020), PoliBERTweet (Kawintiranon and Singh, 2022), UPPAM (Mou et al., 2023), and MentalBERT (Ji
Our work aims to consider the various components and relationships within online communities as much as possible. The vast amount of data in on- line communities necessitates customized designed self-supervised tasks, especially focusing on reduc- ing memory use and enhancing pre-training effi- ciency. Our model has a high memory usage due to the large volume of text and the use of pre- pre-trained language models, leaving much room for further exploration.
et al., 2021).
Ethics Statement
5.2 Methods in Community Profiling
8.1 Data Use
Connection- based and content-based solutions are two mainstream ways of modeling online commu- nity. Connection-based solutions (Mishra et al., 2018; Arviv et al., 2021; Kumar et al., 2018; Mar- tin, 2017; Janchevski and Gievska, 2019) are based the idea that similar communities house sim- ilar users. The core focus of connection-bast based solutions is on users. Content-based solutions (Strzalkowski et al., 2020; Trujillo et al., 2021; Khalid et al., 2022) usually believe similar com- munities have similar semantics. The core focus
Data Source Compliance Data collection requires ethical reflection and precautions to preserve the dignity and privacy of users. Our experimental data comes from Pushshift. Pushshift is Reddit data repository and now it is open for subreddit moderators who have Pushshift access. What's more, Pushshift is actively working with Reddit find a solution to provide scholars with data access. Pushshift has provisions to delete data from the dataset upon user's request.

---

Data Release We release data in a manner that complies with the General Data Protection Regula- tion (GDPR). We don't 't provide any user-specific information. We release only the comment IDs and submission DDs. Reddit's s Terms of Service do not prohibit the distribution of IDs. The researchers using the dataset need to retrieve the data with Pushshift access.
8.3 Bias Alleviating
Text on social media can incorporate unintentional biases, so our model may inevitably be biased. To alleviate these problems, we will incorporate bias removal in future work to make our models fairer. In particular, mitigating bias can be approached from three aspects: cleaning data, adjusting mod- els, and assessing the bias level in pre-trained mod- els. Data cleaning can be done by filtering biased communities (Ferrer et al., 2021) and texts (Hube et al.). Previous research on adjusting the model focuses on how to involve bias reduction in the model structure (Xia et al., 2020; Mozafari et al., 2020). There have been some methods to evaluate the degree of bias of pre-trained models (Kurita et al., 2019; Field and Tsvetkov, 2019), and use them evaluate our models.
Personal Privacy We focus on protecting personal privacy throughout the entire research. Users are anonymous in data. Text-user edge generation used in pre-training does not lead to problems of indi- vidual identification. This task is achieved through negative sampling and contrastive learning, which cannot be directly used to identify individuals in reality. We won't 't release any user-specific informa- tion.
Acknowledgements
8.2 Societal Impacts
This work is supported by National Natural Sci- ence Foundation of China (No. 6217020551 and No. 62206056) and Science and Technol- ogy Commission of Shanghai Municipality Grant (No.21QA1400000)
Dual Use of Approaches The correct use of our research will have a positive impact social net- works by solving social problems without any stance or commercial purpose. But we also accept that it may have adverse effects on society such as leaking personal data, violating personal pri- vacy, threatening users' right to freedom of speech, and being abused to achieve personal interests, etc. Our proposed model will be shared selectively and subject to the approval of the Institutional Review Board (IRB), guaranteeing that any research or application based on this study is for research pur- poses only, Any attempt to use the proposed model to infer user characteristics and violate user privacy is strictly prohibited.
References
Eyal Arviv, Simo Hanouna, and Oren Tsur. 2021. It'sa thin line between love and hate: Using the echo in modeling dynamies of racist online communities. In Proceedings of the International AAAI Conference on Web and Social Media, volume 15, pages 61-70.
Nancy K Baym. 2000. Time in, log on: Soaps, fandom, and online community, volume 3. Sage.
Marilia A Calcia, David R Bonsall, Peter S Bloomfield. Sudhakar Selvaraj. Tatiana Barichello, and Oliver D Howes. 2016. Stress and neuroinflammation: a sys- tematic review of the effects of stress microglia and the implications for mental illness. Psychophar- macology, 233: 1637-1650.
Community Recommendation Community rec- ommendation (Janchevski and Gievska, 2019; Sun- daresan et al., 2014) quickly meet users' social needs and improve users' online experience. But it may indeed cause some potential social harm such as the leakage of personal data, the invasion of personal privacy by studying personal character- istics, and user disturbance caused by inaccurate or excessive recommendations. In order to mitigate the potential harm, the system can develop more fine- fine-grained methods to learn the user's interest, such as using degree of belonging instead of 0/1 for recommendations. And it is necessary to fully respect the user's wishes. If the user does not want to be recommended, the system will not read any his historical data.
Tommaso Caselli, Valerio Basile, Jelena Mitrović, and Michael Granitzer. 2020. Hatebert: Retraining bert for abusive language detection in english. arXiv preprint arXiv:2010. 12472.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. 2018, Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint 10X.v:8810.08800
Xavier Ferrer, Tom van Nuenen, Jose M Such, and Na- talia Criado. 2021. Discovering and categorising language biases in reddit. In Proceedings of the Inter- national AAAI Conference on Web and Social Media, volume 15, pages 140-151.

---

Anjalic Field and Yulia Tsvetkov. 2019. Entity- centric contextual affective analysis. arxiv preprint htl::1906.07666
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprini urXivv1907.11000
Rishav Hada, Sohi Sudhir, Pushkar Mishra, Helen Yannakoudakis, Saif M Mohammad, and Ekate- rina Shutova. 2021. Ruddit: Norms of offensive- ness for english reddit comments. arXiv preprint arXiv:2106.0555 006.05664.
llya Loshchilov and Frank Hutter. 2016. Sgdr: Stochas- tic gradient descent with warm restarts. arXiv preprint arXiv:
Caroline Haythorntwaate. 2007. Social networks and online community. The Oxford handbook of Internet psychology. pages 121-137.
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint arXiv:1711.05101.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):
Sonia J Lupien, Bruce S McEwen, Megan R Gunnar, and Christine Heim. 2009. Effects of stress through- out the lifespan on the brain, behaviour and cognition. Nature reviews neuroscience, 10(6):434445.
1780.
Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang. and Yizhou Sun. 2020a. Gpt-gnn: Gener- ative pre-training of graph neural networks. Pro- ceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1857-1867.
Trevor Martin. 2017. community2vec: Vector represen- tations of online communities encode semantic rela- tionships. In Proceedings of the Second Workshop on NLP and Computational Social Science, pages 27-31.
Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. 2020b. Heterogeneous graph transformer. In Proceedings of the web conference 2020, pages 2704- 2710.
Miller McPherson, Lynn Smith-Lovin, and James Cook. 2001. Birds of a feather: Homophily in social networks. Annual review of sociology. 27(1):415- 444.
Christoph Hube, Robert Jäschke. and Besnik Fetahu. Towards bias detection online text corpora, Jo Bates Paul D. Clough Robert Jäschke, page 19.
Pushkar Mishra, Marco Del Tredici, Helen Yan- nakoudakis, and Ekaterina Shutova. 2018. Author profiling for abuse detection. In Proceedings of the 27th international conference computational lin- guistics, pages 1088-1098 Association for Compu- tational Linguistics (ACL).
Andrej Janchevski and Sonja Gievska. 2019, A study of different models for subreddit recommendation based on communitt interaction. In ICT Innovations 2019. Big Data Processing and Mining: 11th Inter- national Conference, ICT Innovations 2019, Ohrid, North Macedonia, October 17-19. 2019, Proceed- ings 11, pages 96-108. Springer.
Xinyi Mou, Zhongyu Wei, Qi Zhang, and Xuan-Jing Huang. 2023. Uppam: A unified pre-training ar- chitecture for political actor modeling based on lan- guage. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Vol- ume 1: Long Papers), pages 11992-12012
Shaoxiong Ji, Tianlin Zhang, Luna Ansari, Jie Fu, Prayag Tiwari, and Erik Cambria. 2021. Mentalbert: Publicly available pretrained language models for mental healthcare. arXiv preprint arXiv:21
Marzich Mozafari, Reza Farahbakhsh, and Noël Crespi. 2020. Hate speech detection and racial bias mitiga- tion in social media based on bert model. PloS one, e023776..
Kornraphop Kawintiranon and Lisa Singh. 2022. Polib- ertweet: a pre-trained language model for analyzing political content on twitter. In Proceedings of the Thirteenth Language Resources and Evaluation Con- ference, pages 7360-7367.
Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. Bertweet: pre-trained language model for english tweets. arXiv preprint arXiv:2005. 10200.
Osama Khalid, Jonathan Rusert, and Padmini Srini- vasan. 2022. Suum cuique: Studying bias in taboo de- tection with a community perspective, arXiv preprint arXiv:2203.1 1440
Chan Young Park, Julia Mendelsohn, Karthik Radhakr- ishnan, Kinjal Jain, Tushar Kanakagiri, David Jur- gens, and Yulia Tsvetkov. 2021. Detecting commu- nity sensitive norm violations online conversations. arXiv preprint arXiv:
Srijan Kumar, William I. Hamilton, Jure I Leskovec, and Dan Jurafsky. 2018. Community interaction and con- flict on the web. In Proceedings of the 2018 world wide web conference, pages 933-943.
Judy Hanwen Shen and Frank Rudzicz. 2017. Detecting anxiety through reddit. In Proceedings of the Fourth Workshop Computational Linguistics and Clini- cal Psychology- FFom Linguistic Signal to Clinical Reality, pages 58-65.
Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Measuring bias in con- textualized word representations. arXiv preprint arXiv:7900.03337.

---

Tomek Strzalkowski, Anna Newheiser, Nathan Kemper. Ning Sa, Bharvee Acharya, and Gregorios Katsios. 2020. Generating ethnographic models from com- munities' online data. In Proceedings of the Second Workshop on Figurative Language Processing, pages 165-175.
A
Appendix
A.1
Pretrain Dataset Description
The data description of pre-training dataset is shown in Table 3
Dataset
Vishnu Sundaresan, Irving Hsu, and Daryl Chang. 2014. Subreddit recommendations within reddit communi- ties.
# of total communities
5126
# of total users
987945
# of total texts
1887178
Milo Z Trujillo, Samuel F Rosenblatt, Guillermo de Anda Jáuregui, Emily Moog. Briane Paul Sam- son, Laurent Hébert-Dufresne, and Allison M Roth. 2021. When the echo chamber shatters: Examin- ing use of subreddit ban. arXiv preprint arXiv:2106. 16207.
# of total community-use user edges
8701250
# of total community-ters edges
1887178
# of total user-text edges
1887178
language post-
Table 3: Summary statistics of pre-training datasets
Elsbeth Turean and Kathleen McKeown, 2019. Dread- dit: reddit dataset for stress analysis in social me- dia. arXiv preprint 191.0339.
A.2 Downstream Dataset
We crawl the data based on the IDs for published datasets. Because some data has been deleted by the original author, we usually cannot obtain a com- plete dataset. For unpublished datasets, we rebuild the dataset ourselves according to the data collec- tion method in paper. For published data without validation set, we divide one-fifth of the training set into validation set. For the dataset built ourselves, divide the dataset into training set, validation set and test set as 70:15:15, The statistics of downstream dataset is shown in Table 4.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention all need. Advances in neural information processing systems, 30.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710. 10903.
Isaac Waller and Ashton Anderson. 2019. Generalists and specialists: Using community embeddings to quantify activity diversity online platforms. In The World Wide Web Conference, pages 1954-1964.
Target # of Target
Objects
Objects
NORMVIO
Text
44643
Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. 2019. Heterogeneous graph attention network. In The world wide web conference, pages 2022-2032.
Ruddit
Text
5937
Dreaddit
Text
3517
Anxiery-on-Redee
Text
30920
Community
User
242840
Mengzhou Xia, Anjalie Field, and Yulia Tsvetkov. 2020. Demoting racial bias hate speech detection. arXiv preprint arXiv:2005.1 12246.
Recommendation
Table 4: Summary statistics of downstream datasets
Yi Yang and Jacob Eisenstein. 2017. Overcoming lan- guage variation in sentiment analysis with social at- tention. Transactions of the Association for Compu- lational Linguistics, 5:295-307.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ R Salakhutdinov, and Quoe Le. 2019. Xlnet: Generalized autoregressive pretraining for lan- guage understanding. Advances in neural informa- tion processing systems, 32.
Jiawei Zhang, Haopeng Zhang, Congying Xia, and Li Sun. 2020. Graph-bert: Only attention is needed for learning graph representations. arXiv preprint arXiv:2001.051

---

A.3 Training Details
We provide our parameters in order to quickly ap- ply the pre-trained model to downstream tasks. Be- fore formal pre-training, we sample 10% of the data to help select appropriate parameters because pre-training on all data is price-expensive and time- consuming.
Pre-training
À # of Sampled Negative Edges
0.5
255
Queue Size
256
Sample Depth
6
Sample Width
128
Hidden Dimension
768
Attention Head
GNN Layers
Dropout
0.2
Gradient Norm Clipping
0.5
Maximum Learning Rate
1e-3
Fine-tuning
Sample Depth
Sample Width
128
Gradient Norm Clipping
0.5
Learning Rate
5e-4
Dropout
0.2
Table 5: Training details


Reviews:
[{"review_id": "4MbKZ4Ikcb", "reviewer": "Reviewer_bxaU", "summary": "", "questions": "A. Have you considered how Deep Graph Infomax for graph-based pre-training differs from your framework? I believe some comparisons there would be useful in showing the theoretical benefits of your work, particularly since the baselines are not pre-trained either.\nB. Did the authors finetune the baselines directly on the downstream tasks? Or did they pre-train the baselines on their pre-training dataset and then fine-tune them to get comparable results?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. It may be premature to make the claims about the benefit of this paper.\n   \n    Response: This paper focuses on the online community in Reddit and selects five datasets from Reddit as downstream tasks.  These five datasets are not only broad in type, as they can be divided into node-related&edge-related and text-related&user-related, but also rich in content, including popular tasks on Reddit in recent years. So we believe that these datasets are representative, and the experimental results on these datasets can represent the effectiveness of our model in modeling the Reddit online community.\n\n2. The primary research question and the problem the authors are trying to address is unclear in the paper.\n   \n    Response: The main problem that this paper wants to address is how to better model online communities, and then use the communities to provide assistance for social media related tasks. Relevant studies have proved that using community information can assist tasks such as abusement detection[1] and conflict analysis[2]. This article aims to provide help on how to make better use of communities. Specifically, we consider community, user, and text simultaneously, instead of only considering community and user[3-5] or community and text[6-7], and we use graph structure to simulate the relationship between these three items, instead of simply adding community's name[1] after text or concatenated community's vectors after text's vectors. Finally, we also use the rich historical data on social media to pretrain a pre-trained model, so that it can be more easily to adapt to multiple downstream tasks.\n\n3. About baselines.\n\n   Response: We have continued pre-training the baseline pre-trained language models on the collected historical Reddit data, and the epoch is set to 2.\n\n4. About DGI.\n\n   Response: DGI relies on maximizing mutual information between patch representations and corresponding high-level summaries of graphs—both derived using established graph convolutional network architectures.  It is not suitable for our task, mainly for the following reasons.\n\n   (1) The graph we constructed is heterogeneous, and DGI does not have any additional advantages at the heterogeneous graph level.\n\n   (2) The construction of DGI's negative samples and global information has a significant impact on the results. \n\n   (3) In social networks, we believe that \"close\" nodes should also be more \"close\" in the representation space, as they usually have similar features in reality. So random walk is suitable.\n\nReferences:\n\n[1] Chan Young Park, Julia Mendelsohn, Karthik Radhakrishnan, Kinjal Jain, Tushar Kanakagiri, David Jurgens, and Yulia Tsvetkov. 2021. Detecting community sensitive norm violations in online conversations. arXiv preprint arXiv:2110.04419.\n\n[2] Marilia A Calcia, David R Bonsall, Peter S Bloomfield, Sudhakar Selvaraj, Tatiana Barichello, and Oliver D Howes. 2016. Stress and neuroinflammation: a systematic review of the effects of stress on microglia and the implications for mental illness. Psychopharmacology, 233:1637–1650.\n\n[3] Srijan Kumar, William L Hamilton, Jure Leskovec, and Dan Jurafsky. 2018. Community interaction and conlict on the web. In Proceedings of the 2018 world wide web conference, pages 933–943.\n\n[4] Trevor Martin. 2017. community2vec: Vector representations of online communities encode semantic relationships. In Proceedings of the Second Workshop on NLP and Computational Social Science, pages 27–31.\n\n[5] Isaac Waller and Ashton Anderson. 2019. Generalists and specialists: Using community embeddings to quantify activity diversity in online platforms. In The World Wide Web Conference, pages 1954–1964.\n\n[6] Tomek Strzalkowski, Anna Newheiser, Nathan Kemper, Ning Sa, Bharvee Acharya, and Gregorios Katsios. 2020. Generating ethnographic models from communities’ online data. In Proceedings of the Second Workshop on Figurative Language Processing, pages 165–175.\n\n[7] Milo Z Trujillo, Samuel F Rosenblatt, Guillermo de Anda Jáuregui, Emily Moog, Briane Paul V Samson, Laurent Hébert-Dufresne, and Allison M Roth. 2021. When the echo chamber shatters: Examining the use of community-specific language post-subreddit ban. arXiv preprint arXiv:2106.16207.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The work has great potential application in merging both the structural (community user graphs) and semantic (text content) information for a social network analysis - particularly in computational social science.\n2. A good analysis of downstream tasks, and a strong improvement upon the author's chosen baselines.\n3. The discussion regarding why each part of the model (text node, user node) is helpful which is a strong point of this paper.", "reasons_to_reject": "1. In line 093 the authors identify that \"extensive experiments have to be completed to confirm the effectiveness of our frameworks\" indicating that it may be premature to make the claims about the benefit of this paper. \n2. The primary research question and the problem the authors are trying to address is unclear in the paper. They mention in the abstract that their study will \"provide assistance on how to better model online community\", but they are unclear exactly how. \n3. Another weakness of this paper is in its experimental setup and experiments. The chosen baselines are justified, however, I am unsure about the process in which said baselines were used. Did they finetune the baselines directly on the downstream tasks? Or did they pre-train the baselines on their pre-training dataset and then fine-tune to get comparable results? I am including this weakness as a question to the authors.\n4. Deep Graph Infomax (DGI) exists as a well-established graph-based pre-training approach (node representation tasks only) - the pre-training framework suggested by the author while well thought out and developed would be much better established in comparison to DGI. (Included in missing references). ", "questions_to_authors": "A. Have you considered how Deep Graph Infomax for graph-based pre-training differs from your framework? I believe some comparisons there would be useful in showing the theoretical benefits of your work, particularly since the baselines are not pre-trained either.\nB. Did the authors finetune the baselines directly on the downstream tasks? Or did they pre-train the baselines on their pre-training dataset and then fine-tune them to get comparable results?", "paper_topic_and_main_contributions": "This paper proposes a pre-training framework to leverage the structural strength of heterogeneous graphs and the semantic strength of textual content (in the context of Reddit) to enable downstream tasks for community-related analysis. Primary contributions as per the author are (1) the proposed pre-training framework, (2) the graph-based approach, and (3) an analysis of downstream tasks on social media.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "1. Veličković, P., Fedus, W., Hamilton, W. L., Liò, P., Bengio, Y., & Hjelm, R. D. (2018). Deep graph infomax. arXiv preprint arXiv:1809.10341.", "Typos_Grammar_Style_and_Presentation_Improvements": "1. line 074 `facilities`  -> `facilitates`\n2. Section 5 `Releted`-> `Related` (other instances as well)"}, "extra_text": {}}, {"review_id": "w9AsTqhF6a", "reviewer": "Reviewer_2SWw", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Not sure how well the proposed method is applicable to other datasets/problems.\n   \n    Response: This paper focuses on the online community in Reddit and selects five datasets from Reddit as downstream tasks.  These five datasets are not only broad in type, as they can be divided into node-related&edge-related and text-related&user-related, but also rich in content, including popular tasks on Reddit in recent years. So we believe that these datasets are representative, and the experimental results on these datasets can represent the effectiveness of our model in modeling the Reddit online community.\n\n2. Not effective for some online communities in which enough amount of texts is not available.\n   \n    Response: For a community without a lot of text information, although it cannot obtain a good representation in the initialization stage, the model will provide additional information through communities with the same users during the training process, which is very friendly for communities with few texts. \n\n3. Although three different types of nodes are considered, they are eventually initialized based on texts. So, they might eventually make contributions in the same way by adding more textual information, rather than they actually provide structural/community information. There needs other experiments to verify it.\n   \n    Response: Although all three types of nodes are initialized through text, they not only contribute by supplementing additional text information. During pre-training, predicting the structure is a pre-training task which can enable the pre-trained model to learn structural information from graph. So the pre-trained model contains structural information. \n   \n    Our baseline pre-trained language models have been further pre-trained on the collected Reddit historical data, with epoch set to 2. And they are not as effective as our model. This indicates that structural information is useful in pre-trained models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Given the complexity and size of the online community data like Reddit, the proposed is able to effectively learn the representations of different types of entities.\n* The experiments were done with various baselines/tasks along with ablation study, which cover various aspects of the community modeling.\n* Overall the paper is easy to read and follow although it needs to improve presentation a little bit.", "reasons_to_reject": "* Not sure how well the proposed method is applicable to other datasets/problems. It seems to heavily rely on the textual information, and thus the proposed method might not be effective for some online communities in which enough amount of texts is not available.\n* Lack of technical novelties. How to initialize the nodes and the objectives used to optimize them have been commonly used in previous works.\n* Although three different types of nodes are considered, they are eventually initialized based on texts. So, they might eventually make contributions in the same way by adding more textual information, rather than they actually provide structural/community information. There needs other experiments to verify it.\n* There is no connection between the same type nodes, and ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a pre-training framework for a graph of online community where there exist heterogeneous nodes types such as users, communities, and texts. It first constructs a graph from online communities data and initializes the node embeddings by using textual information. Then, the nodes are given to a GNN model to leverage the network structural information for better node representations. It uses two different self-supervised tasks, text reconstruction and edge generation for pre-training. In order to demonstrate the learned node representations are useful, the authors did experiments with downstream tasks and show the proposed method can achieve better performance across different datasets, compared to the baselines.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 237, 260: releted -> related"}, "extra_text": {}}, {"review_id": "KBBcX5lh7s", "reviewer": "Reviewer_equG", "summary": "", "questions": "If using XLNet to extract text representation and feed it to HGT to encode the structure of social networks, will do these operations have the same effect as pre-training a neural network?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. If using XLNet to extract text representation and feed it to HGT to encode the structure of social networks, will do these operations have the same effect as pre-training a neural network?\n   \n    Response: Our baseline graph models use XLNet to extract text representations and feed them to graph neutral network(such as HGT)  to encode the structure of social networks. They are not as effective as our model. This indicates that pre-training a neural network is useful. What's more, we compare the experimental results with and without pre-training, as shown in Chapter 4.3(Why pretrain help). The only difference between the two curves is whether they have been pre-trained, and the other experimental steps are exactly the same.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The paper has some nice ideas and some interesting insights that may be of benefit to the social network modeling community. And good writing makes this paper easy to follow.\n\n(2) Better performance than baselines on five real-world datasets. The proposed framework achieves significant improvement on five datasets for two downstream tasks over several strong baselines, which shows the effectiveness of the framework.\n\n(3) Extensive empirical analysis of the framework. The authors conduct a series of ablation studies to further prove the effectiveness of the framework.\n", "reasons_to_reject": "(1) The novelty of this paper is a little limited. The basis models of this framework are HGT and XLNet, and the pre-trained tasks are node generation and edge generation, all of them are proposed by other research work. It seems this framework just stacks these models together. \n\n(2) The fundamental reason for the improvement in model performance has not been clearly stated. It lacks experimental results to prove whether the improvement in performance comes from the design of a new framework or the integration of multiple features. \n", "questions_to_authors": "If using XLNet to extract text representation and feed it to HGT to encode the structure of social networks, will do these operations have the same effect as pre-training a neural network?\n", "paper_topic_and_main_contributions": "This paper primarily focuses on the online community modeling area and proposes a unified graph pre-training model as the basis to apply to downstream tasks. The authors think previous research hasn’t fully utilized the information communities, users, and texts in social media. Therefore, they construct a ternary heterogeneous graph to comprehensively model the social network and propose a text generation task and the edge generation task to pre-trained an HGT network. The authors conducted extensive experiments on five Reddit-related datasets for hate speech detection and stress analysis tasks, demonstrating that the proposed model outperforms pre-trained language models such as BERT and XLNet, and some graph models.\n", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5wIhHlLazR", "reviewer": "Reviewer_uL1E", "summary": "", "questions": "1. The overall pretraining objective involves a hyperparameter \\lambda. Considering that pretraining is costly, how do you determine the value of \\lambda before starting pretraining?\n\n2. \"The entire pre-training procedure consumes a lot of memory, so it is conducted on only one GeForce RTX 3090 GPU\", in this case, shouldn't multiple GPU cards be more helpful?\n\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Take into account  subreddits which have only a few posts.\n    \n    Response: Subreddits with few posts are rarely involved in Reddit related tasks, so we have not considered them.\n\n2. Why use two matrices.\n    \n    Response: These two matrices are transposed matrices, which are used to illustrate that there are two types of edges between two nodes. For example, between the user node and the text node, there are two kinds of edges: \"user-write-text\" and \"text-from-user\".\n\n3. whether the pre-trained model could generalize to datasets from other sources such as Twitter, Facebook.\n    \n    Response: The pre-trained model is pre-trained on the Reddit dataset, so it is difficult to adapt to other social platforms. But the whole framework of modeling online communities can be generalized to other social media. The only difference is that on social platforms such as twitter, there is no strict concept of community. So we need to artificially set the scope of the community. For example, consider a hashtag discussion topic as a community.\n\n4. how to determine the value of  $\\lambda$?\n    \n    Response: Pre-training is price-expensive and time-consuming. Before formal pre-training, I sampled 10% of the data for pre-training to help me select appropriate parameters.\n\n5. Conducted on only one GeForce RTX 3090 GPU.\n    \n    Response: This is the shortcoming of my experiment, because the constructed graph is too large, and my pre-training method needs to read their initialization vectors into memory at one time, almost occupying the entire memory.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is in general well-written and easy to follow.\n\n2. Pretraining for online community modeling is of practical value and is worthy of study.\n\n3. The proposed pretraining framework is simple yet effective, as verified by the downstream experiments.", "reasons_to_reject": "1. The pretraining dataset is relatively small, it contains only 5K communities and less than 1M users. This work considers only the top 2% of subreddits that have a large number of posts. However, in practice, many subreddits would have only a few posts, these subreddits (or communities) should also be taken into account.\n\n2. In the proposed framework, each text is treated as a node in the heterogeneous graph, which could lead to a large number of nodes, which limits the utilization of the framework. Besides, it is unclear why treating text is a good choice, are there other approaches to modeling text? For example, treating textual information as user node features.\n\n3. In section 2.1.2, each relation (edge type) is represented by two matrices (A, B, C, D, E, F). Based on my understanding, the two matrices are the same. Can you explain why two matrices are used? It also seems that many matrices are not used in the following text.\n\n4. The pretraining tasks only consider nodes corresponding to text content and edges that have text as one node. This ignores community nodes and user (author) nodes and the relationship between the community node and user node.\n\n5. All five downstream datasets are related to Reddit. It is not sure whether the pre-trained model could generalize to datasets from other sources such as Twitter, Facebook.\n", "questions_to_authors": "1. The overall pretraining objective involves a hyperparameter \\lambda. Considering that pretraining is costly, how do you determine the value of \\lambda before starting pretraining?\n\n2. \"The entire pre-training procedure consumes a lot of memory, so it is conducted on only one GeForce RTX 3090 GPU\", in this case, shouldn't multiple GPU cards be more helpful?\n\n", "paper_topic_and_main_contributions": "This paper presents a unified pretraining framework involving communities, users, and textual content simultaneously. Graph-based approaches are employed to model the relationship between these three elements. Two self-supervised tasks including attribution generation and link generation are used to pre-train this ternary graph, making it easy to adapt to multiple downstream tasks related to social media. The pretraining data were constructed from Reddit and downstream task datasets are also Reddit-related. Experimental results show that pretraining can help achieve better results in 4 out of 5 downstream tasks. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "013, Attrbution -> Attribution\n\n017, We -> we\n\n077, level -> levels\n\n094, be -> been\n\n", "Justification_for_Ethical_Concerns": "As stated in Section 8.1, it may not be great to use Reddit data as the pretraining data, as pushshift is in violation of Reddit's Data API terms. "}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates a unified graph pre-training method for online communities. The reviewers appreciated the novelty, method, experiments, and analysis. The authors are recommended to address the reviewers' concerns in the camera ready, if the paper is accepted.
Decision: Accept-Findings