Content- and Topology-Aware Representation Learning for Scientific Multi-Literature
Authors: Kai Zhang
Kaisong Song
Yangyang Kang
Xiaozhong Liu
Venue: EMNLP 2023 Main (2023)
Keywords: Representation Learning
Graph-Text Joint Learning
Wasserstein Distance
TL;DR: This work focuses on learning both semantic and relatedness representation for scientific multi-literature and proposes a novel learning frame which learns content and citation information jointly.

Abstract: Representation learning forms an essential building block in the development of natural language processing architectures. To date, mainstream approaches focus on learning textual information at the sentence- or document-level, unfortunately, overlooking the inter-document connections. This omission decreases the potency of downstream applications, particularly in multi-document settings. To address this issue, embeddings equipped with latent semantic and rich relatedness information are needed. In this paper, we propose SMRC$^{2}$, which extends representation learning to the multi-document level. Our model jointly learns latent semantic information from content and rich relatedness information from topological networks. Unlike previous studies, our work takes multi-document as input and integrates both semantic and relatedness information using a shared space via language model and graph structure. Our extensive experiments confirm the superiority and effectiveness of our approach. To encourage further research in scientific multi-literature representation learning, we will release our code and a new dataset from the biomedical domain.

Content- and Topology-Aware Representation Learning for Scientific Multi-Literature
Kai Zhang 2 Kaisong 2 Yangyang Kang", Xiaozhong Liu"
Worcester Polytechnic Institute, Worcester, USA
2 Alibaba Group, Hangzhou, China
Northeastern University, Shenyang, China
(kzhang8, xliu14]@wph edu, (kaisong. .ks,, yangyang Janggyyell
Abstract
topics and access multiple appropriate publications, or even more innovatively, a synthesized summary of the overall set of sources. Such capabilities have the potential to revolutionize the modern research process (Wang et al., 2019; Zhang and Liu, 2016). Unfortunately, existing methods only focus on sin- gle document representation learning (SDRL) and the recommended paper selected based the rank of similarity between the learned representa- tion and the embedding of input query. While lan- guage models, such as Bert family(Devlin et al.. 2018; Maheshwari et al., 2021; Lee et al., 2020) and GPT family(Brown et al., 2020; Ouyang et al., 2022), has seen tremendous progress in understand- ing scientific documents (Wei et al., 2022) and graph-based approaches like GraphFormer( Yang et al., 2021), SPECTER(Cohaae et al., 2020) have studied the relatedness of a scientific paper in its ci- tation networks, little effort has been devoted to ex- ploring scientific litcrature representation learning at the multi-docummet level (MDRL). In this paper, we extend the community's s attention of representa- tion learning to dollm-d level and explore how to learn both semantic and relatedness infor- mation from content and topological networks.
Representation learning forms an essential building block in the development of natu- ral language processing architectures. To date, mainstream approaches focus on learn- ing textual information at the sentence- or document-level, unfortunately, overlooking inter-document connections. This omission decreases the potency of downstream applica- tions, particularly in multi-doccumnt settings. To address this issue, embeddings equipped with latent semantic and rich relatedness infor- mation are needed. In this paper, we propose SMRC2, which extends representation learn- ing to the multi-document level. Our model jointly learns latent semantic information from content and rich relatedness information from topological networks. Unlike previous studies, work takes multi-document input and in- tegrates both semantic and relatedness informa- tion using a shared space via language model and graph structure. Our extensive experiments confirm the superiority and effectiveness our approach. To encourage further rescarch sci- entific mutti-literature representation learning, we will release our code and a new dataset from the biomedical domain
1 Introduction
Intuitively, knowledge entities and text content of the given paper can provide latent semantic in- formation. For example, could simply infer paper belongs the biomedical domain by knowing its entities such as "diabetes", "GPXI" (a gene), etc. However, there a risk of incorrect inference when a computer science paper takes biomedical in- stances as examples. Same situation for using text content only, one can classify a paper simply by reading its abstract. Unfortunately, an NLP model will be hallucinated due to the close lexical similar- ity of papers from different categories. Thus, how to jointly learn from both entities and paper content can be critical provide latent semantic informa- tion. Typically, eled in the form of a graph which poses a challenge
With the increasing amount of scientific publica- tions, researchers progressively turn intricate nat- ural language processing (NLP) tools for paper dis- covery and recommendation. However, most exist- ing tools are tailored towards specific downstream tasks(Lu et al., 2020; Viswanathan et al., 2021) and thus their capabilities seem limited when presented with more complex tasks. This is attributed to the focus either the content topology information of scholarly documents, instead of both.
An aspiring scholarly search engine offers re- searchers with the capacity to comprehend query
"Corresponding Author "tuns://gthub..omMMattewwWakaa
triplets will be mod-

---

of balancing the learning between graph neural ner- works (GNNs) and language models (LMs). We novelly introduce Wasserstein distance to enable GNN and LM in parallel and interactively learn from each other while ensuring the learning of the latent semantic information.
2 Related Work
Representation Learning Most existing represen- tation learning works in NLP focus capturing the dependency in word-level, token-level and sentence-level(Peterss et al., 2018; Devlin al., 2018; Wu et al., 2020). With the success in previ- ous work, some efforts have been made to extend the learning paradigm to document-level (Yasunaga et al., 2022a; Li et al., 2021). However, these meth- ods suffer from the massive noises introduced by irrelevant words/sentencess Several Context-based and Graph-based models are proposed(Jeong et al., 2020; Wang et al., 2022b) to reduce noises in the training stage. Given document and its salient entities, for example, graph- models can casily build a graph where the entities are nodes and their relations are edges to derive/infer the knowledge within this graph. Unfortunately, these methods lack the consideration of rich relatedness in citation networks. Recent works (Cohan et al.. 2020; Yasunaga al., 2022b; Yang et al., 2021) push the boundary forward by including citation information. These works are orthogonal to ours as the input to our model is also multi-literature with citation relations, but the difference is that they leverage citation information for enhancing the source document's 's representation while our work is forcing each document to learn both semantic and relatedness information and at the same time, output unified representation for the entirety. Fur- ther, research work(Zhao et al., 2022; Chien al., 2021; Duan et al., 2022; Sun et al., 2020; Kong et al., 2022) proposed to tackle the Text--utributed Graph learning task (TAG) are also similar to our approach which aims at better schema for jointly graph and language model learning. However, our work goes beyond these approaches by explicitly integrating semantic and relatedness information in scientific multi-literature representation.
By leveraging the multi-document relations, e.g., citation graph, graph neural networks can simply enable each scientific paper to learn from its neigh- bors via the message passing mechanism. For ex- ample, An et al. (2021) proposed citation graph- based memorization model (CGSUM) to incor- porate the information of both the source paper and its references to obtain summarization. Wang et al. (2022b) utilized a disentangled representation based model DisenCite to encode paper accord- ing to both textual contexts and structure informa- tion for citation recommendation and generation. These works share resemblances to our method since they also take multi-document as input. How- ever, our work aims to explore output representa- tion equipped with latent semantic and rich relat- edness information for each document, as well as a unified embedding for the entire input. This ap- proach enables the learned representations not to be limited to specific tasks. Inspired by the effective masking strategy graph neural network equipped with novel masking strategy enable each document to learn the relatedness of its neighbors.
al., 2022), we propose to
To this end, we propose SMRC² which learns the Scientific Multi-literatie Representation from both Content information under Wasserstein con- straint and Citation information via graph structure. The main contributions our work are a threefold:
Conceptual: We extend scientific representa- tion learning to multi-doc level and demonstrate its necessity in various downstream tasks as well as the main difference compared with existing work.
Methodological: The proposed SMRC?, in- corporates a Wasserstein constraint to optimize the multi-view content learning and a graph masking strategy to complete the multi-document topology learning which enable us to learn both latent se- mantic and rich relatedness information from scientific ecosystem.
Multi- Dociment as Input In real-world scientific document applications, obtaining information from multiple documents is a common requirement. Researchers may want to know the whole story of a topic without reading every paper related to it, which falls under the subfield of NLP known multi-document summarization (MDS). While other multi-document tasks, such as multi- document reading comprehension. which aims to understand the content of multiple documents, are also of interest, our focus in this paper is on MDS to evaluate the learned molti-docummnttinn rep-
Experimental & Resource: We conduct exten- sive experiments using three benchmarks on three tasks and outperform several state-of-the-art base- lines. We also release a new Bio-Sci dataset that can be used for multiple downstream tasks.

---

resentation. Extractive methods usually produce summary by selecting the ranked sentences from the given document set Wan et al., 2015; Mendes et al., 2019; Zhong et al., 2020). For example, Nallapati et al. (2017) treats MDS as a sequence classification problem where each sentence is visited in sequential order and then adopts an RNN- based model to decide whether the sentence should be included in the summarization not. Liu et al. (2019) conceptualizes MDS to induce multi-root dependent tree representation of the documents. However, similar sentences may be close in the vector space and share close scores which would cause redundancy problem(Narayan et al., 2018). Recently, graph-based approaches which aim to extract salient textual units from documents based on graph structure representations of sentences are proposed to eliminate this(Chen al., 2021; Pasumuru et al., 2021). Different from these works, our method leverages document representation with latent semantic information as nodes and their citation relations as edges to construct the multi-document level graph. Then, the message passing mechanism can be resorted to equipping relatedness information for the final representation which can be applied in multi-document tasks.
graph embedding via a GNN encoder and the ab- stract text² embedding via language model. These two levels of information are then concatenated to form the initialized node features, along with their citation relations, to create a multi-doc graph. To learn rich relatedness information, we randomly mask some of their citation relations and train a GNN-based decoder to recover them based the node features using the message passing mecha- nism. By leveraging both the latent semantic and rich relatedness information, SMRC² is capable of not only single document but also multi-document applications.
3.1 Semantic Information Learning
Intuitively, the content of a scientific paper contains abundant semantic information. However, using the entire text content can be challenging since it may exceed length and computation resource limits, and would also introduce significant noise. Various attempts have been made to learn seman- tic representation from different aspects such as entities (Pivovarova and Yangarber, 2018) 8) and ab- stract (im and Gil, 2019) etc. Our hypothesis is that entities can provide a knowledge backbone and abstract can provide an overall summary which should be modeled and learned interactively to- gether. Hence, we utilize two encoders for both entities and text, and novelly introduce the Wasser- stein constraint to jointly train them.
3 Methodology
Problem Formulation Before going further, we first give preliminary definitions of concepts in this paper. Let P- = {PI....... }v} denotes a paper group with N papers and Gp = A, X) is the multi-doc graph of P with each node e E V rep- resents paper, A {0, 1 is the adjacency matrix and X' Bxdxd is the node feature matrix. Moreover, given fee as the graph encoder, fLas as the language model, and fGD as the graph decoder, our goal here is to obtain the document representa- tion H; as well as the multi-document representa- tion HM as follow:
Entity-Level Salient entity information can be used to somehow infer the content of a given pa- per. For instance, consider a scientific paper such as Zeng et al. (2020). If you know the "Dataset Degg.DD" "Metrics (e.g.. F1 score)", "Task (e.g., Relation Extraction)", "Method (e.g., GAIN)", you can conclude that this paper is using GAIN tackle relation extraction task and evaluate their approach on DocRed with Fl scores. Moti- vated by this, we follow Neumann et al. (2019); Jain et al. (2020)'s work and extract the entities within the given paper. Then we retrieve the re- lations among entities from a pre-defined relation set and use the extracted entities as nodes and their relations edges build entity-level graph. Then we apply a two-layer GCN to obtain the rep- resentation h! as shown in Eq.2.
h; = (fCE(P), Ны (oD(Cp(H))
(1)
where h E H, [:-1] is the concatenation operation. Overview As be seen in Figure 1, SMRC2 consists of two modules: Semantic representation learning module and Relatedness representation learning module. In the semantic lcarning module, SMRC² first in parallel obtain the entity-
n B0 = Re LU T)h EG WW!
(2)
Other sections are also considered, however, experiments show that they would introduce noises and require more com- putational resourees.

---

Dioert Dipritttoon
Corpus
frst
C-R.Graph
GNV Encodeer
Citation Recover
MI
Multi-Dac Graph
Contralm
Modaa Indiiiinoon
GNN Decoder
Fgy
fuo
MLP TM
Encoder
True Citation
fuM
Diiceete Old toutot
Holatoness Information Learning
Semantic Information Learning
The overall frame for SMRC² involves first obtaining semantic embeddings for each document in a paper group via the semantic information learning module. Then these embeddings serve as the initialized node features and their citation relations information. The concatenation operation is denoted as b.l.
Figure I:
the edges form the multi-doc graph, which can later learns the relatedness
where 11, E(()) is the 1-th layer's hidden embedding where
where e is Softmax function, MLP is multi- layer perceptron. Kullback-Liebler divergence(Joyce, 2011) and other likehood-basee divergence methods suffer from statistical limitations such as invariant to any invertible transformation et al., 2019). They are also not suitable for representation learning due to their sensitivity to the trivial differences in the data samples. However, inspired by the success of WD in GAN training(Arjovsky al., 2017). we consider using WD to enable fce and fLM learn from each other by minimizing the distance between their predicted label distributions. Wasser- stein Distance is a metric-aware divergence derived from transport optimal theory. It measures the dis- crepancy between two distributions in terms of min- imum total costs associated with some transport function(Hou et al., 2020). Drawing inspiration from Villani al. (2009), we give the definition of Wasserstein Constraint in our settings:
(hf h; (() E() kem(1)) Radd and is the number of entities), 1 is the identity matrix and W trainable param- eters. Notably, for entity starting from s-th word to t-th word, h1 (0m(0) T- Ej= 9j. where gj is the word embedding.
h. ( (,
h
Texe-Level However, relying solely on entity-level information may not be sufficient for representing the entire paper, as prescribed in Section 1. There- fore, we opt to incorporate the abstract text since it contains the most general content of a scientific paper. Given the abstract of scientific publication P.. language model will generate d-dimensional embedding vector denoted h Rd:
b = JEM (Pabee)
where fLM is a language model.
Entity-Text Interaction Given P. the corre- sponding distribution of document labels juy) = p((n, V2, ......... (NN) can be the supervised signals for fGE and fLA to learn salient semantic informa- tion. However, relying solely on abstract or entity information for predicting labels can be unreliable. To overcome this limitation, we introduce novel approach using Wasserstein Distance (WD) to re- strict fer and fLM from learning independently of each other while ensuring they predict accurate labels for the documents. For LM and GNN, the predicted distribution can be described follows:
Definition 1. Given two distributions μl, P2, a cost function est: N x N - RR and the set of joint probability T(HI, (2), the Wasserstein Distance can be defined as:
Wea(pi.po) inf Jxx est(yi.32)dy
nev(es.22)
where y1 and y2 data points in distributions A1 and μaz, respectively.
Therefore, our current objective can be refor- mulated as minimizing the optimal cost function while simultancously forcing the encoders to learn semantic information for accurate document label
(P() = Cat(y (O(((()T)(MMM((EE)
(4)
((P) Cat(FF1(MMLP())))

---

prediction. Assuming that the optimal cost function can be replaced with certain metric d or its k-th power dt with 2 1, then the Wasserstein Loss of our settings can be defined as:
NB(i) is the neighbor nodes of We can create Gp by considering each document in P as node and their citation relations as edges. Then the message- passing mechanism can be leveraged to enable each document to learn about the relatedness within its citation network. Intuitively, one can judge the possibility of citation relation between two papers by knowing their semantic content (c.g.. if two pa- pers are all about using graph learn document representation, there exists a chance that they have citation relation). Inspired by the success of mask- ing strategy on Variational Graph Auto Encoder (VGAE) to predict the adjacency matrix(Hou et al., 2022), we employ a similar approach. We intu- itively mask some of the citation relations and train VGAE to recover the citation relations. However, the difference is that don't 't need the encoder part of VGAE and instead of an unsupervised process, we adopt self-supervised settings. Specifically, we first initialized a random masking matrix M with masking ratio w and obtain the masked adjacency matrix Amasked by applying element wise multipli- cation between M and A (c.g., w of . are zeros and the reset are ones). Then given the semantic representation of each document as the initial node embedding, GNN can iteratively update them by applying Eq. on Gp (V, Amakked: [li]' 1())). Now, our goal here becomes reconstructing the in terms of Icarned I-th node embedding. However, directly applying the final-layer hidden represen- tation n!!) can cause performance degradation for producing Á since there would be long-distance information lost and irrelevant inclusion problems as the number of layers increase, hence we aim to capture the cross-correlations for every node-pair:
Wa ,r)=
inf E(d (vt vs )) (6) "Enlu" (1))
where the set of joint distribution is nlu² 1) {y E RAX 71 -²" 1 all-one vector. By calculating the derivative of μ², ne with respect to W in an attempt to approach true document label distribution, we can describe the overall loss function of semantic representation learning module
r²}. 1 is the
(5) Cat (M h bFI)))
W + (-
1-- E Plui)log
(7)
iCN
where the second term is the Cross-Entropy loss. By incorporating the Wasserstein Constraint, the two encoders can enhance their learning ability by leveraging each other's 's outputs, while also captur- ing the necessary semantic information for docu- ment classification. As far as our knowledge ex- tends, this is the first attempt to introduce Wasser- stein Distance for multi-view semantic representa- tion learning within a scientific document.
3.2 Relatedness Information Learning
To achieve effective multi-document representation learning, it is imperative to consider the rich re- latedness information of the topological network. While the knowledge of each paper's semantic in- formation can aid in determining its content, it may be challenging produce summaries based solely this information. Relying solely semantic representation may also lead to sub-optimal results, especially in cases with insufficient textual features. Therefore, it is crucial to capture the semantic infor- mation while also incorporating the rich relatedness information. A promising aspect of Graph Neural Networks (GNNs) is their ability to be aware of their neighbors' facts through the message-passing mechanism (details can be seen in Appendix 5), which can be described as follows:
hij [h_ e hT. O
(9)
where @ denotes pair-wise element multiplication, i [0, NJ. We now reconstruct Á as:
p(A(h,)) IIII pl A(Ar,)hi))
(10)
=-
with p(Au = 11h4r) = or (listtu))
where Ar is the element of Á and a is an activation function. By having A and A, we follow Kipf and Welling (2016b)'s work to optimize fen as:
(-1) A = (AGGG( NBB) NBB)) A))
(8)
where nn0) [h] a an activation function (c.g., ReLU), M MSG()) and AGG(-) stand for the message and aggregation functions respectively,
l - - p(A)log plA)
(11)
IEN

---

Task
Type
Dataset
Ogbn-Arxi
Ogbl- Ogbl-Citation2
Bio-Sci Sci
Document Classification
SDT
Citation Prediction
Multi-doc Summarization
MDT
Table 1: Different tasks on different datasets
Model
Type
Ogbn-Arsiv
BioSSi
to predicting the labels by / (MLL()))) We
Bio-Ben
68.21 # 0.17
76.67
0.17
Sci-Rert
IM
73.80
70.49
0.31
have selected several SOTA baselines of different
GraphSAGE
EaOCN
Graph
T7.57 + 0.07
71.49
0.21
79.35
85.27 = 0.06
0.07
types to assess the efficacy of our model: LM- Bio-
GIANT-XRT
16.37 0.11
84.81
= 0.13
Bert(Lee et al., 2020), Sci- Cii-Bert(MMahshhweee et al.,
GLEM(Graph)
LM+Graph
75.50 1. 0.24
82.93
0.11
GLEMILM)
74.33 + 0.12
78 0.07
2021); Graph EnGCN(Duan et al., 2022), Graph-
SMRC
LM-Graph
77.66
Sage(Kong et al., 2022); LM+Graph GIANT- XRT(Chien al., 2021), GLEM(Zhao al., 2022). Citation Prediction (Cit Pre.) which refers to predicting the citation relations by knowing the semantic content of each paper. Our model pre- diets citation relations by reconstructing the adja- cency matrix A via Eq. 10. For evaluating the performance of our proposed model, compare against several SOTA baselines: GCN(Kipf and Welling, 2016a), GraphSage(Kong et al., 2022). GraphSaint(Zeng al., 2019).
Table Mean accuracy (%) + one standard deviation comparison for Document Classification in terms of different types of baselines.
Now, with the latent semantic and rich relatedness information, the learned representation is h, h according Eq. 8 and HM- [.... hi.....
3.3 Training
Overall, we consider the semantic representation learning module supervised process and the re- latedness representation learning module as self- supervised process, the whole pipeline of SMRC2 can be trained through / as follows:
Model
Ogbn-Citation2
Bio-Sel
Full-batch GCN
83.14 1 0.21
71.11
GraphSage
82.60 + 0.36
69.17 + 0.31
GraphSaint
79.85 + 0.39
70.31 + 0.26
SMRC"
83.31 + 0.27
73.97 :: 0.19
1 = 1 +
(12)
Table 3: MRR score in percent comparison for Citation Prediction in terms of different baselines.
We perform full-batch gradient descent on / which enables SMRC² to obtain both semantic and re- latedness information within supervised plus self- supervised learning settings for scientific multi- literature representation learning.
Multi-Document Summarization (MDS) aims to generate a summary for given multiple docu- ments. Here, we gauge the similarity between HM and the sentences from sentence derived from original papers. The top sentences will be picked and copied as the generated summary. SMRC² out- performs several baselines: HiMAP((Fbbrri et al.. 2019), Pointer-Gennrrator(Se et al., 2017), Hier- Summ(Liu and Lapata, 2019), KGSum (Wang et al., 2022a), REFLECT(Song et al., 2022).
4 Experiments
this section, we empirically evaluate the perfor- mance of our learned document representation h; and the unified multi-doc representation НM-
4.1 Tasks & Datasets
Datasets We choose three benchmark datasets cording to the tasks and would like to introduce new benchmark dataset for the community Bio- Sci which consists of 32,330 publications, 222,652 citation relations derived from PubMed Central Specifically, we follow the work of Achakulvisut et al. (2020) to extract the abstract, introduction and citation sentence for each paper (Notably, we only use abstract for LM in this work). Further, every
Tasks The key distinguishing aspect of our work from previous research is that our learned represen- tation is applicable not only to hingle-doumment tasks (SDT) but also to multi-document tasks (MDT). Thus, we have evaluated our learned repre- sentation both SDT and MDT for four datasets, shown in the Table 1:
Document Classification (Doc Cla.) which aims to predict the class for each paper using se- mantic or structural information. Here, we resort
hhttp::/ttp.ncbi.nnt nlm.nih www.hhoowwbb/pm///aaa bulk/oa commm/

---

Model
Type
Multi-XScience ROUGE-2
Rio-Sci
ROUGE-1
ROUGE-L
ROUGE-1
ROUGE-2
ROUGE-L
BioBerABBS
24.77
4.01
21.67
31.66
5.91
28.43
SciBertABS
Concat
25.11
4.17
22.91
30.02
27.60
REFLECT
29.41
6.11
24.91
34.17
29.57
HIMAP
26.61
6.19
25.31
34.11
6.76
29.77
HIERSUMM
25.43
6.01
24.91
35.02
7.04
29.66
Multi
KGSum
30.16
6.24
25.37
34.98
6.91
30.07
SMRC
31.47
6.73
26.03
36.23
7.31
30.63
Table 4: Rouge- I, Rouge-2, Rouge-L score comparison with different baselines.
paper is associated with an id called pmid which enables us to find the citation relations by matching the pmid. For each paper, we use its MeSH term its category label. Every paper group Bio-Sci consists of 10 papers and their citation relations*.
model. Additionally, the citation prediction results demonstrate the importance of node features in in- ferring the connections among publications since our model outperforms other baselines by aggregat- ing the semantic embedding first as the initial node features while other baselines use word embedding solely. Furthermore, the learned multi- multi-literature embedding of our model holds great promise for multi multi-document tasks such as multi-document sum- marization due to its ability of aggregating both latent semantic and rich relatedness information into the final representation.
4.2 Implementation Details
For Bio-Sci, follow the work of Wang et al. (2021) and use SciSpacy(Neumann al., 2019) to obtain entity-relation graph can detailed Ap- pendix 5, the base LM model is Bio-Bert(Lee et al., 2020). For other datasets, we follow the work of Ye et al. (2022) in the data processing stage and use Cei-Ber(MMaheswwari et al., 2021) as the base language model. We leverage Adam(Kingma and Ba, 2014) the optimizer. Hyperparameters set- ting can be seen in Appendix 5. For evaluation, we follow the commonly used metries on these benchmarks. All experiments are conducted with KeekachKKetkar and Moolayil, 2021) and Trans- formers( Wolf et al., 2020) on Tesla P100 GPUs.
Table 4 confirms our assumption, as our model outperforms current SOTA baselines. Surprisingly, HiMAP, which adapts a pointer-generator model for weight computation over multi-document, and HierSumm, which leverages a ranking baagg selec- tion mechanism for sentence selection, perform bet- ter than pre-trained large language models. We be- lieve this is because these models focus on find- ing the most important parts over multi-document, whereas pretrained models inevitably suffer from noise by simply concatenating all documents to- gether as a single document before performing summary generation. It's also very interesting that on Bio-Sci, the goal is to recover abstract, sen- tence selection based models such as HierSumm and SMRC² performs better than other models. We also observe that KGSum outperforms REFLECT which we attribute the fact that REFLECT only focuses on sentence selection which ignores the fine-grained semantic information from entity level and also the rewriting process is finished based on the inner natural language understanding ability of chosen pre-trained model without awareness of topological information. However, it is evident that our method surpasses KGSum which we attribute to the fact that our method leverages Wasser- stein Constraint for modeling the interactions be- tween entity-relation graph and semantic text in- formation while KGSum learns only from salient
4.3 Comparative Study
As shown in Table 2, SMRC² has demonstrated a relatively 0.12% and 1.33% improvement over the best baseline on Ogbn-Arxiv and Bio-Sci, re- spectively. Generally, we can conclude that re- lying solely on either LM or Graph alone is not comparable in terms of results. This can be ex- plained by the fact that plain text can introduce noise, while graph structure can mitigate this issue. Furthermore, the results highlight the effectiveness jointly leveraging Graph and I.M, which validate our efforts in modeling the interactions of different modules. However, EnGCN produced competitive results among all baselines which we attributed to its ensemble training schema from multi-view of input which boosts the convergence of the whole
*Detailed comparison can be seen Appendix "All baseline, we use the best purameter settings reported in their paper and we keep the base language model the same for fair comparison

---

Type
w/o
w/
Doe Classification
Citation Prediction
Summarization
LM
78.91 0.11
69.33 0.11
19.11
Graph
79.61 0.20
67.47 0.10
23.71
(Part
WD
82.91 0.34
69.71 + 0.02
24.31
KL
81.31 0.17
71.27 + 0.14
20.51
JS
85.77 0.08
72.41 + 0.22
24.41
(Full
88 41 T 0.21
73.97 0.19
26.03
Table 5: Ablation study on different components. w/o denotes without, denotes with, the metries stay the same in Comparative Study (For the Summarization task, the metric used is ROUGE-L).
entity-sentence graph. Our model grasps not only semantic information from LM but also related- ness information from graph structure which help to perform the best against all basclines.
We assess the effectiveness of our masking strat- egy applied in the relatedness learning module by setting different values of w. As can be seen in Table 6, we can conclude that masking ratio can influence the effectiveness of overall performance since the masking strategy is associated with the relatedness information learning. When masking more edges, the performance of citation prediction will inevitably decrease since the complexity of recovering can be higher. However, further exper- imentation revealed that 5% masking for citation prediction produced a result of 70.21 + 0.11, sug- gesting decrease due to the risk of overfitting. We can also notice that the performance on Document Classification and Multi-Doe Summarization tasks decreased as masking more edges since there can be semantic information lost and noise introduction due the update of hidden representations which further validates the necessity of Eq. 9. Never- theless, the recover process can be insufficient for the model to learn relatedness information when masking 10% of edges. The results suggest that masking 20% can be the best choice for SMRC?.
4.4 Ablation Study
To further evaluate the completeness of SMRC?, we conduct ablation study and the results are pre- sented Table 5. During experiments without LM, we use random embedding initialization to replace the initial embedding from LM which results in the huge drop of performance in Document Clas- sification task compared to experiments without graph, showing the importance of semantic infor- mation. The citation prediction results reveal that solely using semantic information is not enough for inferring the connections while graph structure is necessary for including relatedness information. We can also witness the superiority of introducing Wasserstein Constraint as the performance drops when we apply KL. or IS divergence as replace- ment for semantic representation learning. Inter- estingly, we found that KL divergence is not stable during the training stage, with performance oscilla- tion as the training iteration progresses, which we attribute to its asymmetry. Introducing Wasserstein Constraint not only solved the imbalance training problem but also enabled our model to learn seman- tic information from both text and entity aspects while forcing it to make correct label predictions, further confirming the necessity and superiority of the entire structure of our model.
4.6 Visualization
To further demonstrate the effectiveness of SMRC2 we randomly select 100 papers for each category from Bio-Sci and utilize t-SNE(Van Der Maaten, 2014) to visualize these learned embeddings by SMRC² and compared them with the ones gener- ated by BioBert. We set the perplexity and the number of iterations for t-SNE as 30 and 300, re- spectively. As shown in Figure 2, the clusters of pa- pers from different categories generated by SMRC2 are more compact than those by BioBert, indicat- ing that our model is better at modeling semantic information. Our model's 's ability to learn rich latedness information is also reflected by the inter- category distance. Intuitively, diseases that share high correlations should be close the embedding space. For instance, the relationship between dia- betes and kidney disease is accurately reflected, as
4.5
Masking Study
Settings
Doe Cla.
Cit Pre.
MDS
10%
86.19 + 0.11
74.13 + 0.31
24.91
20%
88.41 + 0.21
73.97 + 0.19
26.03
30%
84.17 + 0.07
70.31 + 0.26
22.51
Table Masking study on evaluating the effectiveness of Masking Strategy, all the metrics stay the same as in Ablation Study.

---

Cardiovescula
Kidney disease
Respiratory diseases
Diabetes melitus
Digestive discases
Hy
Hopatitss
Mental diserders
Musculskkeletll
Neaplasms
Neuroleg disprders
(a) BioBert
(b) SMRC²
Figure 2: Representation visualization of selected papers using t-SNE.
the representations of "Diabetes mellitus(green)" and "Kidney disease(orange)" are close in the em- bedding space, which is consistent with the fact that diabetes is the leading cause of chronic kidney disease, and people with kidney disease are at higher risk of developing diabetes. These findings further confirm the ability of SMRC² to capture latent semantic and rich relatedness information.
guage Model, two Graph Neural Network-band based modules, and multiple operations to be trained in tandem. Furthermore, this multi-MLP embedding concatenation process makes challenging to parse the encoded embeddings, thus potentially impact- ing performance. These limitations suggest that further investigations are needed to address the scal- ability and generalizability of the proposed model to improve its applicability in broader range of scientific data.
5 Conclusion
As conclusion, we have advanced the field of rep- resentation learning to the multi-doc level for sci- entifie literature, with the introduction of SMRC² This novel approach effectively combines two el- ements semantic content and topological relat- edness, to create a superior learning space. To address the imbalance learning problem that occurs during the semantic learning stage, we introduced Wasserstein distance. To our best knowledge, this is the first attempt to introduce WD for multi-view content learning. The extensive experiments and ablation study that we conducted validate the effec- tiveness of the proposed model. Additionally, we released a new dataset to encourage the community to further explore the multi-document representa- tion learning. These contributions advance research in scientific literature mining and provide useful resource for the community to further investigate this new but important topic.
Ethics Statement
After carefully reviewing the ACL Ethics Policy, we are committed to show our respect and obey to consent all.
Acknowledgements
We gratefully acknowledge support from NSF, Award # 2122232 SCISIPBIO: Constructing Het- erogeneous Scholarly Graphs to Examine Social Capital Duringgennorrd K Awardees Transition to Research Independence: Explicating Matthew Mechanism.
References
Titipat Achakulvisut, Daniel Acuna, and Konrad Ko- rding. 2020. Pubmed parser: A python parser for pubmed open-access xml subset and medline xml dataset xml dataset. Journal of Open Source Soft- ware, 5(46):1979.
Limitations
Chenxin An, Ming Zhong, Yiran Chen, Danqing Wang. Xipeng Qiu, and Xuanjing Huang. 2021. Enhancing scientific papers summarization with citation graph. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages
Despite its empirical success, the use of SMRC² has two notable limitations. Firstly, its application is restricted requires interconnections between documents (e.g. citation graph) to form the multi- document graph. Secondly, the method incurs high computational costs due to its inclusion of a Lan-
Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein generative adversarial networks.

---

In International conference on machine learning, pages 214-223. PMLR.
James M Joyce. 2011. Kullback-leibler divergence.
In International encyclopedia of statistical science. pages 720-722. Springer.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Nikhil Ketkar and Jojo Moolayil. 2021. Introduction to pytorch. In Deep learning with python, pages 27-91. Springer.
Sang-Woop Woon Kim and Joon-Min Gil. 2019. Research paper classification systems based on tf-idf and lda schemes. Human-cene centric Computing and Informa- tion Sciences, 9:1-21.
Moye Chen, Wei Li, Jiachen Liu, Xinyan Xiao, Hua Wu, and Haifeng Wang. 2021. Sgsum: Transform- ing multi-document summarization into sub-graph selection. arXiv preprint arXiv:2110.12644
Diederik P Kingma and Jimmy Ba. 2014. Adam: A
method for stochastic optimization. arXiv preprint 1rXiv:1412.690
Eli Chien, Wei-Cheng Chang, Cho-Jui Hsieh, Hsiang- Fu Yu, Jiong Zhang, Olgica Milenkovic, and In- derjit S Dhillon. 2021. Node feature extraction by self-supervised multi-scale neighborhood prediction. arXiv preprint arXiv:2111.0006
Thomas N Kipf and Max Welling. 2016a. Semi- supervised classification with graph convolutional networks. arXiv preprint arXiv:669.002900
Thomas N Kipf and Max Welling. 2016b. Vari- ational graph auto-encoders. arXiv preprint arXiv: 16110773 1.07308.
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld. 2020. Specter: Document-level representation learning using citation-informe transformers. arXiv preprint arXiv:200..07180.
Kezhi Kong, Guohao Li, Mucong Ding, Zuxuan Wu, Chen Zhu, Bernard Ghanem, Gavin Taylor, and Tom Goldstein. 2022. Robust optimization data aug- mentation for large-scale graphs. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 60-69.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arxiv preprint
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 10:0:)224-11440
Keyu Duan, Zirui Liu, Peihao Wang, Wenqing Zheng, Kaixiong Zhou, Tianlong Chen, Xia Hu, and Zhangyang Wang- 2022. A comprehensive study on large-scale graph training: Benchmarking and rethinking. arXiv preprint arX14:2210.07494.
Peizhao Li, Jiuxiang Gu, Jason Kuen, Vlad I Morariu. Handong Zhao, Rajiv Jain, Varun Manjunatha, and Hongfu Liu. 2021. Selfdoc: Self-supervised doc- ument representation learning. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 066-5660..
Alexander R Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. arXiv preprint 1906.0144.
Yang Liu and Mirella Lapata. 2019. Hierarchical trans- formers for multi-document summarization. arXiv preprint arXiv: 1905. 13164.
Linfang Hou, Liang Pang, Xin Hong. Yanyan Lan, Zhiming Ma, and Dawei Yin. 2020. Robust reinforce- ment learning with wasserstein constraint. arXiv preprint 2006:20000009444
Yang Liu, Ivan Titov, and Mirella Lapata. 2019. Single document summarization tree induction. In Pro- ceedings of the 2019 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 1745-1755, Min- neapolis, Minnesota. Association for Computational Linguistics.
Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjie Wang, and Jie Tang. 2022. Graphmae: Self-supervised masked graph autoen- coders. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Min- ing, pages 594 604.
Sarthak Jain, Madeleine Van Zuylen, Hannaneh Ha- jishirzi, and Iz Beltagy. 2020. Scirex: A challenge dataset for document-level information extraction. arXiv preprint arXiv:2005.00512.
Yao Lu, Yue Dong, and Laurent Charlin. 2020. Multi- xscience: A large-scale dataset for extreme multi- document summarization of scientific articles. arXiv preprint 20Xiv:2010.142:5
Chanwoo Jeong, Sion Jang, Eunjeong Park, and Sungchul Choi. 2020. A context-aware citation rec- ommendation model with bert and graph convolu- tional networks. Scientometrics, 124(3):1907-19)
Himanshu Maheshwari, Bhavyajeet Singh, and Va- sudeva Varma. 2021. SciBERT sentence representa- tion for citation context classification. In Proceed- ings of the Second Workshop on Scholarly Document

---

Processing, pages 130-133, Online, Association for Computational Linguistics.
Abigail See, Peter Liu, and Christopher D Man- ning. 2017. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:/704.04580
Afonso Mendes, Shashi Narayan, Sebastiao Miranda, Zita Marinho, André FT Martins, and Shay B Cohen. 2019. Jointly extracting and compressing documents with summary state representations. arXiv preprint (ifi::990402022
Yun-Zhu Song, Yi-Syuan Chen, and Hong- Han Shuai. 2022. Improving multi-document summarization through referenced flexible extraction with credit- awareness. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 1667-1681, Seattle, United States. Association for Computational Linguisties.
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: recurrent neural network based sequence model for extractive summarization of doc- uments. In Thirty-first AAAI conference on artificial intelligence.
Chuxiong Sun, Jie Hu, Hongming Gu, Jinpeng Chen, and Mingehuan Yang. 2020. Adaptive graph diffu- sion networks. arXiv preprint arXiv:2012. 15024.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Ranking sentences for extractive summariza- tion with reinforcement learning. arXiv preprint arXiv:1802.08636.
Laurens Van Der Maaten. 2014. Accelerating t-sne using tree-based algorithms. The journal of machine learning research, 15(1):322132455
Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. 2019. ScispaCy: Fast and robust models for biomedical natural language processing. ceedings of the 18th BioNLP Workshop and Shared Task. pages 319-327, Florence, Italy, Association for Computational Linguistics.
Pro-
Cédric Villani et al. 2009. Optimal transport: old and new, volume 338. Springer.
Vijay Viswanathan, Graham Neubig, and Pengfei Liu. 2021. Citationie: Leveraging the citation graph for scientific information extraction. arXiv preprint wrXiv:2100.01560.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances Neural Information Processing Systems, 35:27730-27747
Xiaojun Wan, Ziqiang Cao. Furu Wei, Sujian Li, and Ming Zhou. 2015. Multi-document summariza- tion via discriminative summary reranking. arXiv preprint 1eXiv:500700000
Sherjil Ozair, Corey Lynch. Yoshua Bengio, Aaron Van den Oord, Sergey Levine, and Pierre Sermanet. 2019. Wasserstein dependency measure for repre- sentation learning. Advances in Neural Information Processing Systems, 32.
Pancheng Wang, Shasha Li, Kunyuan Pang, Liangliang He, Dong Li, Jintao Tang, and Ting Wang. 2022a. Multi-document scientific summarization from a knowledge graph-centric view. In Proceedings of the 29th International Conference Computational Linguistics, pages 6222-6233, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Ramakanth Pasunuru, Mengwen I Liu, Mohit Bansal, Su- jith Ravi, and Markus Dreyer. 2021. Efficiently sum- marizing text and graph encodings of multi-document clusters. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 4768 4779.
Shike Wang, Fan Xu, Yunyang Li, Jie Wang, Ke Zhang, Yong Liu, Min Wu, and Jie Zheng. 2021. Kg4sl: knowledge graph neural network for synthetic lethal- ity prediction in human cancers. Bioinformatics, 37(Supplement 1):4418-4255
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zeulemoyer. 2018. Deep contextualized word repre- sentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume I (Long Papers), pages 2227-2237. New Orleans, Louisiana. Association for Computa- tional Linguistics.
Yifan Wang. Yiping Song, Shuai Li, Chaoran Cheng, Wei Ju, Ming Zhang, and Sheng Wang. 2022b. Disencite: Graph-based disentangled representation learning for context-specific citation generation.
Yongzhen Wang, Xiaozhong Liu, and Zheng Gao. 2019. Neural related work summarization with a joint context-driven attention mechanism. arXiv preprint 191.99190..09492.
Lidia Pivovarova and Roman Yangarber. 2018. Com- parison of representations of named entities for docu- ment classification. In Proceedings of the Third Work- shop on Representation Learning for NLP, pages 64-68, Melbourne, Australia. Association for Com- putational Linguistics.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le. and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.1190

---

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite. Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of tofte-of-theaarr the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 48-45, Online. Association for Computational Linguistics.
Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2020. Extrac- tive summarization as text matching. arXiv preprint arXiv:2004.08795.
Training Settings
The parameter settings of trainable module are de- tailed in Table 7 and Table 8.
GCN
Training
Parameter
Value
Parameter
Value
Number of layers
I. 2,
learning rate
10
10
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. 2020. Clear: Contrastive learning for sentence representation. arXiv preprint arXiv:2012.15466.
emb size
100
dropout
0.3. 0.6
hidden size
512
butch size
20,
weight decay
10
Table 7: The experimental settings of graph and training with the best parameter settings are highlighted.
Junhan Yang, Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanjay Agrawal, Amit Singh, Guangzhong Sun, and Xing Xie. 2021. Graphform- ers: Gnn-nested transformers for reprexentation learn- ing on textual graph. Advances in Neural Information Processing Systems, 34:28798-28811
Pre-trained Mudel
Transformer
Parameter
Value
Parameter
Value
emb size
768
of Attention Head
max length
256, 512
Num of Layers
16. 32, 64, 128 512, 768
dimension
Michihiro Yasunaga. Jure Leskovec, and Percy Liang. 2022a. Linkbert: Pretraining language models with document links. arXiv preprint arXiv:2203. 15827.
Table 8: The experimental settings of language model with the best parameter settings are highlighted.
Michihiro Yasunaga, Jure Leskovee, and Percy Liang. 2022b. LinkBERT: Pretraining language models with document links. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 8003- 8016, Dublin, Ireland. Association for Computational Linguistics.
Dataset Statisties
Category Bio-Sci contains 11 different paper cate- gories in total: ease, Respiratory diseases, Diabetes mellitus, Di- gestive diseases, Hiv, Hepatitis, Mental disorders. Musculoskeletal disorders, Neoplasms, Neurologi- cal disorders, of which "Cardiovascular discases" and "Mental disorders" take the most percent 26%, 18% respectively. Others share evenly for the rest.
Cardiovascular diseases, Kidney dis-
Deming Ye, Yankai Lin, Peng Li, and Maosong Sun. 2022. Packed levitated marker for entity and relation extraction. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 4904 4917. Association for Computational Linguistics.
Dataset Ogbn-Arxiv Ogbl-Citation2 Bio-Sci
Nodes
Edges
Avg. Node Degree
Hanqing Zeng. Hongkuan Zhou, Ajitesh Srivastava, Ra- igopal Kannan, and Viktor Prasanna. 2019. Graph- saint: Graph sampling based inductive learning method. arXiv preprint arXiv:1907.049: 049931.
169,343
1,166,243
13.7
2.927.963
30,561,187
32,330
222.652
13.7
Table 9; Statistics of datasts
Shuang Zeng. Runxin Xu, Baobao Chang, and Lei Li. 2020. Double graph based reasoning for document- level relation extraction. In Proceedings of the 2020 Conference on Empiricul Methods in Natural Lan- guage Processing (EMNIP),, pages 1630-1640, On- line. Association for Computational Linguistics.
SciSpacy Details
For SciSpacy usage, we use them to ob- tain the entities with "en_co _aar__orre__eeeeeeeeeeeeeeeeeeee sci_sm"" and "en peer_bionlpl.cg.c settings are trained in large-scalt biomedical do- main corpus. The fl score is 67.87% on the entity extraction task for "eee_eore_seeeeeeeeeeeeeeeeeeeee sci_sm'' and 76.75% for *en www_perrbioolleeggg_._m
corpus because these two
Lin Zhang and Xiaozhong Liu. 2016. Synopsizing "liter- ature review" for scientific publications. IConference 2016 Proceedings.
Jianan Zhao, Meng Qu, Chaozhuo Li, Hao Yan, Qian Liu, Rui Li, Xing Xie, and Jian Tang, 2022. Learning on large-scale attt-bttri graphs via variational inference. arXiv preprint arXiv:2210.144 14709.
We mainly lever- age "en_ cee__eerbboonly Beepulppppe__mpeee in our experiments. What we do for linking entities is we first extract

---

the entities and retrieve the BKG to if there matches for them.
Message Passing Mechanism Details
What we do for using MPM is that we use the semantic information, lef and h learned from Section 3.1 to initialize hç- As we obtain the initial- ized document embedding, the MSG function is for finding and collecting the embeddings of nearby nodes for node and the AGG function will be used to perform aggregation (e.g., concatenation, pooling etc.) of embeddings in terms of node and its neighbors. Eq. 8 will update the representation of node iteratively and thus both the semantic in- formation of node itself and its neighbors can be learned and will be used in the following training procedure.


Reviews:
[{"review_id": "EYmJ51xtcp", "reviewer": "Reviewer_LfTX", "summary": "", "questions": "A: Equation **(9)** proposed by the authors in *Section 3.2* seems to compute the cross-correlation of pairs of nodes at the same level and then concatenate them together because *directly applying the final-layer hidden representation can be difficult for producing the reconstruct matrix*. However, it seems that applying the last level of hidden representation directly is also feasible in this context, and the approach of Eq. **(9)** introduces more unnecessary complexity, can the authors elaborate on the reasons for doing so?\n\nB: How exactly do the GNN used by the authors in Section 3.2 perform message passing?\n\nC: Have the authors conducted experiments to discuss the effectiveness of the masking strategies? What are the specific results?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your useful suggestions. We notice there exists some implicit and unclear method descriptions which lead to your confusions and concerns. We admit to enhance the writings more explicit and some additional empirical results are provided to further support my response and hopefully can reduce some of your concerns. \n\n**Q A**: Why Equation 9 is applied which is more complex instead of using the last layer hidden representation?  \n\n**A**: This is a commonly-used technique in representation learning inspired from self-supervised learning on graph [1]. Our validation confirms that a combined-layer representation outperforms the last-layer representation due to the latter's tendency to forget long-distance neighbors with an increasing number of layers.\n\nFurther, we indeed tried to use the last layer of hidden representation initially. However, the performance of the citation adjacent matrix reconstruction was suboptimal. Upon investigations, we attributed this to the fact that the embedding of l-th layer can contain irrelevant information from neighbors which leads to potential noises inclusion. Subsequently, we adopted the strategy of computing the cross-correlation of pairs of nodes to identify the shared patterns between node i and its neighbors and thus alleviate the long-distance and irrelevant inclusion problem. We apologize for any confusions caused in our paper and we promise to provide more details in the revision.\n\n**Q B**: How the message passing mechanism of GNN is performed?\n\n**A**: Thanks for this question. Typically, the message passing mechanism (MPM) follows an iterative schema of updating the node representation based on the aggregation from its neighbor nodes[2]. The powerful MPM enables GNN to realize the topological information. However, due to the page length limitation, we resort to saving the space for other details of our overall frame instead of these common-used MPM information. Our primary design goal in this work was to enable the nodes within the multi-doc graph to be aware of the information of their neighbors through the use of MPM. To this end, what we do is that we use the semantic information learned from Section 3.2 to initialize $h_{i} (h_{i}^{0} = [h_{i}^{E}, h_{i}^{T}])$ as can be seen in line 377. As we obtain the initialized embedding, the MSG function is for finding and collecting the embeddings of nearby nodes for node i and the AGG function will be used to perform aggregation (e.g., concatenation, pooling etc.) of embeddings in terms of node i and its neighbors. Eq. 8 will update the representation of node i iteratively and thus both the semantic information of node i itself and its neighbors can be learned and will be used in the following training procedure. Due to the page length limitation, we are not able to put these details. However, we recognize that there is an opportunity to further enhance our writing to reduce any confusion, and we will address this problem in the revision.\n\n**Q C**: The experiments on evaluating the effectiveness of masking strategy.\n\n**A**: Thank you for your constructive feedback regarding our approach. We did try different masking ratio (this parameter can be found in line 400) and recognize that the selection for masking ratio is an experimental selection. We have evaluated different masking ratio strategies. Specifically, we set the ratio as 10%, 20% and 30% and the results are detailed as below: \n|Settings | Document Classification | Citation Prediction | Summarization|\n|:---:|:---:|:---:|:---:|\n|10% | $86.19\\pm 0.11$ | $74.13\\pm 0.31$ | 24.91\n|20% | $88.41\\pm 0.21$ | $73.97\\pm 0.19$ | 26.03 |\n|30% | $84.17\\pm 0.07$ | $67.89\\pm 0.13$ | 22.51|\n\nWhat we can conclude from the results is that masking ratio can influence the effectiveness of overall performance since the masking strategy is associated with the relatedness information learning. When masking more edges, the performance of citation prediction will inevitably decrease since the complexity of recovering can be higher. However, further experimentation revealed that 5% masking for citation prediction produced a result of $70.21\\pm 0.11$, suggesting a decrease due to the risk of overfitting.  We can also notice that the performance of other tasks is influenced since there can be semantic information lost and noise introduction due to the update of hidden representations which further validates the necessity of Eq. 9. We will enhance the analysis writing and include this part in the revision.\n\n**In responding to the “Reasons To Reject”:**\n\n**1.** GNN and Message Passing Mechanism Design:\n\n**A:** Please refer to the answer for **Q B**.\n\n**2.** Insufficient Experiments:\n\n**A:** We appreciate your constructive feedback regarding the experiments. To address your concern, we further conducted supplementary experiments beyond the basslines chosen in our paper, the results can be seen as follows:\n| Model |  | Multi-XScience |  |  | Bio-Sci |  |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n|  | ROUGE-1 | ROUGE-2 | ROUGE-L | ROUGE-1 | ROUGE-2 | ROUGE-L|\n| KGSum | 30.16 | 6.24 | 25.37 | 34.98 | 6.91 | 30.07 |\n| $SMRC^{2}$ | **31.47** | **6.73** | **26.03** | **36.23** | **7.31** | **30.63** |\n\nWe follow the suggestions of **Q F** from Reviewer FNgn. KGSum is a KG-enhanced method for Multi-Document Summarization task. Its original paper conducted experiments on Multi-XScience (we run it again on the Multi-XScience in terms of fair comparisons), we further run the code on our Bio-Sci. As can be seen above, it is evident that our method surpasses the other baselines which we attribute to the fact that our method leverages a Wasserstein Constraint for modeling the interactions between entity-relation graph and semantic text information entity while KGSum learns only from salient entity-sentence graph. We will host additional narrative to enhance the experiment section.\n\n**3.** No dataset details.\n\n**A:** We have included the statistics in the Table 8 in Appendix and elaborate them in data section. The number for nodes, edges and avg. node degree are 32,330, 222,652 and 13.7, respectively. The comparisons against two benchmark datasets are displayed in the Appendix and the meta-data. We also detail construction methods in Section 4.1. Because of space limitations, we cannot put these details in the main text, but we will add more data information to the appendix in the revised version and more detailed data description will be available in the project web page.\n\n**Reference:**\n\n[1] Qiaoyu Tan et al., MGAE: Masked Autoencoders for Self-Supervised Learning on Graphs.\n[2] Hamilton, William L., Graph Representation Learning. Journal of Synthesis Lectures on Artificial Intelligence and Machine Learning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Representation learning for multi-scientific documents is a research direction of interest in the NLP community, and this work may be of interest to many related researchers.\n\n- This paper is easy to follow. In describing the methods they use, the authors have analysed their reasons for adopting them and compared the advantages and disadvantages of other methods. In addition, the authors provide a detailed and comprehensive analysis of their experimental results, making it easy for the reader to understand the conclusions reached by the authors.\n\n- The authors introduce the *Wasserstein Constraint* for multi-view semantic-level representation learning, which is experimentally demonstrated to have considerable improvement over KL or JS divergence. This discovery is a noteworthy contibution. It is possibly a worthwhile practice for scientific document representation learning after passing further validation.\n\n- The authors provide a new Bio-Sci dataset of high quality in this paper, which is valuable for the evaluation of research in this field.", "reasons_to_reject": "- Some of the detailed information in this paper is not provided in its entirety as the authors describe their methods. For example, when using GNN for relatedness information learning in *Section 3.2*, how exactly is the message passing mechanism of GNN designed?\n\n- The experiments conducted in this paper seem to me to be insufficient to some extent. Firstly, the experiments carried out by the authors compare few baseline methods and do not have more recent work from recent times, more experimental methods should have been included for comparison. Also, more ablation experiments should have been conducted to demonstrate the effectiveness of some of the strategies proposed by the authors (e.g., the masking strategy).\n\n- The authors did not show and analyse the statistics of the datasets used in the article. What are the specific domains of these datasets? How much literature and citations are included? These are important to analyse the scope of applicability of the methodology of this article and to analyse the reliability of the experimental results.", "questions_to_authors": "A: Equation **(9)** proposed by the authors in *Section 3.2* seems to compute the cross-correlation of pairs of nodes at the same level and then concatenate them together because *directly applying the final-layer hidden representation can be difficult for producing the reconstruct matrix*. However, it seems that applying the last level of hidden representation directly is also feasible in this context, and the approach of Eq. **(9)** introduces more unnecessary complexity, can the authors elaborate on the reasons for doing so?\n\nB: How exactly do the GNN used by the authors in Section 3.2 perform message passing?\n\nC: Have the authors conducted experiments to discuss the effectiveness of the masking strategies? What are the specific results?", "paper_topic_and_main_contributions": "This paper is about representation learning in scientific literature at the multi-document level. The authors propose a novel approach called SMRC$^2$ (Scientific Multi-Literature Representation Learning with Content- and Topology-Aware Representation Learning). They propose an approach that combines both semantic content and topological relatedness information.\n\nThe main contributions of this paper are as follows:\n- Introducing SMRC$^2$, which effectively combines semantic content and topological relatedness to create a superior learning space for scientific literature representation learning.\n- The authors introduce Wasserstein distance to address the imbalance learning problem during the semantic learning stage.\n- Experiments conducted by the authors demonstrate that the SMRC$^2$ method achieves excellent results.\n- The authors release a new dataset to encourage further exploration of multi-document representation learning in scientific literature mining.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TncZ4G8zHh", "reviewer": "Reviewer_phqf", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your suggestions, we highly appreciate your time and efforts spent on reviewing our paper. To address your concerns for our work, we will further explore the potential for reducing the computation requirement and flexibility in the continuous work. For example, document hyperlinks can also be used to model the multi-doc level graph and the pattern can be different from citation relations. Also, cascade training schema and the power of large language models can be used to boost our model. Thanks again!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Beside text embedding using language model, this paper introduces entity-relation graph representation, and relatedness representation to form the multi-doc graph. In text presentation, Wasserstein Distance is introduced to minimize the distance between entity-level and text-level representation. The loss contains loss functions for both document-level semantic representation learning and relatedness representation. The results on three different tasks shows the efficiency of this model.", "reasons_to_reject": "The scope of this model is limited, since it requires both document types and citation networks among documents.", "questions_to_authors": "", "paper_topic_and_main_contributions": "To represent multi-document texts in different levels, this paper proposes a new document representation learning method, which contains sementic representation with entity relations for each document and constructs a multi-doc graph based on relatedness information in document citation networks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "b7jTn6uMrC", "reviewer": "Reviewer_FNgn", "summary": "", "questions": "Question A: Is the goal to model multiple documents only in the biomedical domain? If not, why not compare against well-known scientific benchmarks like S2ORC[1]?\n\nQuestion B: What are the documents labels (mu(y)), as mentioned in line 327, and how are they obtained?\n\nQuestion C: What is the motivation behind creating a new dataset? How are the 10 papers in the paper group in the Bio-Sci dataset selected? How big is the citation graph, and how is diversity maintained? Details around dataset creation, methodology, correctness, comparison with existing benchmarks, and thorough analysis of citation relations are necessary for understanding the main contributions of the paper.\n\nQuestion D: As mentioned in line 398, why is the VGAE trained in a self-supervised manner, and how does the approach exactly differ from Hou et al. 2022's method?\n\nQuestion E: What is the dimension of the representation learned for each paper in the group? How big is the unified representation (H_m, as mentioned in line 423)? Did concatenation of h_i's give the best representation? Detailed analysis of the learned unified representation is required to understand the effectiveness of the \"related information learning\" module.\n\nQuestion F: The baselines chosen on the Multi-XScience dataset seem old. Why have recent models like KGSum[2] not been chosen?\n\nQuestion G: In section 4.5, how is the compactness of the t-SNE clusters determined? There are no numbers to support the claim. \n\nQuestion H: How did the SMRC model perform with KL divergence? Giving explicit numbers to showcase the effectiveness of Wasserstein distance would be very useful.\n\n-------\n\n[1] Lo, Kyle, et al. \"S2ORC: The Semantic Scholar Open Research Corpus.\" Annual Meeting of the Association for Computational Linguistics (2020).\n\n[2] Wang, Pancheng, Shasha Li, Kunyuan Pang, Liangliang He, Dong Li, Jintao Tang and Ting Wang. \"Multi-Document Scientific Summarization from a Knowledge Graph-Centric View.\" International Conference on Computational Linguistics (2022).", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for taking the time to review our work and recognize the value of the insightful comments you have made. We are committed to addressing the issues you have noted and endeavoring to assure the quality of this study.\n\n**Q A/C:** Is the goal only in biomedical domain, why not use another dataset such as S2ORC? What's the motivation of creating new datasets? (Statistics Details)\n\n**A:** The proposed method is adaptive to different domains. First, our method is extensible and not depend on any domain-specific graph and document. In the Section 4.1, we have evaluated our method on Ogbn-Arxiv, Ogbn-Citation2 and Multi-XScience datasets which are from the computer science domain. Second, we introduce our method based on Bio-Sci corpus for better understandings because high-quality graphs are easy to construct from public biomedical knowledge graphs.\n\nWe admit that S2ORC is a high-quality and popular dataset, however, the S2ORC contains 81.1M papers and requires incredible computing and storing resources consumption. Because of limited conditions, we choose Obgn benchmarks and Multi-XScience benchmarks, which have the medium scale and is more suitable for method validation. It should be emphasized that the datasets we used have proved the effectiveness of the method and can be extended to massive data sets. \n\nWe also made task and statistic comparisons with these benchmarks as shown in Table 1 and Table 8, respectively. Our motivation for creating Bio-Sci is that we want to include both content and topology information into learned representation which can be used not only for single-document tasks but also multi-document tasks. To achieve this goal, the dataset should be able to contain both text, citation graph and summary information. Thus, we attempt to create a new dataset which can be easily accessed and used. Our Bio-Sci is a continuous work of our whole project which motivates us to further create this dataset for the community who have interests in BioNLP as well as Scientific NLP. We highly appreciate your suggestion for using S2ORC. We indeed considered utilizing this high-quality dataset, however, the overall complexity for obtaining and processing this dataset can be too high for us since it contains 81.1M papers and we need to construct a citation graph for the whole collection. These discouraged us for giving up S2ORC and turned to a replacement of which Obgn benchmarks contains both text and citation information, Multi-XScience benchmarks provide summary information. We highly appreciate your suggestions and will try our best for exploring the usage of S2ORC dataset! \n\n**Q B:** Document labels description problem\n\n**A:** As indicated in line 488, the document labels are their MeSH terms. The Mesh Terms are the category entries defined by PubMed. Specifically, each paper in PubMed is associated with its Mesh Term, we use off-the-self tool called pubmed_parser (https://github.com/titipata/pubmed_parser) to obtain the mesh term and put the terms as the document labels.\n\n**Q D:** Why VGAE is trained in a self-supervised manner, different from Hou et al. 2022?\n\n**A:** As indicated in line 385, our assumption here is that the citation relation between two papers can be inferred by examining their respective contents, corresponding with the standard approach in GNN for inferring edge connections based on node features. We have the citation graph of given paper group and ideally the topology information can be learned through enabling VGAE to learn the invisible citation relation (citation masking). To further reduce the potential noise, we also use pair-wise node cross-relation representation in each layer instead of last-layer hidden representation to enable sure the shared pattern between node i and its neighbors can be learned (please refer to the answer to Q B for reviewer LfTX). As indicated in line 395-398, the main difference between our methodology and Hou et al. 2022 is that they mask a subset of node features and the corresponding hidden representation and train their model in an unsupervised setting which back for great performance of node classification task as can be seen in the Section 3.2 of its original paper. However, our node features are ensured by the joint text- and entity-level representation learned under Wasserstein Distance constraint and thus we resort to masking the edges to enable the topology information learning. By different masking strategies and pair-wise cross relation representation usage, consequently the loss function and the learning goal for decoder is also different. We will provide additional narrative to make this content more noticeable. \n\n**Q E:** Dimension of the representation learned? Is concatenation the best configuration?\n\n**A:** As indicated in the Table 6, the dimension of the representation for each paper is 512. The unified representation can be a matrix of 10x512. We did try attention mechanism for combinations of $h_i$, however, the results for multi-doc task (i.e., multi-doc summarization) decreased to 20.01 which is not comparable with 26.03 of using concatenation. We thought that is because the attention multiplication can introduce unnecessary/irrelevant noises to the shared patterns learned by the pair-wise cross-correlation operation (Please refer to the answer to **Q A** for reviewer LfTX). We further replace the pair-wise cross-correlation with last layer's hidden representation with attention combination, the result for multi-doc summarization is 25.31 which is close to 26.03 and further confirm our assumption that the cross-correlation computation plays similar/better role than attention mechanism. We also tried using mean pooling for combinations of $h_i$ and the result of 22.51 is also not as good as the settings reported in our paper. We may can't conclude that with the assistance of cross-correlation computation, concatenation is the best for forming unified representations. But it is the most computation-effective way to obtain the best performance so far. Due to the time limitation, we only test the attention mechanism and mean pooling methods, however, we will further try other methods for evaluate the effectiveness of the whole frame. Many thanks for your useful suggestions, we will include these results and analysis in our revision.\n\n**Q F:** Baseline is old for MDS, why not included KGSum?\n\n**A:** Following your suggestions, we compared the proposed method with KGSum on Multi-XScience and Bio-Sci. As following, the experimental results on two public datasets show that our method performs outperforms KGSum. We further check their method and attribute these results to the fact that our method leverages a Wasserstein Constraint for modeling the interactions between entity-relation graph and semantic text information entity while KGSum only learns from salient entity-sentence graph.\n| Model |  | Multi-XScience |  |  | Bio-Sci |  |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n|  | ROUGE-1 | ROUGE-2 | ROUGE-L | ROUGE-1 | ROUGE-2 | ROUGE-L |\n| KGSum | 30.16 | 6.24 | 25.37 | 34.98 | 6.91 | 30.07 |\n| SMRC2 | 31.47 | 6.73 | 26.03 | 36.23 | 7.31 | 30.63 |\n\n**Q G:** How the t-SNE's parameters are chosen?\n\n**A:** The parameters are set experimentally. See the details in https://projector.tensorflow.org/, I have uploaded the learned representation of 100 samples to this tool and tuned the parameters to obtain the best results as shown in Section 4.5. The figure is produced by matplotpib library using the parameters we obtained from the above link.\n\n**Q H:** How the KL/JS is used for SMRC?\n**A:** As shown in line 338 and Eq. 6, once we have the results of $y^T$ and $y^E$, the KL can be detailed as $L(y^T, y^E) = y^T*log(y^T/y^E)$, the ordering can be changed and for JS computation we plus the opposite order together[2]. We highly appreciate your kind suggestions and we did include the analysis to showcase the effectiveness of Wasserstein distance as can be seen in Table 5 and Section 4.4. We evaluate the different distribution alignment methods on the three tasks and among them, model equipped with Wasserstein distance performs the best. \n\n**In Responding to the “Reasons To Reject”:**\n\n**1.** Writing problem\n\n**A:** Many thanks for your suggestions towards our writing issues, we will follow your suggestions and improve our writing in the revision which will include all the suggestions and additional experiments as well as the analysis. Thanks again.\n\n**Reference**\n\n[1] Y Lu, Y Dong, L Charlin. Multi-XScience: A Large-scale Dataset for Extreme Multi-document Summarization of Scientific Articles. Proceedings of the 2020 on Empirical Methods in Natural Language Processing.\n\n[2] https://pytorch.org/docs/stable/generated/torch.nn.KLDivLoss.html", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The SMRC^2 model incorporating content and topological structure is exciting and makes intuitive sense. Using GNNs and LMs to encode input documents shows higher performance on the chosen benchmark. Moreover, applying Wasserstein distance as a constraint to model the output distribution has proven effective in various graph learning theories but has yet to gain much attention in the NLP community. It would be worthwhile to study the application of Wasserstein distance as opposed to KL divergence for NLP tasks.", "reasons_to_reject": "The experiments and analysis of the proposed SMRC^2 are weak. Clear reasoning for the chosen baselines and constructed dataset is missing. The paper was hard to follow, and the central claims of the paper need experimental justification.", "questions_to_authors": "Question A: Is the goal to model multiple documents only in the biomedical domain? If not, why not compare against well-known scientific benchmarks like S2ORC[1]?\n\nQuestion B: What are the documents labels (mu(y)), as mentioned in line 327, and how are they obtained?\n\nQuestion C: What is the motivation behind creating a new dataset? How are the 10 papers in the paper group in the Bio-Sci dataset selected? How big is the citation graph, and how is diversity maintained? Details around dataset creation, methodology, correctness, comparison with existing benchmarks, and thorough analysis of citation relations are necessary for understanding the main contributions of the paper.\n\nQuestion D: As mentioned in line 398, why is the VGAE trained in a self-supervised manner, and how does the approach exactly differ from Hou et al. 2022's method?\n\nQuestion E: What is the dimension of the representation learned for each paper in the group? How big is the unified representation (H_m, as mentioned in line 423)? Did concatenation of h_i's give the best representation? Detailed analysis of the learned unified representation is required to understand the effectiveness of the \"related information learning\" module.\n\nQuestion F: The baselines chosen on the Multi-XScience dataset seem old. Why have recent models like KGSum[2] not been chosen?\n\nQuestion G: In section 4.5, how is the compactness of the t-SNE clusters determined? There are no numbers to support the claim. \n\nQuestion H: How did the SMRC model perform with KL divergence? Giving explicit numbers to showcase the effectiveness of Wasserstein distance would be very useful.\n\n-------\n\n[1] Lo, Kyle, et al. \"S2ORC: The Semantic Scholar Open Research Corpus.\" Annual Meeting of the Association for Computational Linguistics (2020).\n\n[2] Wang, Pancheng, Shasha Li, Kunyuan Pang, Liangliang He, Dong Li, Jintao Tang and Ting Wang. \"Multi-Document Scientific Summarization from a Knowledge Graph-Centric View.\" International Conference on Computational Linguistics (2022).", "paper_topic_and_main_contributions": "The paper is on learning better representations for scientific papers. Specifically, the authors propose the SMRC^2 model that has two main components: 1) semantic information learning phase, where the input documents' salient features are captured using a GNN encoder and LM encoder and jointly trained with Wasserstein distance constraint; 2) relatedness information learning phase, where the input documents are represented using the vectors learned in phase 1 and a multi-document citation graph is created. A GNN decoder predicts the masked edges in the citation graph. Such a pipeline aims to capture the content, and topological information originating from the citation links, to better represent scientific literature.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. The title of the paper is misleading. There is no synthesis of knowledge per se. I believe \"Content and Topology aware representation learning for multiple scientific literatures\" would be more relevant. \n2. Line 013: Say SMRC^2 is a model\n3. The task presented in the paper was tough to follow. The writing of abstract, introduction, and related work can be significantly improved. For example, the abstract mentions that the input is multiple documents; however, it is unclear what the output is. Is it an enriched representation of each input document or only a unified representation of all the documents? Even in the introduction section, this point remains unclear. Only after reading the methodology section can it be inferred that the modeling strategy incorporates both the enriched document representation and the concatenation of all the input document representations to form a unified representation. It might be helpful to give an illustrative example on page 1 and explain the task clearly in the introduction section.\n4. Line 081 introduces the challenge of balancing the learning between GNNs and LMs. Authors should mention the motivation for using GNNs for modeling scientific papers beforehand and then move on to the challenges.\n5. Line 074 typo, \"an NLP model suffers from hallucination problem...\"\n6. Line 076 typo, \"Thus, how to ...\"\n7. Line 151, the \"output representation\" of what? Each input paper or the entire set of papers?\n8. The contributions of the paper part in the Introduction section, from line 165, seem handwavy. For example, concrete details for each point would be necessary. In conceptual, what are these downstream tasks, and in experiments, how much on average is the gain in performance, and why is there a necessity to create a new dataset for this well-known problem?\n9. In related work, the paragraphs are too long to understand. Authors should put considerable effort into clearly comparing their work with existing literature. Similarly, section 3.2 has only a single paragraph spanning over two columns."}}]

Author rebuttal:

Meta-review: This paper introduces SMRC^2, a new method for representation learning for scientific literature. The paper also introduces a new bio-sci dataset for evaluating systems in this field. SMRC^2 is demonstrated to be an effective strategy, useful for downstream tasks.

SMRC^2 incorporates a *Wasserstein constraint*, which the authors empirically demonstrate to perform considerably better than alternatives; this is likely to have implications beyond SMRC^2 itself. 

The paper initially lacked details, including specifications of some model components as well as comparisons to recent baselines; the authors have been diligent in addressing these in their responses, which when incorporated into the main text will much improve the paper. The research presented in this paper is of high quality, but initially held back by a lack of clarity. Two reviewers raised their soundness scores in response to the rebuttals, mainly because additional details around implementation and baselines were provided.
Decision: Accept-Main