Summarizing Multiple Documents with Conversational Structure for Meta-Review Generation
Authors: Miao Li
Eduard Hovy
Jey Han Lau
Venue: EMNLP 2023 Findings (2023)
Keywords: Multi-document Summarization
Text Generation
Multi-task Learning
Meta-review Generation
Inter-document Relationships
TL;DR: Summarizing Multiple Documents with Conversational Structure for Meta-Review Generation

Abstract: We present PeerSum, a novel dataset for generating meta-reviews of scientific papers. The meta-reviews can be interpreted as abstractive summaries of reviews, multi-turn discussions and the paper abstract. These source documents have a rich inter-document relationship with an explicit hierarchical conversational structure, cross-references and (occasionally) conflicting information. To introduce the structural inductive bias into pre-trained language models, we introduce RAMMER (Relationship-aware Multi-task Meta-review Generator), a model that uses sparse attention based on the conversational structure and a multi-task training objective that predicts metadata features (e.g., review ratings). Our experimental results show that RAMMER outperforms other strong baseline models in terms of a suite of automatic evaluation metrics. Further analyses, however, reveal that RAMMER and other models struggle to handle conflicts in source documents, suggesting meta-review generation is a challenging task and a promising avenue for further research.

Summarizing Multiple Documents with Conversational Structure for Meta-Review Generation
Miao Li and Eduard Hovy!.2 and Jey Han Lau'
School of Computing and Information Systems, The University of Melbourne Language Technologies Institute, Carnegie Mellon University miao40student un unimell lb. edu.au. feduard. hovy, laujh) @ @unimelb. edu. au
Abstract
automatically generated meta-review
We present PEERSUM, novel dataset for generating meta-reviews of scientific papers. The meta-reviews can be interpreted as ab- stractive summaries of reviews, multi-turn dis- cussions and the paper abstract. These source documents have rich inter-document relation- ships with an explicit hierarchical conversa- tional structure, cross-references and (occa- sionally) conflicting information. To introduce the structural inductive bias into pre-trained language models, we introduce RAMMER Relaaiionhhipggwar Multi-task Meta-review Generator), model that uses sparse atten- tion based on the conversational structure and a multi-task training objective that predicts metadata features (c.g., review ratings). Our experimental results show that RAMMER out- performs other strong baseline models in terms a suite of automatic evaluation metrics. Fur- ther analyses, however, reveal that RAMMER and other models struggle to handle conflicts in source documents of PEERSUM, suggesting meta-review generation is challenging task and a promising avenue for further research.'
public- review-1
author- comment-1
official- review-1
paper abstract
author- response-1
response-2
response-3
response-4
public- response-1
official responso-1
official- response-2
public author comment thread
paper abstract Uhread
thread
thread
Figure 1: An illustration of the hierarchical conversa- tional structure that PEERSUM features.
et al., 2019). This makes it hard to research inter- document relationship comprehension for informa- tion integration and aggregation in abstractive text summarization.
To enable this, we introduce PeERSUM, an MDS dataset for automatic meta-review generation. We formulate the creation meta-reviews as an ab- stractive MDS task as the meta-reviewer needs to comprehend and carefully summarize information from individual reviews, multi-turn discussions be- tween authors and reviewers and the paper abstract. From application perspective, generating draft meta-reviews could serve reduce the workload of meta-reviewers, as meta-reviewing a highly time- consuming process for many scientific publication venues.
1 Introduction
Text summarization systems need recognize in- teraal relationships among source texts and effec- tively aggregate and process information from them to generate high-quality summaries (El-Kassas et al., 2021). It is particularly challenging in multi- document summarization (MDS) due to the com- plexity of the relationships among (saale)p source documents (Ma al., 2020). However, ex- isting MDS datasets do not provide explicit inter- document relationships among the source docu- ments (Liu et al., 2018; Fabbri et al., 2019; Gha- landari et al., 2020; Lu et al., 2020) although inter- document relationships may also exist nature and should be considered in methodology (Fabbri
PeerSUM features a hierarchical conversational structure among the source documents which in- cludes the reviews, responses and the paper ab- stract in different threads as shown in Figure 1. It has several distinct advantages over existing MDS datasets: (1) we show that the meta-reviews are largely faithful the corresponding source doc- uments despite being highly abstractive; (2) the source documents have rich inter- document rela- tionships with an explicit conversational structure;
'The datasct and code are available at https:// gi github..c. cot.bb..cc//cee wwm/ca/.caaoooooo /w///aaiimi iewiimmmmeeeeeeeee PuerSum

---

(3) the source documents occasionally feature con- flicts which the meta-review needs to handle as reviewers may have disagreement on reviewing a scientific paper, and we explicitly provide indica- tors of conflict relationships along with the dataset; and (4) it has a rich set of metadata, such as review rating/confidence and paper acceptance outcome the latter which can be used for assessing the qual- ity of automatically generated meta-reviews. These make PEERSUM serve as a probe that allows us to understand how machines can reason, aggregate and summarise potentially conflicting opinions.
2 Related Work
2.1 MDS Datasets
There are few popular MDS datasets for ab- stractive summarization in these years, such WCEP (Ghalandari et al., 2020), Multi-News (Fab- bri et al., 2019), Multi-XScience (Lu et al., 2020), and WikiSum (Liu et al., 2018) from news, sci- entific and Wikipedia domains. Multi-XScience is constructed using the related work section of scientific papers, and takes a paragraph of related work as summary for the abstracts of its cited papers. Although the summaries are highly ab- stractive, they are not always reflective of the cited papers - this is attested by the authors' finding that less than half the statements in the summary are grounded by their source documents. WikiSum and WCEP have a similar problem as they augment source documents with retrieved documents and as such they may only be loosely related to the summary. Notably, none of the source documents in these datasets provides any explicit structure of inter-document relationships or confticting infor- mation, although different inter-document relation- ships may exist among source documents in these datasets (Ma et al., 2020). This leads to under- explored research on inter-document relationship comprehension of abstractive summarization mod- els. In the peer-review domain, Shen et al. (2022); Wu et al. (2022) developed datasets for meta-review generation. However, they only consider official reviews, or their datasets do not feature the rich hi- erarchical conversational structure that PEERSUM has.
However, there is limited study on abstractive MDS methods that can recognize relationships among source documents. The most promising approaches are based on graph neural networks (Li et al., 2020, 2023), but they introduce additional trainable parameters, and it is hard to find effec- tive ways to construct graphs to represent source documents. To make pre-trained language models have the comprehension ability of complex rela- tionships among source documents for MDS, we propose R RAMMER, which uses relationship-awan attention manipulation -a lightweight approach to introduce an inductive bias into pre- pre-trained lan- guage models to capture the hierarchical conver- sational structure in the source documents. Con- cretely, RAMMER replaces the full attention mech- anism Transformer (Vaswani et al., 2017) with sparse attention that follows a particular relation- ship in the conversational structure (e.g., the parent- child relation). To further improve the quality of generated meta-reviews by utilising the metadata information, RAMMER is trained with multi-task objective additionally predict source document types, review ratings/confideeces and the paper ac- ceptance outcome.
2.2 Structural Inductive Bias for Summarization
Transformer-based pre-trained language models (PLMs) (Lewis al., 2020; Zhang al., 2020a; Guo et al.. 2022; Phang al., 2022) are the predom- inant approach in abstractive text summarization. However, challenging to incorporate structural information into the input as Transformer is de- signed to process flat text sequences. As such, most studies for MDS treat the input documents long flat string (via concatenation) without any explicit inter-document relationships (Xiao et al., 2022; Guo al., 2022; Phang et al., 2022). To take into account the structural information, most work uses graph neural networks (Li al., 2020; Jin et al., 2020; Cui and Hu, 2021; Li et al., 2023) but it is difficult to construct effective graphs to
We conduct experiments compare the perfor- mance of RAMMER with number of baseline mod- els over automatic evaluation metries including the proposed evaluation metric based on predicting the paper acceptance outcome and human evalua- tion. We found that RAMMER performs strongly, demonstrating the benefits of incorporating of the conversational structure and the metadata. Fur- ther analyses on instances with conflicting source documents, however, reveal that it still struggle to recognise and resolve these conflicts, suggesting that meta-review generation is a challenging task and promising direction for future work.

---

Features
ICLR
NeurIPS
A meta-review (summary) and its correspond- ing source documents (i.e., reviews, discussions and the paper abstract) form a sample in PEER- SUM.* The source documents has an explicit tree-like conversational structure," 5 as illustrated in Figure (a real example is presented in Ap- pendix A). In total, PEERSUM contains 14,993 samples (rain//allidttion//ee)
#samples
9,835
5,158
coofficialrevieen review-thread thread/cluster
3.51
3.67
Mauthor-comment cowment-threddeeeeeecooo thread/cluster
0.59
0.01
#public-review thread/cluster
0.22
0.00
#paper- cosstacc-thhead/chlue
1.0
1.0
Table I: PEERSUM statistics.
11.995/1,499/1,444 for ICLR 2018-2022 and NeurIPS 2021-2022; see Table for some statisties. To summarise, PEER- SUM has seven types of source documents (shown in different colors in Figure 1): (1) official reviews (reviews by assigned reviewers); (2) public reviews (comments by public users); (3) author comments (an overall response by paper authors); (4) offi- cial responses; (5) public responses; (6) author re- sponses within thread; and (7) the paper abstract. also features some metadata for each sample: (I) puper acceptance outcome (accept or reject); and (2) a rating (1-10) and confidence (1-5) for each official review.
represent multiple documents and they introduce additional parameters to the pre-trained language models. Attention manipulation is one approach to introduce structural inductive bias without in- creasing the model size substantially. Studies that take this direction, however, by and large focus incorporating syntax structure of sentences or internal structure of single documents (Bai al.. 2021; Cao and Wang, 2022) rather than higher level inter-document discourse structure. RAMMER inspired by these works, and the novelty is that it uses attention manipulation to capture broader inter-document relationships.
To compare PEERSUM with other MDS datasets, we present some statistics on sample size and docu- ment length for PEERSUM and several other MDS datasets in Table 2.
3
The PEERSUM Dataset
3.1 Dataset Construction
We next present some analyses to understand the degree of conflicts in the source documents, and abstractiveness and faithfulness in the summaries.
PEERSUM constructed using peer-review data scraped from OpenReview for two international conferences computer science: ICLR and NeurIPS. As meta-reviewers are supposed to follow the meta-reviewer guidelines' with comprehending and carefully summarizing information shown in the peer-reviewing web page (the example shown in Appendix A), and observe from example meta- reviews as shown in Table 3 that meta-reviewers are complying with the guidelines, we collate the paper abstract, official/public reviews and multi- turn discussions as the source documents, and use the meta-review as the summary. We note that there may be private discussion among the review- ers and meta-reviewer which may influence the meta-review. However, our understanding is that reviewers advised amend their reviews if such a discussion changes their initial opinion. For this reason, we believe the meta-review reflective of the (observable) reviews, discussions and the paper abstract, and this is empirically validated in Section 3.3.
3.2 Conflicts in Source Documents
One interesting aspect of PEERSUM is that source documents are not only featuring explicit hierar- chical conversational relationships but also present- ing conflicting information viewpoint occasion- ally such as conflicting sentiments shown Table 4. We extract conflicts among source documents based on review ratings in different official reviews. Denoting CF for samples with conflicts where at least one pair of official reviews that have a rat- ing difference 2 4 (otherwise Non-CF), we found that 13.6% of the dataset are CF samples. The meta-reviews for these instances will need han- dle these conflicts. In our experiments (Section 5) we present some results to show whether summa- rization systems are able to recognize and resolve conflicts in these difficult cases.
hnttps://ogr http::s:////eer.eeww..eeo /ttas///ccrr.ee
ACGu i de, ht htt.://ccccoooeeenceeee...e /2022/ /
'Henceforth we use terms summary and meta-review interchangeably in the context discussion PEERSUM. 'The average tree height/width = 3.63/5.31.
AC- Ge-Guudeliine

---

Metric
PEERSUM
WikiSum
Multi-News
WCEP
Multi-XScience
Domain
Peer-review
Wikipedia
News
News
Scientific
#Samples
15,983
1,655,709
56,216
10,200
40,528
#Documenns//ample
10.48
40
2.79
63.38
4.45
#Sentenccs/Document
19.66
2.85
30.40
18.24
7.10
#Tokens/Document
397.32
54.54
690.97
439.24
172.90
#Sentence/Summary
6.51
5.17
10.12
1.44
5.06
#Tokens/Summary
142.74
121.20
241.61
30.53
116.41
Table 2: Statisties of PEERSUM and other MDS datasets.
M1 "This meta-review is written after considering the re- views, the authors' responses, the discussion, and the paper itself."
S1: The approach proposed in the paper seemss to be small incremental change on top of the previous GNN pre-train work. The novelty aspect is low.
S2: The main contribution is the novel pre-training strategy introduced. The work has potential high im- puct in the research area...
M2
authors made substuntial improvements during the discussion phase ..."
M3
but the har for introducing yet another variant of memory-augee augmented neural nets has been significantly raised, which sentiment shared by the reviewers. author's response not swayed reviewers' opinion, am sticking to reviewers' decisions.
SI: Introduction section is not well-written.
P2
S2: This paper well written and looks correct.
Table 4: Two example pairs (P1 and P2) of contradic- tory sentiments between official reviewers for two sci- entific papers, and italic texts are conflicts between the sentences (SI and S2).
Table 3: Three example meta-reviews (MI, M2, and M3) of meta-review sentences to show that the meta- reviewer is trying to comprehend and carefully summa- rize information from the paper, the individual reviews, and multi-turn discussions between paper authors and reviewers.
Dataset
Unigram
Bigram
Trigram
PEERSUM
28.28
82.31
92.95
3.3 Abstractiveness and Faithfulness of Summaries
WikiSum
22.75
63.55
79.34
Multi-News
23.49
66.10
82.01
WCEP
5.25
37.62
65.27
Abstractiveness - the degree that summary con- tains novel word choices and paraphrases is an important quality for MDS datasets. Following Fabbri et al. (2019) and Ghalandari al. (2020), we preprocess source documents and summaries with lemmatisation and stop-word removal, and calculate the percentage of unigrams, bigrams, and trigrams in the summaries that are not found in the source documents and present the results in Ta- ble 5. We see that PEERSUM summaries are highly abstractive, particularly for bigrams and trigrams. Although
MultiXXScience
44.09
86.54
96.40
Table 5: Percentage novel n- n-grams the summaries of different datasets.
and 35 CF) highlight text spans in the summary that can be semantically anchored the source documents (full instructions for the task is given in Appendix C) Based on the results in Table 6, we can see that for samples with non-confficting reviews (first row), almost 80% of the words in the meta-reviews are grounded in the source doc- uments. Although this percentage drops to 72% when are looking at the more difficult cases with conflicting reviews (second row), our analy- sis reveals that the meta-reviews are by and large faithful, indicating that they function as a good summary of the reviews, discussions and the paper abstract.
is the most abstractive, discussed in Section 2.1 the summaries are not always reflective of the content of source docu- ments.
To understand whether the summarics in PEER- SUM are faithful the source documents, i.e.. whether the statements/assertionss in the meta- review are grounded in the source documents, we perform manual analysis to validate this. recruit 10 volunteers to annotate 60 samples (25 Non-CE

---

reviewer confidences
document types
review ratingss
Regression Layer
Classification Layer
Regression Layer
acceptance
He
<doe- -
We
<doc-sep>
Based
cdoc-sepa
This
<doc-sep>
Thank
<doc-sep
Generation
<doc
116
<doc-sep-
Thank
sep
Thanks
<doc
We
<doc-sep>
Classification Layer
H
Transformer Decoder
Relationship- aware Sparse Encoder
meta-review
Y
<doc-Seps
<doc-sepi
<doc-seps
<doc-sep>
Thanks
<doc-seps
Moreover
<doc-sep>
<doc-sep
Trank
<doc-sep
Thanks
<doc-sep>
<doc-sep>
<B>
Our
H
Glad
Your
full attetion within source document
relationship-awar sparse attention
paper
author comment--
oftiaa
author reeponse-1
authar response-2
pubric
author resporsee3
public
author
offical reeporse-1
response-1
resporie-4
Figure 2: There are six main components in the RAMMER architecture: (1) relationship-award aware sparse encoder to encode source documents; (2) vanilla Transformer decoder to generate meta- reviewe: (3) two different regression layers predict reviewer confidences and review ratings; (4) two classification layers to predict the type of each source document and the paper acceptance outcome. There are two different types of attention mechanisms: 'full attention within a document' denotes that there are attention calculation between tokens within a document and "relationship-wware sparse attention' denotes that there are attention calculation between tokens in documents only when there is connection between the two documents the corresponding tree-like hierarchical structure.
Mean Variance
Anchored Words (%)
4.1 Relationship-Aware Sparse Attention
Data
#Samples
To explicitly incorporate hierarchical relation- ships among source documents into the pre- trained Transformer model, we propose an encoder with relationship-aware sparse attention (RSAttn), which improves the summarization performance with the introduction of structural inductive bias. The main idea is to use sparse attention by sidering hierarchical conversational relationships among source documents.
Non-CF
25
0.717
79.67%
CF
6.668
72.74%
Table 6: Percentage of words in the meta-review grounded the source documents in CF and Non-CF samples. "Mean Variance" denotes the average of rat- ing variance of official reviews.
4 The RAMMER Model
Based on the tree-like hierarchical conversa- tional structure and the nature of meta-review gen- eration, we extract seven types of relationships which are represented as matrices (an clement is if one document is connected to another, else O):
We now describe RAMMER, meta-review genera- tion model that captures the conversational struc- ture in the source documents (Section 4.1) and uses a multi-task objective to leverage metadata information (Section 4.2). RAMMER built on an encoder-decoder PLM to automatically generate sunmary/meta--review Y from cluster of source documents D; its overall architecture is presented in Figure 2. The input RAMMER is the concate- nation of all source documents (D) and we insert a delimiter <doc-sep> to denote the start of each document."
R1, ancestor-1 which captures the parent asymmetric relationship and the attention from the parent document towards to the cur- rent one:
R2, ancestor-all which captures the ancestor asymmetric relationship as the ancestor docu- ments would provide context for the current one:
R3. descendant-1 which captures child asym- metric relationship and the attention from the child document towards to the current one;
"All volunteers are PhD students who major in computer science and are familiar with peer-reviewing. FFor PL.Ms that do not have <dne-sep> in their tokenizers we use </s instcad.
R4, descendant-ee all which captures descen-

---

dant asymmetric relationship as sometimes concerns would be addressed after the discus- sion descendant documents;
train RAMMER on four auxiliary tasks. We use the output embeddings from the encoder to predict review ratings/confidences and source document types, and the output embeddings from the decoder to predict the paper acceptance outcome. Formally, the overall training objective is:
Rs, siblings which captures the sibling sym- metric relationship as usually reviewers or the paper authors use sibling documents to pro- vide more complementary information:
Rs document-self which captures the full self-attention among each individual docu- ment as token representations are learned based on a rich context within the document; R7, same-thread which captures the symmet- ric relationship among documents which are in the same thread (source documents in each grey dashed rectangle in Figure 1) as usu- ally documents in the same thread are talking about the same content.
== 0igBy + acLe + ar + taoo + Oaha (2)
where a used to balance different objectives. Lg the standard cross-entropy loss for text generation based on the reference meta-review, (Ce, the mean squared error for predicting the review confi- dence and review rating respectively, and {Cos Ca the cross-entropy loss for predicting the paper ceptance outcome and the source document type respectively. Next, we describe more details about auxiliary tasks for the encoder and the decoder.
Next, we use a weighted combination of these relationship matrices to mask out connections to those source documents not included in any rela- tionships for each source document and scale the at- tention weights. The output of each head in RSAttn in the /-th layer calculated
4.2.1 Encoder Auxiliary Tasks
We use Idd to denote the set of indices contain- ing the special delimiters in the input. Auxiliary objectives of multi-task learning for the encoder are then based on the embeddings of these delim- iters. Denoting the output embeddings produced by RAMMER'S encoder as He, sion layers to predict the review confidences and review ratings f respectively:
Hi = softmax QK RETOE,B R
two regres-
V.
(I)
Vdh
where Q, K. and are representations after the non-linear transformation of H; the output of the previous layer, or X, the output of the embed- ding layer from the input D'' with delimiter tokens; B is a very small-scale trainable balancing weight vector for different relationships initialized with uniform distribution, and different heads have different 33 as different heads in each layer may fo- cus on different relationships: R! automatically extended from R, (if an element of Ry is I, ele-
é; Signoid H )),
(3)
r; = sig
(4)
where I denotes the index of the delimiter token
of the i-th official review, and H denotes the corresponding embedding. Co and L, then computed as:
(5) where mse denotes mean squared error, and and denote the normalised ([0 1]) vector of ground truth review confidences and the review ratings, respectively.
Le = ())
L, = r),
ments of R. from tokens of the p-th document to tokens of the q- one are I, else O.).
To reduce memory consumption, we implement masking with matrix block multiplication instead of whole attention masking matrices, which means that we only calculate attention weights between every two documents that have at least one relation. This makes the model work for long source docu- ments without substantially increasing computation complexity.
To predict the types source documents we apply a classification layer on the contextual em- beddings of its delimiter tokens. The predicted classification distribution O, of the j-th source document is computed as follows:
4.2 Multi-Tast Learning
=ofttmaxMI.P((2
(6)
To utilise metadata information in PFERSUM review rating, review confidence, paper acceptance outcome and source document type (Section 3.1) !)-
where He denotes the embedding of the j-th source document and I is the corresponding index

---

Initialization
R-L
BERTS
ACC
each document and truncate each document based on that length. During training of RAMMER, we use batch size of 128 with gradient accumulation and label smoothing of 0.1 (Müller et al., 2019) 9). We tune RAMMER'S 's a (Section 4.2) using the val- idation partition and the optimal configuration is:
BART
27.51
15.57
0.738
PRIMERA
29.30
13.24
0.745
LED
30.31
17.35
0.759
Table 7: RAMMER performance when initialized with different pre-trained language models.
2, ar 1, 1, Qu 2. indicating that all metadata benefit the final per- formance and the reviewer confidence and paper acceptance outcome are the more important fea- tures. We present more details on training and hyper-parameter configuration in Appendix B.
0g 2,
in Id. The total loss for predicting all the document types, L.. is:
C. [dd
cross-entropy(Op, O,),
5.2 Automatic Evaluation on Generated Meta-Reviews
where Oj is the one-hot embedding of the ground truth document type of the j-th source document.
We evaluate the quality of generated meta-reviews with metrics including ROUGE (Lin and Hovy, 2003), BERTScore (Zhang et al., 2020b)" and UniEval (Zhong et al., 2022) ROUGE and BERTScore measure the lexical overlap between the generated and ground truth summary, but the former uses surface word forms and latter contex- tual embeddings. UniEval achieves fine-grained evaluation for abstractive summarization and it is based on framing evaluation of text generation as a boolean question answering task. As faithfulness and informativeness marization, we only report the evaluation results of "consistency" and "relevance" from UniEval, respectively.
4.2.2 Decoder Auxiliary Tasks
There is only one auxiliary objective for the de- coder, to predict the paper acceptance outcome (accept vs. reject):
á MmPan Ha)), C. cross-entropy(C a),
(8)
(9)
where Hg is the output embeddings from the last layer of the decoder and a is the one-hot embed- ding of the ground truth paper acceptance.
more important to sum-
5 Experiments
5.1 Experimental Setup
In addition to these metrics, we introduce an- other evaluation metric (ACC) based on the meta- data of PEERSUM. It is an alternative reference- free metric that measures how well generated meta- reviews are consistent with the ground truth meta- reviews. To this end, we first fine-tune BERT- based classifier using ground truth meta-reviews and paper acceptance outcomes, and then use this classifier predict the paper acceptance outcome using generated meta-reviews. The idea is that if the generated meta-review is consistent with the ground truth meta-review, the predicted paper ac- ceptance outcome should match the ground truth paper acceptance outcome.
We compare RAMMER with a suite of strong abstractive text summarization models. We have three groups of models that target differ- ent types of summarization." (1) short single- document: BART (Lewis et al., 2020) and PE- GASUS (Zhang et al., 2020a); (2) long single- document: LED (Beltagy et al., 2020) and Pega- susX (Phang et al., 2022); and (3) multi-document: PRIMERA (Xiao et al., 2022). We use the large variant for these models (which have a similar num- ber of parameters). We fine-tune these models on PEERSUM using the default recommended hyper- parameter settings, All models have the same maxi- output tokens (512), but they feature different budgets of the maximum input length. Given se- quence length budget, for each sample we divide the budget by the total number source docu- ments to get the maximum length permitted for
As RAMMER can any encoder-decoder pre- trained models the backbone, we first present val-
For ROUGE-I, we use the summary-level version 'RougeLsum' from https://pypii org.projeecttteeeeoo rouge-score
"I "Following Koto et al. (2020), we use metrics ROUGE BERTScore. 12 Phttps: //gl cww..cco/.ooccc..cccooooooooo..c wwt.:/asoooognmnnggg...o..oo UniEval
All experiments are run on NVIDIA "Wo fine-tune these pre-trained models on PEeRSUM with the Huggingface lihrary (htrs://huggingficc..oo)
GPUs.

---

Morrm(#Params)
Test Data
R-L1
BERTS1
UniEval-Con?
UniEval-Rel1
ACC
BART (406M)
Non-CF
27.50
16.61
72.97
79.87
0.728
PEGASUS (568M)
Non-CF
27.24
14.75
74.52
80.78
0.725
PRIMERA (447M)
Non-CF
28.70
12.67
68,56
82.33
0.725
LED (459M)
Non-CF 29.52
16.59
70.98
82.97
0.748
PegasusX (568M)
Non-CF
29.65
17.36
73.44
82.24
0.745
RAMMER (459M)
Non-CF
30.39*
17.42"
75.07*
83.84
0.768
BART (406M)
26.84
14.89
71.85
78.74
0.683
PEGASUS (568M)
CF
26.77
13.66
73.12
79.49
0.649
PRIMERA (447M)
CF 29.13
12.33
66.85
81.70
0.639
LED (459M)
29.19
15.32
70.04
82.82
0.698
PegasusX (568M)
29.30
15.69
71.33
81.30
0.707
RAMMER (459M)
CF 29.19
15.88
73.21
83.15
0.724
RAMMER (459M)
U Non-CF 30.23
17.21
74.82
83.75
0.762
w/o RSAttn (406M)
CF U NF 29.67
16.88
71.36
83.01
0.758
w/o multi-task (406M)
CF Non-CF
30.27
17.01
72.99
83.57
0.749
Table 8: Performance of summarization models PEERSUM in terms of ROUGE-L FI (R-L., BERTScore Fl (BERTS), UniEval consistency (UniBval-Con) and relevance (UniEval-Red and paper outcome (ACC). Higher value means better performance for all metrics. Results ROUGE-1 and ROUGE-2 which are not present are consistent with that of ROUGE-L. significantly better than others in the group (p-value 0.05).
idation results for RAMMER where is initialised with BART, PRIMERA and LED with different maximum input lengths (1,024, 4,096 and 4,096, respectively) Table 7. We see consistently that the LED variant performs better than the other two, and this helps us choose the LED variant as the backbone of R RAMMER. This also indicates that our idea of Att and multi- task learning with meta- data can also work on other pre-traie trained language models.
attention (RSAttn: Section 4. 1) and multi-task ob- jective (Section 4.2), we also present two RAMMER ablation variants (the last three rows in Table 8). It is an open question which method has more impact, as even though most metrics (R-L, BERTScore and UniEval) seem to indicate RSAttn is the winner, ACC - which we believe is the most reliable met- ric appear suggest otherwise. That said, we they complement with each other and such incorporating both produces the best perfor- mance.
We next compare RAMMER with the baseline text summarization models on the test set in Ta- ble 813 Here also break the test partition into CF and Non-Co CF samples. Broadly speaking, sum- marisation performance across all metrics for is lower than that of Non-CF, confirming our suspi- cion that the CF instances are more difficult to sum- marise. The disparity is especially significant for ACC and BERTScore, suggesting that these two are perhaps the better metrics for evaluate the quality of generated meta-reviews. Comparing RAMMER with the baselines, it is encouraging see that is consistently better (exception: R-L results of most models on CF samples are more or less the same). This demonstrates the benefits of incorporating the conversational structure and metadata in the source documents into pre-trained language models. To better understand the impact of RAMMER'S 's sparse
5.3 Human Evaluation Conflict Recognition and Resolution
To dive deeper into understanding how well these summarization models recognize and resolve flicting information in source documents, we con- duct a human evaluation.
We randomly select 40 CF samples and recruit two volunteers We ask them to first assess whether each ground truth meta-review recognises conflicts, i.c., whether the meta-review discusses that
mentions confticting informatioo//ieewpon are in the official reviews. For each sample, the volunteers are presented with all the source docu- ments and are asked to make a binary judgement about conflict recognition. We found that 23 out of 40 ground truth meta-reviews have successfully
Somoe random examples and corresponding model gener- ations are present in Appendix D.
BBth voluntcers major in computer science and are famil- iar with peer-reviewing.

---

Model
Recognition
Resolution
tion/viewpoints in the source documents, the out- look is less encouraging. We found that RAMMER fail to recognise and resolve these conflicts its reta-reviews in the vast majority of cases, suggest- ing this is a challenging problem and promising avenue for further research.
PRIMERA
3/23
2/23
LED
4/23
4/23
PegasusX
5/23
5/23
RAMMER
8/23
3/23
Table 9: Performances of summarization models on conflict recognition and resolution for CF samples.
Limitations
done this, and next focus assessing generated meta--evviww for these remaining 23 samples. For these 23 samples, we ask the volunteers to assess conflict recognition for generated meta- reviews. Additionally, we also ask them to judge (binary judgement) whether the generated meta- review resolves the conflicts in similar manner consistent with the ground truth meta-review. Con- flict recognition and revolution results for RAM- MER and three other baselines are presented Ta- ble 9. In terms of recognition, relatively speaking RAMMER does better than the baselines which is encouraging, but ultimately it still fails to recognise conflicts in majority of the samples. When comes to conflict resolution, all the models perform very poorly, indicating the challenging nature of resolv- ing conflicts in source documents of PEERSUM.
Our work frames meta-review generation as an MDS problem, but one could argue that writing a meta-review requires not just summarising key points from the reviews, discussions and the paper abstract but also wisdom from the meta-reviewer to judge opinions. We do not disagree, and to un- derstand the extent to which the meta-review can be "generated" based on the source documents we conduct human assessment (Section 3.3) to vali- date this. While the results are encouraging (as we found that most of the content the meta- reviews are grounded the source documents) the approach we took a simple one, and the assess- ment task can be further improved by decomposing it into subtasks that are more objective (e.g., by explicitly asking annotators to link statements in the meta-reviews to sentences in the source docu- ments).
6 Conclusion
In the age of ChatGPT and large language mod- els, there also lack of inclusion of larger mod- els for comparison. We do not believe it makes sense to include closed-source models such as Chat- GPT for comparison (as is very possible that they have been trained on OpenReview data), but it could be interesting to experiment with large open- source models such OPT (Zhang et al., 2022), LLaMA (Touvron et al., 2023) or Falcon (Al- mazrouei al., 2023). We contend, however, the results we present constitute preliminary results, and that it could be promising direction to explore how RAMMER'S 's RSAttn can be adapted for large autoregressive models.
We introduce PEERSUM, an MDS dataset for meta- review generation. PEERSUM is unique in that the summaries (meta-reviews) are grounded in the source documents despite being highly abstrac- tive, has a rich set of metadata and explicit inter-document structure, and features explicit conflicting information in source documents that the summaries have to handle. In terms of mod- elling, we propose RAMMER, approach that extends Transformer- based pre-trained encoder- decoder models to capture inter-document relation- ships (through the sparse attention) and metadata information (through the multi-task objective). Al- though RAMMER is designed for meta-review gen- eration here, our approach of manipulating atten- tion to incorporate the input structure can be easily adapted to other tasks where the input has inter- document relationships. Compared with baselines over suite of automatic metrics and human evalua- tion, we found that RAMMER performs favourably, outperforming most strong baselines consistently. That said, when we assess how well RAMMER does for situations where there are conflicting informa-
Lastly, we only consider explicit conversational structure in this paper. As our results show, in- corporating such structure only helps to recognise conflicts to some degree but not for resolving them. It would be fascinating test if incorporating im- plicir structure, such argument and discourse links, would help. This is not explored in this paper, but it would not be difficult to adapt our methods to incorporate these structures.

---

References
Miao Li, Jianzhong Qi, and Jey Han Lau. 2023. Com- pressed heterogeneous graph for abstractive multi- document summarization. In AAAI.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Al- shamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Hes- low, Julien Launay. Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023, Falcon-4OB: an open large language model with of-teo performance.
Wei Li, Xinyan Xiao, Jiachen Liu, Hua Wu, Haifeng Wang. and Junping Du. 2020. Leveraging graph improve abstractive multi-document summarization. In ACL, pages 6232-6243.
Chin-Yew Lin and Eduard H. Hovy. 2003. Auto- matic evaluation of summaries using n-gram occurrence statisties. HLT-NAACL, pages 71-78.
Jiangang Bai, Yujing Wang, Yiren Chen, Yaming Yang. Jing Bai, Jing Yu, and Yunhai Tong. 2021. Syntax- bert: Improving pre-trained transformers with syn- tax trees. In EACL. pages 3011-3020.
Peter Liu, Mohammad Salch. Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summariz- ing long sequences. In ICLR.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. CoRR, abs/2004.05150.
Shuyang Cao and Lu Wang. 2022. HIBRIDS: Atten- tion with hierarchical biases for structure- long document summarization. In ACI.
Yao Lu, Yue Dong, and Laurent Charlin. 2020. Multi- xscience: A large-scale dataset for extreme multi- document summarization of scientific articles. In
EMNLP, pages 8068-8074.
Peng Cui and Le Hu. 2021. Topic-guided abstrac- tive multi-document summarization. In Findings of EMNLP, pages 1463-1472.
Congbo Ma, Wei Emma Zhang, Mingyu Guo.
Hu Wang, and Quan 7. Sheng. 2020. Multi- document summarization via deep learning tech- niques: survey. CoRR, abs/201 1122011..44 1.04843.
Wafaa S. El-Kassas, Cherif R. Salama, Ahmed A. Rafea, and Hoda K. Mohamed. 2021. Automatic text summarization: A comprehensive survey. Ex- pert Systems with Applications, 165:1 13679.
Rafael Müller, Simon Kornblith, and Geoffrey E. Hin- ton. 2019. When does label smoothing help? NeurlPS, pages 4696-4705.
Alexander R. Fabbri, Irene Li, Tianwei She, Suyi Li. and Dragomir R. Radev. 2019. Multi-news: A large- scale multi-document summarization dataset and ab- stractive hierarchical model. In ACL, pages 1074- 1084.
Jason Phang. Yao Zhao, and Peter J. Liu. 2022. Inves- tigating efficiently extending transformers for long imput summarization. CoRR, abs/2208.0434.
Chenhui Shen, Liying Cheng. Ran Zhou, Lidong Bing, Yang You, Luo Si. 2022. Mred: A meta-review dataset for cructure-controllie Findings of ACL, pages 2235.
Demian Gholipour Ghalandari, Chris Hokamp, Nghia The Pham, John Glover, and Georgiana Ifrim. 2020. large-scale multi-document summarization dataset from the wikipedia current events portal. In ACL, pages 1302-1305
text generation. In
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez. Armand Joulin, Edouard Grave, and Guillaume Lample. 2023, Llama: Open and efficient foundation language mod- els. CoRR, abs/2302. 13971.
Mandy Guo, Joshua Ainslie, David C. Uthus, Santi- ago Ontañón, Jianmo Ni. Yun- Hun--suan Sung, and Yin- fei Yang. 2022, Longt5: Efficient text-to-text trans- former for long sequences. In Findings of NAACL, pages 724-736.
Hanqi Jin, Tianming Wang, and Xiaojun Wan. 2020. Multi-granularity interaction network for extractive and abstractive multi-document summarization. In ACL, pages 6244-6254, 6254.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS, pages 5998-6008 -008.
Fajri Koto, Jey Han Lau, and Timothy Baldwin. 2020. FFCI: framework for interpretable automatic eval- uation of summarization. CoRR, abs/2011.11662.
Po-Cheng Wu, An-Zi Yen, Hen-Hsen Huang, and Hsin- Hsi Chen. 2022. Incorporating peer reviews and rebuttal counter-arguments meta-review genera- tion. In CIKM, pages 2189-2198.
Mike Lewis, Yinhan Liu, Naman Goyal, Mar- jan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zeulemooer.. 2020. BART: denoising sequence-to-sequence pre- training natural language generation, translation, and comprehension. In ACI. pages 1871-7880.
Wen Xiao, Iz Beltagy, Giuseppe Carenini, and Arman Cohan. 2022. PRIMERA: pyramid-based masked sentence pre-training multi-document summa- rization. In ACL, pages 5245-5263.

---

Jingqing Zhang, Yao Zhao, Mohammad Salch, and Pe- ter Liu. 2020a. PEGASUS: pre-training with ex- tracted gap-sentences for abstractive summarization. ICML pages 11328-11339.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylow, Myle Ott, Sam Shleifer, Kurt Shus- ter, Daniel Simig. Punit Singh Koura, Anjali Srid- har, Tianlu Wang. and Luke Zettlemoyer. 2022. OPT: open pre-traie pre-trained transformer language models. CoRR, abs/2205.0
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q- Weinberger, and Yoav Arizi. 2020b. Beriscore: Evaluating text generation with BERT. In ICLR.
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. Towards unified multi- dimensional evaluator for text generation. In EMNLP, pages 2023-2038.

---

A Appendix: Different types of source documents in PEERSUM
We present a real PEERSUM example with annotation in Figure 3. Please note that we randomly select this example from PEERSUM and we have removed the author names of the paper and reviewer names, and the content in this real example is not the same as in the synthesized example in Figure 1, while both of the two feature hierarchical inter-document relationships. As shown in Figure 3, automatic meta-review generation is aiming to generate the meta-review automatically based on the paper abstract, official and public reviews and the multi-turn discussions.
GraphGAN: Generating Graphs via Random Walks
Paper abstract thread
frodder becorme
Paper abstract
Wopeis pf
eatera suree damuaii eunntrenan corous
now unge soop soue, srarcca wiaat
campling from oo the are sore to pe
inverrmett canaarile parfomnaco undiiiie wall anomaing was canan,, mus though qpacifuilyy fualy crandd
Shore grenel eott wegh sighlyhur
ICLR 2018 Conference Acceptance Deciston
Meta-review
JCLP Oerfeence Program Chatrs
Computane
braden meryaae
mopi
ecllo suumeieee uttrr to traniiin
the succl perwaing lakkng praphe cane albornatp you mhree meneiie ara neiting limatuee gracaaton siiiing pertiiaal wnternntees
Ho
Author comment thread
Revision summary
Author comment
Cenfernce
Clalmss and evaluation need some work
Official review
Iote 201 conference Ph Aace
(neee fee. 11 a0ro ic.a 2018 cor rua dilo l' levin suaden Q livrycne
Revlow This paprr aetooo Wob sader prnoriing proano beiet nanoenatt propoit por mitor eide combi no
Katny the ge repestins
Confitnne: There
comsain mvlutten cornect very fomilerve Horattr
Official review thread
Authors'' answer pt.
Author response
cou 2018 Comtermen Appertin
cethe 2017 nere Cuskine
Puutes: Oiteryyne
Commine Thank you foryour nevee
the folovinng commanc your corOma
Authors' answer
Author response
Eaglewace Ripeotti Author
2017 300 Contermer hopert Offiae Conment
Tender:
Conmenc Gerana zatic
prepeeeee desectng dromrrptee beveer nes protes erremdly linging in genera lowe the mep permondi tomoppicc toae the mode inal wartero nede permuse and sctiiiealyy g vhith arong inc predeics per eeroranno we sunyy derthrrncid sutampe "eamoraa the ongle di undthal a pup canaan dde ths pleanil paorec
The constructed matrix 5 while training with ED early stop strategy
Public review
2eledor IGu 2018 conferece Pewett rubli Commer Teaters Ceeyane Cananer:: wny atann sq sustt thev ar soyp nemaa corkuadaate rosd paper
Re: The constructed matrix 5 while training with ED early stop strategy 6 X0A Cogloeae Aper Auries 2o 2017 zuk and canternee ofg Commart Baadet: @ Dveyeno
Author response
Public review thread
cxeweees mark you your
question
Public response
2011 2017 isu Cereeeti the tee
sull
Qhivesone
goler Lero brrpnios lov 20 care Lakeerd Moylrt keii ooeeeer
Re: more question
Author response
coefaecee Pheeshhl
a canaeeiio toll dea budce difee
Figure set of source documents and the corresponding meta-review for a scientific paper in PEER

---

B Appendix: Hyper-parameter for fine-tuning pre-trained text summarization models
We present all haper-parameters in Table 10 for models.
Model
Max-len(in/out)
optimizer
Ir
warm up
scheduler
batch size
beam size
length penalty
BART
1,024/512
Adafactor
3e-5
Ok
constant schedule
64
5
1.0
PEGASUS
1,024/512
Adafactor
5e-5
Ok
square root decay
256
0.8
PRIMERA
4,096/512
Adam
3e-5
0.5k
linear decay
1,0
LED
4,096/512
Adam
3e-5
0.2k
linear decay
32
0.8
PegasusX
4,096/512
Adafactor
8c-4
Ok
constant schedule
64
1.0
RAMMER
4,096/512
Adafactor
5e-5
0.2k
linear decay
128
5
1.0
Table 10: Hyper-parametters for all models in experiments.
C
Appendix: Instructions for annotation of PEERSUM quality
Welcome to the annotation project for PeerSum. Please have careful read of the project introduction and task instructions and finish the tasks in the separate document.
Introduction of the project:
To enhance the capabilities of multi-document summarization systems we present PeerSum, a novel dataset for automatically generating meta-reviews of scientific papers based on reviews, multi-turn discussions and the paper abstract in the peer-reviewing process in https://openneieew.nel/ In the reviewing process, all assigned reviewers and public users can give comments to each paper, and then the author of the paper might respond to those comments. There may be couple of rounds of discussions rebuttals during the reviewing process. In the end, the meta-reviewer will write summary of these comments and discussions, to support their final decision on the paper acceptance. Usually, meta-reviewers are supposed to write the meta-review based on summarizing reviews, discussions, and the paper abstract, but they may sometimes draw on some external knowledge which not present in the source documents, such as their own knowledge in the field, reading of the full paper beyond the paper abstract.
The objective of the annotation task is to assess whether the statements/assertionss in meta-reviews are exclusively drawn from the reviews, discussion and the paper abstract which are the source documents in PeerSum. Annotators are expected to help highlight the statements/assertion in meta-reviews that can be drawn from the source documents. Highlighted texts will be heavily dependent source documents and mainly talk about information that is present in the source documents, while they will not be heavily dependent on meta-reviewer's judgements, the meta-reviewer's own knowledge the field, reading of the full paper beyond the paper abstract, or any other external knowledge relative the source documents. Please note that if assertions have very light judgement from meta-reviewers but the content are mostly drawn from source documents, we will prefer to highlight these assertions, as these assertions usually cover much about critical information in the source documents.
Instructions for the task:
Each of you will get 6 samples in total. For each sample:
Please carefully read the source documents including the paper abstract, reviews by different reviewers, and discussions between reviewers and the author (all responses) in the linked OpenReview page.
Please read the meta-review which is the same as the section of Paper Decision in the corresponding OpenReview link, and highlight all assertions or statements (which may be clause, sentence, or a paragraph) which draws knowledge solely from source documents with the colour of blue.
Annotation examples:
Please also carefully read the following two examples of annotation tasks. We also prepare explanations for unhighlighted or highlighted texts following each example, but you do not need to write explanations when annotating.

---

Example one
Source Documents:
Link to httpa//wwwssss/hhhhoo/////////// Hygy01StvH.
Meta-review:
The reviewers have pointed out several major deficiencies of the paper, which the authors decided not
to address.
Example two
Source Documents:
Link to Ottpe//ww..ttoos.......////////a
H1DKN7ZC2. pdf
Meta-review:
Authors present a method for representing DNA sequence reads as one-hot encoded vectors, with genomic context (expected original human sequence), read sequence, and CIGAR string (match operation encoding) concatenated as a single input into the framework. Method is developed on 5 lung cancer patients and 4 melanoma patients. Pros: The approach to feature encoding and network construction for task seems new. The target task is important and may carry significant benefit for healthcare and disease screening. Cons: The number of patients involved in the study is exceedingly small. Though many samples were drawn from these patients, pattern discovery may not be generalizable across larger populations. Though the difficulty acquiring this type of data is noted. - (Significant) Reviewer asked for use of public benchmark dataset, for which authors have declined to use since the benchmark was not targeted toward task of ultra-low VAI's. However, perhaps authors could have sourced genetic data from these recommended public repositories to create synthetic scenarios, which would enable the broader research community to directly compare against the methods presented here. The use of only private datasets is concerning regarding the future impact of this work. (Significant) The concatenation of the rows is slightly confusing. It is unclear why these were concatenated along the column dimension, rather than being imput multiple channels. This question doesn't 't seem be addressed in the paper. Given the pros and cons, the committee recommends this interesting paper for workshop.
Explanations:
"However, perhaps authors could have sourced genetic data from these recommended public repos- itories create synthetic scenarios," is highlighted, because this assertion is logically based recommended public repositories and synthetic scenarios which are from source documents. "which would enable the broader research community to directly compare against the methods presented here. The use only private datasets concerning regarding the future impact of this work." is not highlighted, because this is heavily based on meta-reviewer's own experience in the field or suggestion about impact of the paper.
"This question doesn't seem to be addressed in the paper." is not highlighted, because it is meta-
reviewer's own judgement about the paper.
In "Given the pros and cons, the committee recommends this interesting paper for workshop." which is not highlighted, there is external knowledge about the workshop information. Example three
Source Documents:
Link to OpenReview: https: http:///ww....oooooo//////////a/
ZeE8 1 SFTs1, pdff
Meta-review:
Dear authors, apologize to the authors for insufficient discussion in the discussion period. Thanks for carefully responding to reviewers. Nevertheless, I have read the paper as well, and the situation clear me (even without further discussion). I will summarize what the paper is about, but will instead mention some of the key issues. 1) The proposed idea is simple, and in fact, has been known to me for a number of years. did not think it was worth publishing. This on its own not a reason for rejection, but wanted to mention this anyway to convey the idea that I consider this work very incremental. 2) The

---

idea is not supported by any convergence theory. Hence, it remains a heuristic, which the authors admit. In such a case, the paper should be judged by its practical performance, novelty and efficacy of idcas, and the strength of the empirical results, rather than on the theory. However, these parts of the paper remain lacking compared to the standard one would expect from an ICLR paper. 3) Several elements of the ideas behind this work existed in the literature already (e.g., adaptive quantization, time-varying quantization, ...) Reviewers have noticed this. 4) The authors compare to fixed / mon-adaptive quantization strategies which have already been surpassed in subsequent work. Indeed, QSGD was developed 4 years ago. The quantizers of Horvath et al in the natural compression/natural dithering family have exponentially better variance for any given number of levels. This baseline, which does not use any adaptivity, should be better, believe, to what the author propose. If not, a comparison is needed. 5) FedAvg is not the theoretical nor practical SOTA method for the problem the authors are solving. Faster and more communication efficient methods exist. For example, method based on error feedback (c.g., the works of Stich, Koloskova and others). MARINA method (Gorbunov et al), SCAFFOLD (Karimireddy et al) and so on. All can be combined with quantization. 6) The reviewer who assigned this paper score 8 was least confident. I did not find any comments in the review of this reviewer that would sufficiently justify the high score. The review was brief and not very informative to me as the AC. All other reviewers were inclined to reject the paper. 7) There are issues in the mathematics although the mathematics is simple and not the key of the paper. This needs to be thoroughly revised. Some answers were given in author response. 8) Why should expected variance be good measure? Did you try to break this measure? That is, did you try to construct problems for which this measure would work worse than the worst case variance? Because of the above, and additional reasons mentioned in the reviewers, I have no other option but to reject the paper. Area Chair
Explanations:
In this meta-review, the meta-reviewer write it based on own reading of the full paper. In this kind of cases, meta-reviewers draw on external knowledge, but some of the assertions are still based on source documents, such as "All other reviewers were inclined to reject the paper".
"Dear authors, I apologize to the authors for insufficient discussion in the discussion period. Thanks for carefully responding to reviewers." is not highlighted, because this is coordination words and some own judgements from the meta-reviewer.
"Nevertheless, I have read the paper well, and the situation is clear me (even without further discussion)." is not highlighted, because this is based on meta-reviewer's own reading of the full paper.
"I will not summarize what the paper is about, but will instcad mention some of the key issues." is not highlighted, because this is coordination words from the meta-reviewer.
"I) The proposed idea is simple, and in fact, it has been known to me for number of years. did not think it was worth publishing. This on its own is not reason for rejection, but I wanted to mention this anyway to convey the idea that I consider this work very incremental." is not highlighted, because this is based the meta-reviewer's
experience.
"In such a case, the paper should be judged by its practical performance, novelty and efficacy of ideas, and the strength of the empirical results, rather than on the theory. However, these parts of the paper remain lacking compared to the standard one would expect from an ICLR paper." is not highlighted, because this is the meta-reviewer's experience about the standard of ICLR.
"which have already been surpassed in subsequent work. Indeed, QSGD was developed 4 years ago. The quantizers of Horvath et al in the natural compression/natural dithering family have exponentially better variance for any given number of levels. This baseline, which does not use any adaptivity, should be better, I believe, to what the author propose. If not, comparison is needed." is not highlighted, because this is based on the meta-reviewer's experience in the field.
"5) FedAvg is not the theoretical nor practical SOTA method for the problem the authors are solving. Faster and more communication efficient methods exist. For example, method based on error feedback (c.g., the works of Stich, Koloskova and others), MARINA method (Gorbunov et al), SCAFFOLD (Karimireddy et al) and so on. All can be combined with quantization." is not

---

highlighted, because this based the meta-reviewer's experience in the field.
"I did not find any comments in the review of this reviewer that would sufficiently justify the high score. The review was brief and not very informative to me as the AC." not highlighted, because this is mete-rrviewer's judgement on the review.
"7) There are issues in the mathematics although the mathematics is simple and not the key the paper. This needs to be thoroughly revised.", this is not highlighted because it is based on meta-reviewer's reading of the full paper.
"8) Why should expected variance be good measure? Did you try to break this measure? That is, did you try construct problems for which this measure would work worse than the worst case variance? Because of the above, and", this is not highlighted because is based on the meta-reviewer's own knowledge in the field.
"I have no other option but to reject the paper. Area Chair" is not highlighted, as this the meta-reviewer's s judgement on the paper.
D Appendix: Generated meta-reviews for PEERSUM by different models
We present five groups of example meta-reviews generated by fully-supervised PRIMERA, LED, Pega- susX, and RAMMER Table 11 with the input varying lengths, 1,024, 4,096, and 4,096, respectively, and also ROUGE scores measuring quality of generated meta-reviews in comparison to the ground truth one. These examples are randomly selected from the test set of PEERSUM. although RAMMER outperforms other strong baseline models in terms of evaluation metrics Table 8, the quality of generated meta-reviews still needs to be improved. This further confirms our claim that PEErSUM is really challenging dataset.
clear to see that
pddf.

---

Ground Iruth meta-review
Understanding neural networks once they have been trained is big open problem for machine learning. This manuscript designed graph theoretic and informa- tion theoretic measures aimed at helping us understand community structure and function in trained networks. In particular, they measure community structure (modularity) and entropy for trained networks and related these to the performance of the networks. The manuscript runs experiments with fully connected networks on problems such as MNIST and CIFAR. Both community structure and entropy measures are shown correlate (Spearman and Pearson correlation coefficients) with performance metrics in the networks studied. Reviewers tended to agree that the paper was well written and motivated by an interesting and timely question (understanding trained networks). However, on the whole, most of the reviewers believe that the manuscript is too preliminary for publication at ICLR and agree. A central issue cited by most of the reviewers is that the experiments are performed on small/toy models for small tasks and under particular hyperparameter regimes. It is therefore unclear to what extent the results would generalize to other situations. E.g. would the results hold for larger dataset or for convolutional neural networks? Connected to this complaint, reviewers worry that there is not enough connection to the literature and baseline methods that could be used to predict performance given measures of trained network activity. Even allowing that the observed correlations are true and generalizable, are these measures better than those covered elsewhere in the literature? Additionally problematic, the measures are not theoretically justified either. Thus, we are missing both reasoned arguments for the metrics and robust quantification beyond a limitted experimental setting. One reviewer, Xmnm, is compelled by the work and recommends acceptance. However, they do not present compelling case for acceptance, and even repeat several of the concerns raised by other reviewers. In sum, the work is on an interesting subject and timely, but needs further work to be ready for publication.
PRIMERA
This paper proposes two methods to analyze the behavior of neurons in neural nets. The main idea is to study the neuron activation patterns of classification models and explore if the performance can be explained through neurons' activation behavior. The authors propose two approaches: one that models neurons' activation behavior as a graph and examines whether the neurons form meaningful communities, and the other that examines the predictability of neurons' behavior using entropy.
The paper is interesting, easy to read, and very well-structured. Furthermore, the subject of the paper is very relevant: the exploration of tools to increase the understanding and explainability of neural models. However, several major con- cerns are obvious: (1) The contribution of this paper is not properly compared against existing methods, making it difficult to tell their significance. (2) The scalability of this work is suspicious. There 's no guarantee that proposed acti- vation analysis methods will also show the same correlation tendency in other neural networks with more complicated architectures than multi-layer perceptron networks. (3) Experiments on other various hyperparameters/condiiioo that can affect neuron configurations or learning ability of the neural network(c.g, pruning) could add more validities to their 'comprehensive experimental study'. (4) The suggested explanatory methodologies and approaches were pretty persuasive to explain the network behavior in microscopic context. Nevertheless, the scalability and practicality of the work should be examined carefully. (ROUGE-1: 0.37184, ROUGE-2: 0.04348, ROUGE-L: 0.34657)

---

LED
This paper proposes two new metrics to analyze the behavior of neurons in neural nets. The main idea is to study the neuron activationpaterns of classification models and explore if the performance can be explained through neurons' acti- vation behavior. The authors propose two approaches: one that models neurons' activation behavior as graph and examines whether the neurons form meaningful communities, and the other that examines the predictability of neurons' behavior using entropy.
The reviewers agrec that this is an interesting paper and the authors have done good job in their rebuttal. However, the paper is not ready for publication at ICLR due the following reasons:
There are many existing approaches in explaining deep learning models, such as turning deep models into decision trees in "Distilling a Neural Network Into Soft Decision Tree". Then complexity measurement of the decision tree can be baseline. It would be interesting to see how the techniques presented in this paper compare to other techniques already present in the literature. (ROUGE-1: 0.36032, ROUGE-2: 0.07317, ROUGE-L: 0.34008)
PegasusX
The paper proposes two methods to analyze the behavior of neurons in neural nets. The main idea is to study the neuron activation patterns of classification models and explore if the performance can be explained through neurons' activation behavior. The authors propose two approaches: one that models neurons' Activation behavior as a graph and examines whether the neurons form meaningful communities, and the other that examines the predictability of neurons' behavior using entropy. The paper is interesting, easy to read, and well-structured. However, the paper is below the acceptance threshold. The reviewers raised several concerns, including the lack of a thorough literature survey, the lack and limited experimental validation, and concerns about the scalability of the proposed methods. The AC agrees with the reviewers and recommends rejection.
(ROUGE-1: 0.32743, ROUGE-2: 0.03111, ROUGE-L: 0.29646)
RAMMER
The paper proposes two methods to analyze the behavior of neurons in neural nets. The authors propose two approaches: one that models neurons' activation behavior as a graph and examines whether the neurons form meaningful communities, and the other that examines the predictability of neurons' behavior using entropy. They found that entropy is negatively correlated withtraining accuracy, modularity is correlated with training accuracy, and that the number of well defined neural communities increase with accuracy.
The reviewers agree that the paper is interesting, casy to read, and very well- structured. The main technical contribution of this paper comes from explaining the neural classifiers by combining the graph-theorets and information- theoretic approaches. The reviewers pointed out that the entropy based performance metric is novel contribution, which sets our work apart from previous researches. However. the scalability and practicality of the work should be examined carefully. (ROUGE-1: 0.35319, ROUGE-2: 0.06410, ROUGE-L: ROULUG 0.34043)
Example http:///tt..s../////cc/////////a
pddf

---

Ground Iruth meta-review
This paper studies the method to achieve the batch size-invariant for policy gradient algorithms (PPO, PPG). The paper achieves this by decoupling the proximal policy from the behavior policy. Empirical results show that the methods are somewhat effective at providing batch size invariance.
After reading the authors' feedback, the reviewer discussed the paper and they did not reach a consensus. On the one hand, the rebuttal made some reviewers change their minds who appreciated the explanations provided by the authors and the new Figure that better highlights the batch size invariance property. On the other hand, some reviewers think that there is still significant work to be done to get this paper ready for publication. In particular, is necessary to improve the theoretical analysis and the evaluation of the empirical results.
I encourage the authors to follow the reviewers' suggestions while they will update their paper for a new submission.
PRIMERA
This paper proposes a new method to deal with batch size-invariance for policy optimization algorithms such as PPO and PPG. The key insight is to decouple the proximal policy (used for controlling policy updates) from the behavior policy used for off off-policy corrections). Our experiments help explain why these algo- rithms work, and additionally show how they can make more efficient use of stale data.
The reviewers agree that this paper studies an interesting problem of batch size invariance. The writing is clear, and the method is easy understand. The experimental results are sufficient to demonstrate the effectiveness of the proposed method.
However, the reviewers also agree that the paper could be improved in terms of clarity of presentation and the design of the experiments. In particular, it is important to provide a more rigorous analysis of our ablations, showing that the effects of all of the ablations are statistically significant at the 0.1 (ROUGE-1: 0.48780, ROUGE-2: 0.17439, ROUGE-L: 0.46612)
LED
This paper proposes a method to achieve batch size-invariance for policy opti- mization. The reviewers agree that this is an interesting problem and the paper is well-written. However, the reviewers also agree that the paper lacks theoretical justification, and the claim of the batch size invariant property is vague and weak. The authors are encouraged to improve the paper and resubmit future venue. (ROUGE-1: 0.41284, ROUGE-2: 0.21296, ROUGE-L: 0.40367)
PegasusX
The paper proposes a method achieve batch size invariance for policy optimiza- tion algorithms such as PPO and PPG. The paper achieves this by decoupling the proximal policy from the behavior policy. The experiments demonstrate the effectiveness of the method. The reviewers were unanimous in their opinion that the paper is not ready for publication at ICLR. The main concerns were around the clarity of the presentation and the experimental design. The authors have addressed some of the concerns in their rebuttal, but the paper still needs significant work before it can be accepted.
(ROUGE-1: 0.52846, ROUGE-2: 0.23770, ROUGE-L: 0.50407)

---

RAMMER
The paper proposes a method to deal with batch size-invariance for policy opti- mization algorithms such as PPO and PPG. The paper achieves this by decoupling the proximal policy from the behavior policy. The authors also provide two new algorithm variants to achieve the batch size invariance property.
The reviewers agree that the paper clearly written, is simple to read, and flows well. The experimental results are sufficient to demonstrate the effectiveness of the proposed method. However, the theoretical analysis how this method could be batch-invARRlat in the surrogate loss final performance is not sufficient. The reviewers also point out the lack of theoretical justification, especially the use of the EWMA.
(ROUGE-1: 0.54682, ROURR- 0.24151, ROUGE-L: 0.50187) 0.0
3, httpt::////
Ground meta-review
truth
This paper reveals that popular data poisoning systems, Fawkes and LowKey, fail to effectively protect user privacy in facial recognition. The methods to defend against poisoning attacks are quite simple-yee you can either adaptively tune the face recognition models or just wait for more advanced facial recognition systems. Given these "disappointed" findings from the technical solution side, this paper further argues that legislation may be the only viable solution to prevent abuses of facial recognition.
Overall, all the reviewers highly appreciate the comprehensive and rigorous evalua- tions provided in this paper and enjoy reading it. The biggest concern is raised by the Reviewer 6s7m, given this work fails to discuss/compare to previous works Facial identity anonymizing and the technical contribution is incremental. During the discussion period, all other reviewers reach consensus that 1) facial identity anonymizing is not relevant; and 2) this work make enough contributions and is worthy to be heard by the general community; the Reviewer 6s7m still hold the opposite opinion, but okay for accepting this paper anyway. In the final version, the authors should include all the clarification provided in the discussion period.

---

PRIMERA
This paper studies the effect of data poisoning in face recognition and the relation to the defense techniques. Two solutions of defense are given: oblivious trainer and adaptive trainer. The claim is that, any existing poisoning methods cannot protect the privacy of users in the face images. The paper shows that perturbing facial images does not offer long term security; future systems can still recognize the once perturbed image(s).
The paper is well written and easy to follow. The topic discussed in this paper is of significant importance as the rapid development of deep learning techniques nowadays also poses great threats people's privacy, especially for the face data. Extensive experiments have been conducted to statistically ascertain the authors' claims.
However, the technical novelty very limited. Rather than fancying legislative alternative, a research puper needs to propose technical solution. The original evaluations performed by the Fawkes and LowKey papers were deemed sufficient by reviewers at the time. We believe our paper thus makes an important contribution in demonstrating how such evaluations may have overestimated the protections offered to users. The design of defense strategies that achieve high robustness and accuracy, in Section 3.5. These strategies demonstrate that robustness need not come at a cost in accuracy in all situations, as is often claimed in the adversarial examples literature.
Overall, this a facial privacy analysis with insightful claims, but the presentation and the discussion is very confusing. Thus, we are not able to agree this argument whether is reasonable and solid. The final rating will depend on the authors' feedback.
(ROUGE-1: 0.41410, ROUGE-2: 0.07522, ROUGE-L: 0.38326)
LED
This paper studies the effect of data poisoning in face recognition and the relation to the defense techniques. It shows that the methods fail over time and they do not generalize to future attacks. Two solutions of defense are given: oblivious trainer and adaptive trainer. Both methods successfully defeat two poisoning attack baselines, raising awareness on the inefficiency the poisoning- identity protection mechanism.
The paper is well written and easy to follow. The topic discussed in this paper is of significant importance as the rapid development of deep learning techniques nowadays also poses great threats to people's 's privacy, especially for the face data, It is in interesting paper and has value for the community to consider longer term security measures for facial recognition privacy concerns. The experimental results within expectation and little technical insight is gained.
(ROUGE-1: 0.40606, ROUGE-2: 0.06707, ROUGE-L: 0.35152)
PegasusX
The paper presents new defense against data poisoning attacks. The paper is well written and the analysis is thorough. The authors also provide discussion of the defense and the attack dynamics. The main concern is that the technical novelty limited. The rebuttal did not convince the reviewers. The AC agrees with the reviewers and recommends acceptance.
(ROUGE-1: 0.29719, ROUGE-2: 0.06478, ROUGE-L: 0.28112)

---

RAMMER
The paper studies the problem of data poisoning face recognition and the relation to the defense techniques. The paper shows that perturbing facial images does not offer long term security; future systems can still recognize the once perturbed image(s). The paper also shows that an "oblivious" model trainer can simply wait for future developments in computer vision to nullify the protection of pictures collected in the past.
The paper is well-written and easy to read. The authors have examined several strategies to enable modern face recognition models to defense attacks from widely used data poisoning methods. However, the technical contribution from this paper is limited as little new insight is provided. The experimental results are within expectation and little technical insight is gained. The main conclusion is that people should not rely on technical solutions to protect users privacy and legislation actions are what is actually needed.
(ROUGE-1: 0,42353, ROUGE-2: 0.08284, ROUGE-L: 0.39412)
Example 4, https: http:///ghhhhhaacccc///ac/a////a http:////abo/maaa///e/me/mllee//
pdf This paper presents method which selects feasible data augmentations suitable for contrastive time series representation learning. The topic in this paper is timely and interesting. One of reviewers did not complete the review, not responding to a few reminders. So, one emergency reviewer, who is an expert in meta- meta-lcarning was added. While there is one review that strongly supports this work, two reviews remained unsupportive after the discussion period ended. I appreciate the authors for making efforts in responding to reviewers' comments. However, after the discussion period, most of reviewers had concerns in this work, pointing out that the technical correctness needs further justification and experiments should be improved. While the idea interesting, the paper not ready for the publication at the current stage. encourage resubmit the paper after addressing these concerns.
Ground meta-review
Iruth
PRIMERA
This paper proposes an information-awan approach to representation learning for time series. The formulation focuses on how to obtain effective data augmentations and addresses the underlying problem from cnormatioo-theorette viewpoints, leading to the two optimization criteria, namely, high fidelity and high variety. The experimental results on several time series datasets for forecasting and classification show improvements over the methods in comparison.
All reviewers agree that this is very good submission that touches upon kind of neglected and less fancy area of data augmentation for time-series authors propose very neat solution that improves upon the state of the art and demonstrates good performance across datasets and various ablation studies.
(ROUGE-1: 0.31621, ROUGE-2. 0.03187, ROUGE-L: 0.26877)
LED
This paper proposes new approach to data augmentation for time series represen- tation learning. The reviewers agree that the paper well-written and the proposed approach is novel and interesting. However, there are concerns about the novelty of the approach and the significance the results. The authors are encouraged to revise the paper based on the reviewers' comments and resubmit to a future venue. (ROUGE-1: 0.41951, ROUGE-2: 2: 0.15764, ROUGE-L: 0.40000)

---

PegasusX
The paper proposes a new data augmentation approach based on information theory. meta learning approach and an approach to select optimal data augmentation for contrastive learning. The paper well written and the proposed approach is interesting. However, the experimental results are not convincing. The authors have provided a detailed response to the reviews, but the paper still needs more work before publication.
(ROUGE-1: 0.34483, ROUGE-2: 0.09950, ROUGE-L: 0.32512)
RAMMER
This paper proposes a meta-learning approach for data augmentation for time series. The reviewers found the paper to be well-written and well-motivated, and the approach to be novel and interesting. However, the reviewers also found the experimental results to be insufficiently convincing. In particular, some of the analyses and ablations could be more complete, in order truly investigate whether the conclusions are correct.
(ROUGE-1: 0.33010, ROUGE-2: 0.06863, ROUGE-L: 0.33010
Example 5, httpp:://gggttieee
pdf This works presents new and interesting notion of margin for deep neural net- works (that incorporates representation all layers). It then develops generalization bounds based the introduced margin. The reviewers pointed some concerns, including some notation issues, complexity in case of residual networks, removal of exponential dependence on depth, and dependence on hard to compute quantity kkapp'de Some of these concerns were addressed by the authors. At the end, most of the reviewers find the notion of all-layer margin introduced in this paper very novel and promising idea for characterizing generalization in deep networks. Agreeing with reviewers, I recommend accept. However, I request the authors to accommodate remaining comments /concerns raised by R1 in the final version of your paper. In particular, in your response RI you mentioned for one case you saw improvement even with dropout, but that is mentioned in the revision; Please include related details in the draft.
com htt.//aamlml..... http:///aaa///eeeee/////////////
Ground meta-review
truth
PRIMERA
The paper presents a bound on the generalization error of deep network in terms of margin each layer of the network. The starting premise that extending the existing margin generalization bounds to deep networks worsen exponentially with the depth. Recent work which removed that exponential dependency is claimed to require a more involved proof and complicated dependence on input. The paper provides a new bound that is simpler and tighter. A second contribution is to extend their bounds to robust classifier. Since their bounds depend on instance-specific margins, the extension to the robust case is straightforward. Finally, they present new algorithm motivated by their bounds, that maximized margin all layers. They show that the resultant network has much lower error than standard training. The paper well rell-presented and in spite of being theoretical is very nicely de- veloped so that the main contributions come out clearly to non-specialists too. A few minor comments: The inner min in Equation 2.2 seems to be a typo. In Theorem 2.1, there is typo around the definition of g. Below thoerem 2. 1, the phrase "depend on the t-th moment" has "q" undefined. Typo "is has a" in Theorem 3.1 is the the most insightful result, though simple to prove in hindsight. It would greatly help the readers if simple figures are used to explain this insightful result in the final manuscript.
(ROUGE-1: 0.36181, ROUGE-2: 0.07576, ROUGE-L: 0.35176)

---

LED
This paper presents new way measure the margin in the context of deep networks that removes the exponential dependency of depth in the corresponding generalization bounds. The reviewers agree that this is valuable contribution both for theory and practices.
(ROUGE-1: 0.32836, ROUGE-2: 0.10050, ROUGE-L: 0.28856)
PegasusX
The paper presents a new generalization bound for deep neural networks and develops a regularizer which optimize quantities related to the bound and improve generalization eттоr on competitive baselines. The paper is well-written and the main contributions come out clearly to non-specialists. However, the paper still requires rewriting to properly emphasize that it does not avoid exponential dependence depth, but rather avoids explicit dependence. The authors also need to discuss the computability of kappadv.
(ROUGE-1: 0.37975, ROUGE-2: 0.08511, ROUGE-L: 0.32911)
RAMMER
The paper shows that the all-layer margin has clear and direct relationship with generalization for deep models. The paper also shows that this bound can be adapted to adversarial robustness the deep model.
The paper is well-written and well-motivated. The reviewers found the paper to a valuable contribution both for theory and practices.
(ROUGE-I: 0.29358, ROUGE-2: 0.04630, ROUGE-L: 0.27523)
Table 11:
Generated meta-reviews by fully-supervised PRIMERA, LED, PegasusX and RAMMER for random samples, and ROUGE scores measuring the quality of generated meta-reviews in comparison to the ground truth one.


Reviews:
[{"review_id": "zTua6JZnd7", "reviewer": "Reviewer_iSYN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your effort in reviewing the paper. Our responses regarding the issues you pointed out are as follows.\n\n_1. “Results are not strong, and work needs further iterations to show significant impact.”_\n\nThe main contribution of our paper is actually the introduction of the novel multi-document summarization task/dataset with explicit conversational structures and conflicts among source documents and the insights it gave to the multi-document summarization community, e.g., the extent the meta-reviews are grounded in reviews and the difficulty in conflict handling. Understanding inter-document structures and relationships would be a critical capability of ideal multi-document summarization systems, but to the best of our knowledge, there were no multi-document summarization datasets explicitly providing these structures/relationships in the literature. Our work would pave the way for the research on modelling and evaluating the recognition and resolution of inter-document relationships in text summarization. The model we present constitutes only a baseline for the task, and our experimental results demonstrate the difficulty of the task and opportunities for future work.  We present these in both the Introduction and Conclusion sections.\n\n_2. “Need stronger examples on how meta review generation can be used on real world use cases.”_\n\nFrom an application perspective, (1) generating drafts of meta-reviews could serve to reduce the workload of meta-reviewers, as meta-reviewing is a highly time-consuming process for many conferences; (2) summarization for meta-review generation can be transferred to other domains which also feature intricate inter-document relationships, such as literary analysis and exegesis, etc.\n\nFrom a scientific research perspective, the task serves as a probe that allows us to understand how machines can reason, aggregate and summarise potentially conflicting opinions. We will clarify these motivations and add use cases in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Code and dataset will be released for other users to work on top of the current work.\nNew area of Scientific paper meta review generation explored. \nStrong experiment setup and detailed experiment analysis", "reasons_to_reject": "Results are not strong, and work needs further iterations to show significant impact. \nNeed stronger examples on how meta review generation can be used on real world use cases.  \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper present PEERSUM, a data set for generating meta reviews for scientific paper and RAMMER, a Relationship aware multi-task meta review generator model that predicts meta features for a paper. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wtimUROrAb", "reviewer": "Reviewer_ULo8", "summary": "", "questions": "1. Why don't you use the same scheduling strategy (warm-up and scheduler) and decoding strategy (beam size and length penalty) for all baselines?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your great comments. Our responses regarding the issues and questions are as follows.\n\n_1. “The RAMMER Model is quite complex and it can be very difficult to balance between different tasks.”_\n\nAgree. As the meta-data (i.e., reviewer confidences, paper acceptance, document types, and review ratings) all intuitively have an effect on the meta-reviewers’ writing process of meta-reviews, we are trying to incorporate as much meta-data as possible to construct the baseline. We will include more ablation experiments in the revision to better understand the impact of each sub-task and see whether we need all of them. One last comment about training convergence: We observed that the overall performance is not very sensitive to the scaling hyper-parameters for the different sub-tasks, so RAMMER is not a difficult model to train in our practice.\n\n_2. \"There are differences in the hyper-parameter settings from one baseline to another, so the experimental results may have some bias.\" and \"Why don't you use the same scheduling strategy (warm-up and scheduler) and decoding strategy (beam size and length penalty) for all baselines?\"_\n\nWe used the hyperparameter configurations as recommended by the original authors for all baselines (which are optimal settings). We made this choice for two reasons: (1) we want to make sure each model is performing optimally; and (2) we found tuning them did not improve the performance much.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The meta-review generation task is different from other MDS tasks since there are rich structure information and metadata in the source documents.\n2. Their method which uses sparse attention and multi-task learning outperforms other baseline models.\n3. This paper offers some interesting observations, such as the model still struggles in solving conflicts between reviewers.\n4. The source code and dataset are provided, so it can be easy to reproduce the experimental results.", "reasons_to_reject": "1. The RAMMER Model is quite complex and it can be very difficult to balance between different tasks.\n2. There are differences in the hyper-parameter settings from one baseline to another, so the experimental results may have some bias.", "questions_to_authors": "1. Why don't you use the same scheduling strategy (warm-up and scheduler) and decoding strategy (beam size and length penalty) for all baselines?", "paper_topic_and_main_contributions": "This paper proposes a novel dataset called PeerSum which includes abstracts and reviewer comments from ICLR and NeurIPS. The structures within documents and conversations make it different from other MDS problems. This paper also proposed a relationship-aware meta-review generation model with multi-task learning techniques. Their method achieves better performance than other baselines but still can be improved.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uyOs78HWG2", "reviewer": "Reviewer_h33m", "summary": "", "questions": "A. Explanation of the coefficients in Table 1\nB. In relevant MDS papers, it is also mentioned that abstractive summarization needs to consider the inter-document relationship, why you said that other popular MDS datasets all lack inter-document relationships?\nC. Figure 2 is a little confusing. How to explain the arcs of relationship-aware sparse attention? Also, the seven types of relationships in section 4.1, are not explained very clearly enough(why is the parent asymmetric relationship?). \nD. For multi-task design, there is a lack of experiments related to analyzing the role of these auxiliary tasks.\nE. Is there a specific design for conflict information in the dataset in the model？\nF. Suggest hiding the author information of the paper from the case in the appendix\n\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your great comments. Our responses regarding the questions are as follows.\n\n_A.“Explanation of the coefficients in Table 1.”_\n\nAs there are no coefficients in Table 1, we assume that you mean the coefficients in Eq. 1. $\\beta$ denotes learnable weights for different relationships in each attention head, as different relationships intuitively have different importance and effects on the meta-review generation process. We will clarify this in our revision.\n\n_B. “In relevant MDS papers, it is also mentioned that abstractive summarization needs to consider the inter-document relationship, why you said that other popular MDS datasets all lack inter-document relationships?”_\n\nThat is because inter-document relationships are not explicitly provided in those datasets although inter-document relationships may exist in nature and should be considered in methodology, while we have them explicitly in PeerSum (the conversational structure and conflict indicators), which we believe is helpful for research that is trying to model and evaluate the recognition and resolution of inter-document relationships in summarization. We will clarify this in the revision.\n\n_C. “Figure 2 is a little confusing. How to explain the arcs of relationship-aware sparse attention? Also, the seven types of relationships in section 4.1, are not explained very clearly enough(why is the parent asymmetric relationship?).”_\n\nThe arcs of relationship-aware sparse attention denote sparse connections among source documents based on the conversational structure.\n\nRelationships such as parent (ancestor-1) are asymmetric because we would have attention coming towards each node from its parent in the attention design for the relationship. As in the relationships, we use both the parent-to-child (ancestor-1) and child-to-parent (descendant-1) relationships in our design and they are intuitively of different importance and effect; therefore, we model them as asymmetric separately. We will clarify these relationships better by incorporating some diagrams in the revision.\n\n_D. “For multi-task design, there is a lack of experiments related to analyzing the role of these auxiliary tasks.”_\n\nFair point. We will include ablation experiments to analyse the impact of each auxiliary task in the revision.\n\n_E. “Is there a specific design for conflict information in the dataset in the model?”_\n\nBy providing explicit conversational structure to the model (via the sparse attention), we are hoping that it would help the model to recognise the conflict information in the dataset. But as our paper found, although it helps the model to **recognise** them (Table 8), it doesn’t do much to help **resolve** them, demonstrating the challenge and paving the path for future work.\n\n_F. “Suggest hiding the author information of the paper from the case in the appendix”_\n\nThanks for your suggestions. We will do that.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposes a new dataset for analyzing multi-document abstractive summarization, which is interesting and meaningful. The analysis and statistics of the dataset are relatively comprehensive and have high credibility.", "reasons_to_reject": "1. The description of the key relationship-aware sparse attention is unclear, and the model diagram is quite difficult to understand for this part.\n2. Since conflict is an important feature of this data, the proposed model does not clearly reflect the design of conflicts.\n3. No analysis was conducted on the design of the four auxiliary tasks.", "questions_to_authors": "A. Explanation of the coefficients in Table 1\nB. In relevant MDS papers, it is also mentioned that abstractive summarization needs to consider the inter-document relationship, why you said that other popular MDS datasets all lack inter-document relationships?\nC. Figure 2 is a little confusing. How to explain the arcs of relationship-aware sparse attention? Also, the seven types of relationships in section 4.1, are not explained very clearly enough(why is the parent asymmetric relationship?). \nD. For multi-task design, there is a lack of experiments related to analyzing the role of these auxiliary tasks.\nE. Is there a specific design for conflict information in the dataset in the model？\nF. Suggest hiding the author information of the paper from the case in the appendix\n\n", "paper_topic_and_main_contributions": "This paper constructed a new dataset PEERSUM for meta-review generation, which is different from other MDS datasets. This dataset includes inter-document relationships with an explicit hierarchical conversational structure and conflicting information, which is meaningful for paper review and abstractive summarization. In addition, it also proposes a baseline for this dataset, which considers multi-task framework and obtains an effectiveness results.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces PEERSUM, an innovative dataset for generating meta-reviews of scientific papers, and RAMMER, a multi-task meta-review generator model. PEERSUM includes features such as explicit inter-document relationships and conflicting information, enhancing the realism in multi-document summarization. RAMMER utilizes sparse attention and multi-task learning, showing promise compared to baseline models. The paper's strengths include a rigorous experimental setup, code and data sharing commitments, and the potential for advancing meta-review generation. However, reviewers unanimously call for stronger results, real-world use cases, clearer model explanations, and improved analysis of hyper-parameters and auxiliary tasks, indicating areas for enhancement in the paper's overall impact.
Decision: Accept-Findings