A Sentiment Consolidation Framework for Meta-Review Generation (2024)

Abstract: Modern natural language generation systems with Large Language Models (LLMs) exhibit the capability to generate a plausible summary of multiple documents; however, it is uncertain if they truly possess the capability of information consolidation to generate summaries, especially on documents with opinionated information. We focus on meta-review generation, a form of sentiment summarisation for the scientific domain. To make scientific sentiment summarization more grounded, we hypothesize that human meta-reviewers follow a three-layer framework of sentiment consolidation to write meta-reviews. Based on the framework, we propose novel prompting methods for LLMs to generate meta-reviews and evaluation metrics to assess the quality of generated meta-reviews. Our framework is validated empirically as we find that prompting LLMs based on the framework — compared with prompting them with simple instructions — generates better metareviews.1

heading: 1 Introduction
text: Meta-Review
Review-1 (R1)
Clarity R1: Structure,
R2: Presentation, ...
...Review-2 (R2) Response-1 (P1)
Novelty R1: Method,
R1: Evaluation, ...
Advancement R1: Results, P1: Results,
... ...
G en
er at
io n
L ay er C on so lid at io n La ye
r In pu t La ye r
AnalysisGeneration
Clarity: Presentation, Novelty: Method, Advancement: Results, ...
Aggregation
This is because (1) meta-reviewers are supposed to understand not only all the reviews from different reviewers but also the multi-turn discussions between the reviewers and the author and write their comments to support the acceptance decision of the manuscript, (2) the logic of arguments (from reviewers and authors) has to be taken into account to arrive at the final sentiment in the meta-reviews and it is not a matter of majority voting and (3) metareviews have to recognize and resolve conflicts and consensus among reviewers.
In this paper, we hypothesize that human metareviewers follow a three-layer sentiment consolidation framework as shown in Figure 1 to write meta-reviews based on reviews and multi-turn discussions in the peer review process. Human and automatic annotation is then conducted to extract sentiments and expressions on various review facets (e.g., novelty and soundness) from corresponding source documents (i.e., reviews and discussions) and these judgements play a critical role in generating the meta-reviews. We also propose two evaluation metrics which focus on assessing sentiments in generated meta-reviews, and experiments empirically validate our proposed three-layer framework when they are integrated as prompts for LLMs to generate meta-reviews.
Contributions of our paper:
• We hypothesize that human meta-reviewers follow a three-layer sentiment consolidation framework when writing meta-reviews;
• We collect human annotations on meta-reviews and corresponding source documents based on the consolidation framework;
• We propose two automatic metrics (referencefree and reference-based) to evaluate the sentiment in the generated meta-reviews.
• Experiments validate the empirical effectiveness of the framework when we incorporate it as prompts for LLMs to generate meta-reviews.

heading: 2 Related Work
text: In this section, we discuss large-scale information consolidation in abstractive summarization, and automated sentiment summarization.

heading: 2.1 Large-Scale Information Consolidation
text: Natural language generation systems are expected to not only have high-quality generations but also have the ability to comprehend the input informa-
tion, especially for conditional text generation such as multi-document summarization which has to integrate and aggregate information from different source documents (Gehrmann et al., 2023). Most work in the text summarization community only attempts to improve the generation quality of text summarization, such as relevance and faithfulness, without considering the intricate generation process (Phang et al., 2022; El-Kassas et al., 2021; Xiao et al., 2022). For example, Li et al. (2023b) use heterogeneous graphs to represent source documents and borrow the idea of graph compression to train the summarization model to get improvement of the generated summaries. However, it is uncertain if these models truly possess the ability to consolidate information from different source documents.

heading: 2.2 Automated Sentiment Summarization
text: Sentiment summarization aims to summarise the overall sentiment given a set of documents (Hossain et al., 2023). However, most datasets for sentiment summarization are in the product review domain (Amplayo et al., 2021), and scientific sentiment summarization is under-explored. Metareview generation, which is a typical scenario of scientific sentiment summarization, is to automatically generate meta-reviews based on reviews and the multi-turn discussions between reviewers and the author of the corresponding manuscript (Li et al., 2023a). It is mostly modelled as an endto-end task (Bhatia et al., 2020; Wu et al., 2022; Shen et al., 2022; Chan et al., 2020). Although Li et al. (2023a) considered the conversational structure of reviews and discussions, their models do not explain how human meta-reviewers write the metareviews. Different from investigating checklistguided iterative introspection for meta-review generation with prompting (Zeng et al., 2024), our work is based on a three-layer sentiment consolidation framework and focuses on various review facets, and we explicitly investigate the sentiment fusion process which is arguably an important aspect of meta-review generation.

heading: 3 Sentiment Consolidation Over Multiple Opinionated Documents
text: In the following section, we introduce the task of scientific sentiment summarization and our threelayer sentiment consolidation framework in metareview generation, conduct sentiment and expres-
Min Max Average
#Documents/Sample 5 30 12.4 #Words/Sample 1,541 11,901 4,260.9 #Words/Source document 10 1,562 360.5 #Words/Meta-review 16 648 150.9

heading: 3.1 Hierarchical Sentiment Consolidation
text: from different review facets to write their metareview.
To conceptualize this, we propose a three-layer framework, as shown in Figure 1. The three layers include the input layer, the consolidation layer, and the generation layer. The input layer is the input documents of different types: official reviews and multi-turn discussions. The consolidation layer represents how meta-reviewers process the documents: they first identify and extract judgements from different documents, reorganize the judgements based on review facets, and then consolidate the opinions to form the final opinions of each review facet. In the generation layer, the meta-reviewer writes the meta-review to express the final opinions that they have developed from the previous layer.

heading: 3.2 Judgement Identification and Extraction
text: Judgements lay the foundation of our proposed framework and the whole peer review process. A judgement here expresses sentiment on a review facet and it contains several components: Content Expression, Sentiment Expression, Review Facet, Sentiment Level, and Convincingness Level (definitions are shown in Table 1, and an example is given in Appendix Figure 5). To automate judgement identification and extraction, we first conduct human annotation, and then leverage in-context learning of LLMs to perform more (automatic) annotation.
In human annotation, there are three types of documents including meta-reviews, official reviews, and discussions (the same definition used in Li et al. (2023a)) to be annotated. We recruit two annotators7 to do this annotation (annotation instructions and design are detailed in Appendix B). 30 samples (i.e., one sample = one meta-review and its corresponding reviews and discussions) are annotated8,
7The two annotators are senior PhD students who are familiar with the peer-review process.
8Annotating one sample takes about one hour on average and it costs about 60 hours and 2,100 US dollars in total.
and in total, we have 1,812 and 1,744 judgements from the two annotators. The statistics of these 30 samples are presented in Table 2. We present the agreement of the two annotators in Figure 2.9 Generally, we see a moderate to high agreement, suggesting that the annotation task is robust and reproducible.
To get more annotated judgements for further experiments and analysis and investigate whether LLMs can be prompted to identify and extract judgements, we split the annotation task into two sub-tasks, extracting content and sentiment expressions and predicting other components of judgements, and use GPT-4 (OpenAI, 2023) with incontext learning (see full prompts in Appendix D and E respectively for the two sub-tasks).10 We
9Calculation details and more results in terms of both Cohen’s κ and Krippendorff’s α are in Table 11, Table 12 and Table 13 in Appendix C.
10The version of GPT-4 we use is gpt-4-0613.
Facets %Judgements %Documents
Advancement 0.2545 0.8000 Soundness 0.2786 0.7833 Novelty 0.1817 0.6833 Overall 0.1414 0.5833 Clarity 0.1264 0.4500 Compliance 0.0174 0.0667

heading: 3.3 Sentiment Fusion for Consolidation
text: To understand how well the judgements from the source documents (i.e., reviews and discussions) predict the overall sentiments in the meta-reviews for each review facet, we next formulate a text classification task where the output is the sentiment level of a content expression for a review facet in the meta-review, and the input is either: (1) the annotated judgements for the facet from the source documents; or (2) the full text of the source documents. We (zero-shot) prompt GPT-4 (full prompt detailed in Appendix F) with either input to predict 100 randomly sampled human-annotated instances and present the results in Table 5. Using judgements only as input, we see that it works better in 4 out of 6 facets — this preliminary result suggests our framework of extracting these judgements as an intermediate step may help generate better meta-reviews.

heading: 4 Sentiment-Aware Evaluation on Information Consolidation
text: In this section, we focus more on how to evaluate the sentiments of the generated summaries or meta-reviews in meta-review generation based on our proposed framework. We propose FacetEval and FusionEval which are reference-based and reference-free metrics, respectively.

heading: 4.1 Measuring Sentiment Similarity to Human-Written Meta-Review
text: To assess the quality of generated meta-reviews, we propose a reference-based evaluation metric, FacetEval, measuring the sentiment consistency c between the generated meta-review and the corresponding human-written meta-review in all review facets. Different from the generic evaluation metrics for abstractive summarization or text generation which mostly adopt surface-form matching, we focus more on review facets and their corresponding sentiment levels.
Specifically, we use the distribution of sentiments in all review facets to represent the metareview and use the cosine similarity of the two vectors as the final score s.
s = cos (mh,mg) (1)
m = ∥∥ f [P+f , Pf , N + f , Nf , Of ] (2)
where ∥∥ denotes concatenation of representations for different facets, mh and mg are representations of the human-written and model-generated meta-reviews respectively. The representation m of the meta-review is the concatenation of vector representations of all review facets. Each facet of the document is represented by the frequency of different sentiment levels on the facet. The facet f is represented by a five-dimension vector [P+f , Pf , N + f , Nf , Of ] where P + f denotes the frequency of Strongly positive for the facet f , Pf the frequency of Positive, N+f the frequency of Strongly negative, Nf the frequency of Negative, and Of whether this facet is involved in the document. All the sentiments are obtained with GPT-4 following in-context learning in Section 3.2.
Following the similarity of meta-reviews, we could also calculate sentiment consistency among official reviews. Specifically, for every two official reviews i and j, the consistency in the facet f is the cosine similarity between two vector represen-
Review Facet w/ Conflicts w/o Conflicts
Advancement 0.463 (0.135) 0.551 (0.137) Soundness 0.526 (0.158) 0.501 (0.110) Novelty 0.300 (0.159) 0.357 (0.168) Overall 0.433 (0.147) 0.597 (0.172) Clarity 0.317 (0.133) 0.337 (0.145) Compliance 0.827 (0.071) 0.771 (0.118)

heading: 4.2 Measuring Sentiment Fusion for Individual Facets
text: same Review Facet in the source documents following Section 3.3, and we get predicted Sentiment Levels for all judgements, L′. Lastly, FusionEval computes an accuracy score by evaluating L′ against L. FusionEval only considers the precision instead of the recall for meta-review sentiments as it is reference-free and we have no information about the count of judgements that should be synthesized.

heading: 5 Enhancing LLMs with Explicit Information Consolidation
text: In this section, we propose two prompting methods to integrate the sentiment consolidation framework to generate meta-reviews. We compare the two methods with other prompting strategies including naive prompting and prompting with LLMgenerated logic. We also run experiments on opensource models besides OpenAI ones to investigate the influence of different prompting methods on different models. The experiments are based on automatic and human annotation on 500 samples from PeerSum.13

heading: 5.1 Prompting LLMs with Sentiment Consolidation Logic
text: Following the process in Figure 1 we propose decomposing the meta-review generation process in the following steps: (1) Extracting content and sentiment expressions of judgements from source documents; (2) Predicting Review Facets, Sentiment Levels, and Convincingness Levels; (3) Clustering extracted judgements for different review facets; (4) generate a “mini summary” for judgements on the same review facet; and (5) Generating the final meta-review based on the mini summaries for all review facets.
We explore two methods to integrate this process for prompting an LLM. (1) Prompt-Ours: we describe the five steps in a single prompt and ask GPT4 to generate the final meta-reviews (full prompt in Appendix G.1); (2) Pipeline-Ours: we create one prompt for each of the five steps, where the input for the intermediate step is the output from the previous step (full prompts in Appendix G.2).
We experiment with four open-source and closesource LLMs: GPT-4, GPT-3.5, LLaMA2-70B and LLaMA2-7B.14
13To avoid data contamination, we only use samples which were produced in and after 2022.
14Precise model names for them are: gpt-4-0613, gpt-3.5turbo-1106, LLaMA2-70B-Chat, LLaMA2-7B-Chat. Note
Competition Groups Preferred IAA
Prompt-Naive LLaMA2-70B 46.67% 0.64 Prompt-Ours LLaMA2-70B 53.33%
Prompt-Ours GPT-4 73.33% 0.74 Human-Written 26.67%

heading: 5.2 Baselines

heading: 5.3 Reference-Based and Reference-Free Automatic Evaluation
text: For automatic evaluation, we adopt ROUGE (Lin and Hovy, 2003)15, FacetEval (Section 4.1) and FusionEval (Section 4.2).16
We present the results in Table 7. Most LLMs perform better with either of our prompting methods (Prompt-Ours and Pipeline-Ours) than the baselines (Prompt-Naive and Prompt-LLM). Comparing between Prompt-Ours and Pipeline-Ours, the former seems to do best for ROGUE while the latter for FusionEval/FacetEval. As FusionEval/FacetEval assesses the accuracy of the overall sentiment specifically, Pipeline-Ours is marginally better when it comes to getting the overall sentiment correct. Comparing different LLMs, GPT-4 seems to work best, although that is mostly true for FusionEval/FacetEval.
15We use the implementation of the algorithm in https: //pypi.org/project/rouge-score/
16We do not consider other metrics such as BERTScore (Zhang et al., 2020), UniEval (Zhong et al., 2022), or G-Eval (Liu et al., 2023) as they have only been validated for summarization in news or a more general domain.

heading: 5.4 Reference-Free Human Evaluation
text: To further validate the effectiveness of our prompting methods, we conduct human evaluations to assess the quality of meta-reviews generated by different prompting methods or written by human meta-reviewers. We recruited three volunteer annotators who are senior PhD students familiar with artificial intelligence research and the peer review process. They are asked to select their preferred meta-reviews based on their own understanding of high-quality meta-reviews without knowing the source.17
Prompt-Naive vs Prompt-Ours We randomly select 30 samples and the annotators are asked to
17We use majority voting to get the final human preference.
compare the generated meta-reviews by PromptNaive and Prompt-Ours (using LLaMA2-70B) and select which one is better. Table 8 shows that the meta-reviews generated by Prompt-Ours are selected more by the annotators.
Prompt-Ours vs Human-Written We repeat the same experiments, but this time comparing metareviews generated by Prompt-Ours (GPT-4) vs. written by humans. Looking at Table 8, interestingly Prompt-Ours are much more preferred by the annotators. We suspect this may be because the generated meta-reviews tend to be more consistent in terms of the amount of detail it writes for each review facet, where else there is more variance for the human-written meta-reviews.

heading: 5.5 Case Study on Generated Meta-Reviews
text: To dive deeper into what difference the integration of sentiment consolidation framework makes, we also conduct a case study on generated metareviews with different prompting methods. We find that generated meta-reviews all seem plausible and machine-generated meta-reviews are much longer than human-written ones. In machine-generated meta-reviews, there are more details which are sometimes unnecessary or redundant. As shown in the example in Table 9, details such as "PGD" or "CIFA-10" are not essential to form the metareview.
Our proposed Prompt-Ours tend to have a more balanced judgements. For example, in Table 9, Prompt-Naive does not talk about the positive aspects for Clarity and only highlights some issues, but Prompt-Ours comments on both the strengths and weaknesses for Clarity. This is consistent with the finding in Table 7 that Prompt-Naive gets worse sentiments than Prompt-Ours.

heading: 6 Conclusions and Future Work
text: In this paper, we explore sentiment-focused multidocument information consolidation within the task of scientific sentiment summarization. We introduce a three-layer framework of sentiment consolidation to focus on generating meta-reviews and it considers the sentiments for each review facet in the reviews and discussions. We also propose automatic evaluation metrics that assess the overall sentiments in the generated meta-reviews. Experiments on meta-review generation show that prompting LLMs by following the processes in the three-layer framework results in better metareviews, providing an empirical validation of our framework for describing the meta-review writing process. As the sentiment consolidation also exist in other domains where human reviews or comments exist such as politics and advertisement, we will explore adapting our proposed sentiment consolidation framework into other domains in the future.
Limitations
Although integration of the sentiment consolidation framework could improve the generation results, there are still some limitations of this work.
• As in other areas peer review data is not publicly available, we use the data only from some
artificial intelligence conferences, and this may make the models biased. We hope that more data from diverse areas could be included.
• Experiments are only in English texts rather than other languages.
• We only inject the information consolidation logic into prompting based models instead of fine-tuning based models. We will investigate leveraging the information consolidation framework to improve fine-tuned models in the future.
• Although GPT-4 can predict meta-review sentiments based on source judgements to some extent, we have to understand more about how these models achieve this and what makes them fail in error cases.
• Meta-review generation is not only about sentiment prediction, future work has to consider more information such as argumentation in source reviews and justification in metareviews.
Ethics Statement
While our experiments demonstrate that the models exhibit potential in generating satisfactory metareviews to a certain degree, we strongly advise against solely relying on the generated results without manual verification and review, as instances of hallucinations exist in the generations. It is important to emphasize that we do not advocate for replacing human meta-reviewers with LLMs. However, it is noteworthy that these models have the capacity to enhance the meta-reviewing process, rendering it more efficient and effective.

heading: Acknowledgements
text: We would thank all the anonymous reviewers for their great comments to improve the work. It is impossible to have this work without the volunteer contributions of the annotators on human evaluations.

heading: A Review Criteria in Different Reviewer Guidelines

heading: B Annotation Instructions for Human Annotation
text: The screenshots of the two-page annotation instruction for human annotation are shown in Figure 4 and Figure 5 in the last two pages of the Appendix.
C Inter-Annotator Agreement Among Human Annotators and GPT-4

heading: D Prompt to Get Content and Sentiment Expressions with GPT-4
text: 1 P l e a s e r e a d t h e document : 2 3 {{ source_documen t }} 4 5 Thi s t a s k r e q u i r e s you t o a n a l y z e t h e above document which i s used t o e x p r e s s o p i n i o n s on t h e q u a l i t y o f a s c i e n t i f i c m a n u s c r i p t . You
a r e good a t u n d e r s t a n d i n g t h e s e n t i m e n t i n f o r m a t i o n wi th j u d g e m e n t s i n t h e document .
6 P l e a s e f i r s t i d e n t i f y t h e s e n t e n c e wi th j u d g e m e n t s on ly on t h e q u a l i t y o f s c i e n t i f i c m a n u s c r i p t s based on t h e r e v i e w f a c e t s f o r s c i e n t i f i c peer − r e v i ew : n o v e l t y , soundness , c l a r i t y , advancement , c o m p l i a n c e and o v e r a l l q u a l i t y w i t h i n t h e g i v e n document . 7 Once you have found a s e n t e n c e t h a t p r o v i d e s judgement i n one or more o f t h e s e a r e a s , you t h e n need t o e x t r a c t t h e s p e c i f i c e x p r e s s i o n o f s e n t i m e n t and t h e c o n t e n t i t r e f e r s t o . 8
9 The p r o c e s s can be broken i n t o two s t e p s : 10 1) I d e n t i f y a judgement s e n t e n c e t h a t f o c u s e s on t h e q u a l i t y o f t h e m a n u s c r i p t based on t h e g i v e n c r i t e r i a . 11 12 2) From t h e i d e n t i f i e d judgement s e n t e n c e , e x t r a c t two p i e c e s o f i n f o r m a t i o n : t h e s e n t i m e n t e x p r e s s i o n and t h e c o n t e n t e x p r e s s i o n . The s e n t i m e n t e x p r e s s i o n i s t h e s p e c i f i c te rm or p h r a s e t h a t conveys t h e s e n t i m e n t o r o p i n i o n . The c o n t e n t e x p r e s s i o n p e r t a i n s t o t h e c o n t e n t t h a t t h i s s e n t i m e n t i s r e f e r r i n g t o .
13 14 P l e a s e p r o v i d e t h e d a t a i n t h e f o l l o w i n g f o r m a t : 15 {" j u d g e m e n t _ s e n t e n c e " : " s e n t e n c e " , " c o n t e n t _ e x p r e s s i o n " : " c o n t e n t " , " s e n t i m e n t _ e x p r e s s i o n " : " s e n t i m e n t "} 16 17 Here a r e a few examples f o r your r e f e r e n c e : 18 {" j u d g e m e n t _ s e n t e n c e " : " The w r i t i n g o f t h e p a p e r i s n o t wel l − w r i t t e n
. " , " c o n t e n t _ e x p r e s s i o n " : " The w r i t i n g o f t h e p a p e r " , " s e n t i m e n t _ e x p r e s s i o n " : " n o t wel l − w r i t t e n "}
19 {" j u d g e m e n t _ s e n t e n c e " : " E x p e r i m e n t a l r e s u l t s a r e n o t s u f f i c i e n t l y s u b s t a n t i a t e d . " , " c o n t e n t _ e x p r e s s i o n " : " E x p e r i m e n t a l r e s u l t s " , " s e n t i m e n t _ e x p r e s s i o n " : " n o t s u f f i c i e n t l y s u b s t a n t i a t e d "} 20 {" j u d g e m e n t _ s e n t e n c e " : " Th i s p a p e r p r e s e n t s two n o v e l a p p r o a c h e s t o p r o v i d e e x p l a n a t i o n s f o r t h e s i m i l a r i t y be tween two samples based on 1) t h e i m p o r t a n c e measure o f i n d i v i d u a l f e a t u r e s and 2) some of
t h e o t h e r p a i r s o f examples used as a n a l o g i e s . " , " c o n t e n t _ e x p r e s s i o n " : " a p p r o a c h e s " , " s e n t i m e n t _ e x p r e s s i o n " : " n o v e l "}
21 22 The p r e d i c t e d judgmen t s ( f o l l o w i n g t h e same j s o n l i n e f o r m a t o f t h e above example ) :

heading: E Prompt to Get Judgement Component Predictions with GPT-4
text: 1 P l e a s e f i r s t r e a d t h e document below :
2 3 {{ source_documen t }} 4
5 6 P l e a s e p r e d i c t t h e f a c e t t h a t t h e g i v e n j u d g e m e n t s a r e t a l k i n g a b o u t . You can r e f e r t o t h e c o n t e x t i n t h e above s o u r c e document .
7 8 P o s s i b l e f a c e t s : 9
10 Nove l ty : How o r i g i n a l t h e i d e a ( e . g . , t a s k s , d a t a s e t s , o r methods ) i s , and how c l e a r where t h e prob lems and methods s i t w i t h r e s p e c t t o e x i s t i n g l i t e r a t u r e ( i . e . , m e a n i n g f u l compar i son ) . 11 12 Soundness : ( 1 ) E m p i r i c a l : how w e l l e x p e r i m e n t s a r e d e s i g n e d and e x e c u t e d t o s u p p o r t t h e c l a ims , whe the r methods used a r e a p p r o p r i a t e , and how c o r r e c t l y t h e d a t a and r e s u l t s a r e r e p o r t e d , a n a l y s e d , and i n t e r p r e t e d . ( 2 ) T h e o r e t i c a l : whe the r a rgumen t s o r c l a i m s i n t h e m a n u s c r i p t a r e w e l l s u p p o r t e d by t h e o r e t i c a l a n a l y s i s , i . e . , c o m p l e t e n e s s and t h e methodology ( e . g . , m a t h e m a t i c a l a p p r o a c h ) and t h e a n a l y s i s i s c o r r e c t .
13 14 C l a r i t y : The r e a d a b i l i t y o f t h e w r i t i n g ( e . g . , s t r u c t u r e and l a n g u a g e ) , r e p r o d u c i b i l i t y o f d e t a i l s , and how a c c u r a t e l y what t h e r e s e a r c h q u e s t i o n i s , what was done and what was t h e c o n c l u s i o n a r e p r e s e n t e d .
15 16 Advancement : I m p o r t a n c e o f t h e m a n u s c r i p t t o d i s c i p l i n e , s i g n i f i c a n c e o f t h e c o n t r i b u t i o n s o f t h e m a n u s c r i p t , and i t s p o t e n t i a l i mp a c t t o t h e f i e l d . 17 18 Compliance : Whether t h e m a n u s c r i p t f i t s t h e venue , and a l l e t h i c a l and p u b l i c a t i o n r e q u i r e m e n t s a r e met .
19 20 O v e r a l l : O v e r a l l q u a l i t y o f t h e m a n u s c r i p t , n o t f o r s p e c i f i c f a c e t s . 21
22 23 You a r e a l s o good a t u n d e r s t a n d i n g s e n t i m e n t i n f o r m a t i o n i n t h e j u d g e m e n t s .
24 25 P l e a s e p r e d i c t t h e o r i g i n a l e x p r e s s e r o f t h e s e n t i m e n t i n t h e judgement s e n t e n c e . You can r e f e r t o t h e c o n t e x t i n t h e s o u r c e document .
26 27 P o s s i b l e s e n t i m e n t e x p r e s s e r s : 28 29 − S e l f : t h e s e n t i m e n t i s from t h e s p e a k e r 30 − O t h e r s : t h e s e n t i m e n t i s quo t e d from o t h e r s 31
32 33 P l e a s e p r e d i c t how w e l l t h e s e n t i m e n t i n t h e judgement s e n t e n c e i s j u s t i f i e d i n t h e document i n your u n d e r s t a n d i n g . You can r e f e r t o
t h e c o n t e x t i n t h e s o u r c e document . 34 35 P o s s i b l e s e n t i m e n t c o n v i n c i n g n e s s : 36 37 − Not a p p l i c a b l e : t h e s e n t i m e n t i s e x p l i c i t l y e x c e r p t e d from o t h e r s . 38 − Not a t a l l : n o t c o n v i n c i n g a t a l l o r when t h e r e i s no j u s t i f i c a t i o n
. How w e l l t h e s e n t i m e n t i s j u s t i f i e d i n t h e document i n your u n d e r s t a n d i n g
39 − S l i g h t l y Conv inc ing : t h e r e i s some e v i d e n c e o r l o g i c a l r e a s o n i n g , b u t i t might n o t be comprehens ive . 40 − High ly Conv inc ing : l e a v i n g l i t t l e room f o r doub t . 41
42 43 P l e a s e p r e d i c t t h e p o l a r i t y and s t r e n g t h o f t h e s e n t i m e n t i n t h e judgement s e n t e n c e . You can r e f e r t o t h e c o n t e x t i n t h e s o u r c e document .
44 45 P o s s i b l e s e n t i m e n t s p o l a r i t i e s : 46 47 − S t r o n g n e g a t i v e : ve ry n e g a t i v e 48 − N e g a t i v e : minor n e g a t i v e 49 − P o s i t i v e : minor p o s i t i v e 50 − S t r o n g p o s i t i v e : ve ry p o s i t i v e 51
52 53 Judgements : 54 {{ j u d g e m e n t _ e x p r e s s i o n s }} 55 56 Your p r e d i c t i o n s f o r t h e above j u d g e m e n t s ( f o l l o w i n g t h e same j s o n l i n e s fo rmat , r e t u r n t h e same number o f l i n e s , and keep t h e same c o n t e n t and s e n t i m e n t e x p r e s s i o n s ) :

heading: F Prompts to Predict Meta-Review Sentiment Levels
text: F.1 Prediction with Judgements of Source documents The judgements are extracted from source documents, and they are in the same review facet to the target meta-review judgement.
1 You w i l l be g i v e n s o u r c e j u d g e m e n t s from r e v i e w e r s f o r a s c i e n t i f i c m a n u s c r i p t . Your t a s k i s t o i m p l i c i t l y w r i t e a meta − r e v i ew f o r t h e s e j u d g e m e n t s and p r e d i c t t h e s e n t i m e n t l e v e l based on t h e s e j u d g e m e n t s .
2 3 Source Judgements : 4 5 {{ s o u r c e _ j u d g e m e n t s }} 6 7 C a n d i d a t e S e n t i m e n t L e v e l s : 8
9 − S t r o n g n e g a t i v e 10 − N e g a t i v e 11 − P o s i t i v e
12 − S t r o n g p o s i t i v e 13 14 C o n t e n t E x p r e s s i o n : 15 16 {{ c o n t e n t _ e x p r e s s i o n }} 17 18 P r e d i c t t h e s e n t i m e n t l e v e l o f t h e g i v e n c o n t e n t e x p r e s s i o n based on t h e above j u d g e m e n t s . You must f o l l o w t h e f o l l o w i n g f o r m a t . 19 {" C o n t e n t E x p r e s s i o n " : t h e above c o n t e n t e x p r e s s i o n , " S e n t i m e n t Leve l " : your p r e d i c t e d s e n t i m e n t l e v e l }
F.2 Prediction with Full Texts of Source documents
The source texts are the concatenation of the source documents.
1 You w i l l be g i v e n m u l t i p l e r ev i ew documents f o r a s c i e n t i f i c m a n u s c r i p t . Your t a s k i s t o i m p l i c i t l y w r i t e a meta − r e v i ew and p r e d i c t t h e s e n t i m e n t l e v e l based on t h e s e documents .
2 3 Source Documents : 4 5 {{ s o u r c e _ t e x t s }} 6 7 C a n d i d a t e S e n t i m e n t L e v e l s : 8
9 − S t r o n g n e g a t i v e 10 − N e g a t i v e 11 − P o s i t i v e 12 − S t r o n g p o s i t i v e 13 14 C o n t e n t E x p r e s s i o n : 15 16 {{ c o n t e n t _ e x p r e s s i o n }} 17 18 P r e d i c t t h e s e n t i m e n t l e v e l o f t h e g i v e n c o n t e n t e x p r e s s i o n based on r e l a t e d i n f o r m a t i o n i n t h e above documents . You must f o l l o w t h e f o l l o w i n g f o r m a t . 19 {" C o n t e n t E x p r e s s i o n " : t h e above c o n t e n t e x p r e s s i o n , " S e n t i m e n t Leve l " : your p r e d i c t e d s e n t i m e n t l e v e l }

heading: G Prompts for Meta-Review Generation with Integration of Information Consolidation Logic
text: G.1 Prompt with Descriptive Consolidation Logic
1 Your t a s k i s t o w r i t e a meta − re v i e w based on t h e f o l l o w i n g r e v i e w s and d i s c u s s i o n s f o r a s c i e n t i f i c m a n u s c r i p t .
2 3 {{ i n p u t _ d o c u m e n t s }} 4 5 F o l l o w i n g t h e u n d e r l y i n g s t e p s below w i l l g e t you b e t t e r g e n e r a t e d meta − r e v i e w s .
6 7 1 . E x t r a c t i n g c o n t e n t and s e n t i m e n t e x p r e s s i o n s o f j u d g e m e n t s i n a l l above re v i e w and d i s c u s s i o n documents ;
8 9 2 . P r e d i c t i n g Review F a c e t s , S e n t i m e n t Leve l s , and C o n v i n c i n g n e s s L e v e l s ;
10 C a n d i d a t e r ev i e w f a c e t s : Novel ty , Soundness , C l a r i t y , Advancement , Compliance , and O v e r a l l q u a l i t y 11 C a n d i d a t e s e n t i m e n t l e v e l s : S t r o n g n e g a t i v e , Nega t ive , P o s i t i v e and S t r o n g p o s i t i v e 12 C a n d i d a t e c o n v i n c i n g n e s s l e v e l s : Not a t a l l , S l i g h t l y Convinc ing , High ly Conv inc ing
13 14 3 . R e o r g a n i z e e x t r a c t e d j u d g e m e n t s i n d i f f e r e n t c l u s t e r s f o r d i f f e r e n t r ev i e w f a c e t s ;
15 16 4 . G e n e r a t e a s m a l l summary f o r j u d g e m e n t s on t h e same r e v i e w f a c e t w i th compar i son and a g g r e g a t i o n ;
17 18 5 . Aggrega t e j u d g e m e n t s i n d i f f e r e n t r e v i e w f a c e t s and w r i t e a meta − r ev i e w based on t h e a g g r e g a t i o n .
19
20 21 You may f o l l o w t h e s e s t e p s i m p l i c i t l y and on ly need t o o u t p u t t h e f i n a l meta − r ev i e w . The f i n a l meta − r ev i ew :
G.2 Prompts Used in the Pipeline Generation Prompts for the first two steps, getting content and sentiment expressions and predicting other judgement components, are the same as prompts in Appendix D and Appendix E, respectively.
For the step of generating sub-summaries for individual facets, the prompt is as follows.
1 {{ i n p u t _ j u d g e m e n t s }} 2 3 Wri te a summary of t h e above j u d g e m e n t s on {{ c r i t e r i a _ f a c e t }} o f a m a n u s c r i p t .
For the step of generating final meta-reviews based on sub-summaries of individual facets, the prompt is as follows.
1 {{ i n p u t _ s u b _ s u m m a r i e s }} 2 3 Wri te a meta − r ev i e w t o summarize t h e above sub −summaries o f r e v i e w s and d i s c u s s i o n s i n d i f f e r e n t r e v i e w f a c e t s f o r a m a n u s c r i p t .
G.3 Prompts from Prompt-Naive For Prompt-Naive in our experiments, the prompt we use is as follows.
1 {{ i n p u t _ d o c u m e n t s }} 2 3 Wri te a meta − r ev i e w based on t h e above r e v i e w s and d i s c u s s i o n s f o r a m a n u s c r i p t .
G.4 Prompts from Prompt-LLM For Prompt-LLM, we have to generate first the steps with GPT-4 and then the meta-review based on the generated steps.
The prompt to generate the steps:
1 {{ i n p u t _ d o c u m e n t s }} 2 3 What a r e t h e s t e p s t o w r i t e a meta − r e v i e w s p e c i f i c a l l y f o r t h e above r e v i e w s and d i s c u s s i o n s o f a m a n u s c r i p t .
The prompt to generate the meta-review:
1 {{ i n p u t _ d o c u m e n t s }} 2 3 Fol low t h e f o l l o w i n g s t e p s and w r i t e a meta − r e v i e w based on t h e above r e v i e w s and d i s c u s s i o n s f o r a m a n u s c r i p t .
4 5 {{ g e n e r a t e d _ s t e p s }}
Annotation Instructions
Peer-review systems play a crucial role in maintaining a level of rigor in scientific publications. In the peerreviewing process, several appointed reviewers, a meta-reviewer, and the author for each submitted manuscript are usually involved. Specifically, reviewers write their comments on the manuscript; there could be responses by the author and discussion with the reviewers of possibly multiple turns; and the metareviewer finally gives the decision on the fate of the manuscript along with a meta-review which is a summary of the reviews and discussions in the whole peer-reviewing process. We find that the whole process of peer-reviewing is mostly about judgements from different participants on the quality and merit of the manuscript, and the meta-reviewers develop their final judgements based on those from the reviewers and authors.
Table 1 The typology of criteria facets for reviewing manuscripts in the peer-review process.
2/2
10177

heading: Future Work
text: As the sentiment consolidation also exist in other domains where human reviews or comments exist such as politics and advertisement, we will explore adapting our proposed sentiment consolidation framework into other domains in the future. Limitations
Although integration of the sentiment consolidation framework could improve the generation results, there are still some limitations of this work. • As in other areas peer review data is not publicly available, we use the data only from some
artificial intelligence conferences, and this may make the models biased. We hope that more data from diverse areas could be included. • Experiments are only in English texts rather than other languages. • We only inject the information consolidation logic into prompting based models instead of fine-tuning based models. We will investigate leveraging the information consolidation framework to improve fine-tuned models in the future. • Although GPT-4 can predict meta-review sentiments based on source judgements to some extent, we have to understand more about how these models achieve this and what makes them fail in error cases. • Meta-review generation is not only about sentiment prediction, future work has to consider more information such as argumentation in source reviews and justification in metareviews