Large-Scale and Multi-Perspective Opinion Summarization with Diverse Review Subsets
Authors: Han Jiang
Rui Wang
Zhihua Wei
Yu Li
Xinpeng Wang
Venue: EMNLP 2023 Findings (2023)
Keywords: opinion summarization
multi-document summarization
large-scale
multi-perspective
contrastive learning
TL;DR: We propose SubSumm, a supervised summarization framework for large-scale and multi-perspective opinion summarization.

Abstract: Opinion summarization is expected to digest larger review sets and provide summaries from different perspectives. However, most existing solutions are deficient in epitomizing extensive reviews and offering opinion summaries from various angles due to the lack of designs for information selection. To this end, we propose SubSumm, a supervised summarization framework for large-scale multi-perspective opinion summarization. SubSumm consists of a review sampling strategy set and a two-stage training scheme. The sampling strategies take sentiment orientation and contrastive information value into consideration, with which the review subsets from different perspectives and quality levels can be selected. Subsequently, the summarizer is encouraged to learn from the sub-optimal and optimal subsets successively in order to capitalize on the massive input. Experimental results on AmaSum and Rotten Tomatoes datasets demonstrate that SubSumm is adept at generating pros, cons, and verdict summaries from hundreds of input reviews. Furthermore, our in-depth analysis verifies that the advanced selection of review subsets and the two-stage training scheme are vital to boosting the summarization performance.

Large-Scale and Multi-Perspective Opinion Summarization with Diverse Review Subsets
Han Jiang, Rui Wang, Zhihua Wei; Yu Li, Xinpeng Wang Department of Computer Science and Technology, Tongji University, Shanghai, China (2230780, rwang, chihua_wei, 2331891, wangxinpeng @t edu on
Abstract
Due the burgeoning amount of online reviews and user needs, opinion summarization expected to (1) process larger sets of documents and (2) pro- vide summaries from different perspectives. One mainstream solution models the cross-document relations with sentence or document representa- tions, using mean function (Chu and Liu, 2019; Bražinskas al., 2020b; Li al., 2020), convex combination (Iso et al., 2021), and graph (Erkan and Radev, 2004; Ganesan al., 2010) as well as other hierarchical structures (Isonuma al.. 2021; Amplayo et al., 2021a). These approaches are proven to achieve remarkable results with a moderate amount of reviews, usually within 10 (Shapira and Levy, 2020); however, they perform unsatisfactorily when the number of reviews further increases, as they focus on the fusion rather than the selection of the information. Another solution concatenates the reviews for long-range language models (Beltagy et al., 2020; Zaheer et al., 2020; Mao et al., 2022; Pang et al., 2023) and Large Lan- guage Models (LLMs; OpenAL, 2023), which con- verts multi-document summarization into single- document summarization (Bražinskas et al., 2020a; Oved and Levy, 2021; Ke et al., 2022; Brazinskas et al., 2022; Bhaskar al., 2023). Despite the ben- efit brought by the LLMs, these methods struggle to handle the overlong combined reviews, miss- ing a step to select from them either. Bražinskas et al. (2021) first proposes to select smaller sub- sets of input reviews and provides verdict, pros, and cons summaries, yet differentiated treatments of different perspectives method. Limited by data, there are seldom works targeting large-scale and multi-perspective opinion summarization.
Opinion summarization is expected to digest larger review sets and provide summaries from different perspectives. However, most existing solutions are deficient in epitomizing extensive reviews and offering opinion summaries from various angles due to the lack of designs for information selection. To this end, we propose SUBSUMM, supervised summarization frame- work for large-scale multi-perspective opinion summarization. SUBSUMM consists re- view sampling strategy set and two-stage training scheme. The sampling strategies take sentiment orientation and contrastive informa- tion value into consideration, with which the review subsets from different perspectives and quality levels can be selected. Subsequently, the summarizer is encouraged to learn from the sub-optimal and optimal subsets successively in order capitalize the massive input. Ex- perimental results on AmaSun and Rotten Tomatose oes datasets demonstrate that SUB- SUMM is adept generating pros, cons, and verdiet summaries from hundreds of input re- views. Furthermore, our in nepththh analysis veri- fies that the advanced selection of review sub- sets and the two-stage training scheme are vital boosting the summarization performance.
1 Introduction
A plethora of online resources has been appealing for techniques of automatic information mining. Opinion summarization, as a task of generalizing views from a group of documents (e.g., reviews, posts, and discussions) related to an entity and pre- senting them in text form, has received consider- able attention. The summarization of user opinions is of great advantage to public opinion research, marketing analysis, and decision-making (Im et al., 2021). While circumventing the tedious document- by-document browsing, also offers more signifi- cant details compared to a single sentiment rating (Wang and Wan, 2021).
not reflected in their
To address the problems, we propose SUB- SUMM, a supervised summarization framework for large-scale and multi- culti-pperspective opinion summa- rization. SUBSUMM comprises review sampling strategy set and a two-stage training scheme. The
Conresponding author

---

review sampling strategies are formulated with sen- timent analysis and contrastive information valu- ation. With different strategies, the review sub- sets from different angles and quality levels can be selected. Then, the two-stage training method enables the summarization model to learn from the sub-optimal and optimal review subsets suc- cessively to fully utilize the input reviews within the model eapacity. During the training stage II, a contrastive loss term is incorporated to further boost the performance of the summarizer.
2021; Isonuma al., 2021) summarize through the aggregation of the latent representations of the reviews. COOP (Iso et al., 2021) considers the convex combination of input review representa- tions. These methods work well with fewer re- views, while they suffer performance drop when processing numerous reviews.
The synthetie-daatsett-aased methods (Amplayo and Lapata, 2020; Bražinskas al., 2020a; Oved and Levy, 2021; Wang and Wan, 2021; Amplayo et al., 2021b; Ke et al., 2022; Brazinskas et al., 2022) transform the unsupervised task into a super- vised task by constructing review- pairs from original data. PASS (Oved and Levy, 2021) applies systematic perturbations to the input views for more candidate summaries and trains classifier to rank the candidates. CONSISTSUM (Ke al., 2022) measures the distances between reviews from aspect, sentiment, and semantics to create highly relevant review-summary pairs. ADA- SUM (Brazinskas et al.. 2022) first fine-tunes the PLM with a synthetic dataset, then performs fine- tuning in few-shot manner. The idea of making full use of the original text is embodied thoroughly in these methods.
By coupling with SUBSUMM, the Pre-trained Language Model (PLM) outperforms previous -tate-of-the-art models and LLMs under zero- shot settings the AmaSum and Rotten Tomatoes datasets in our experiments, which demonstrates the superiority of the proposal. Fur- ther analysis also proves the effectiveness of the two modules in SURSUMM.
The contributions of this paper are as follows.
We propose large-scale opinion summariza- tion framework' to address the challenge of summarizing large review sets and providing opinions from different perspectives by select- ing valuable review subsets.
Benefiting from the growth of annotated data for opinion summarization, there are some emer- gent studies on supervised methods. Bražinskas et al. (2021) provide a large-scale opinion summa- rization dataset enabling supervised training. They formulate the task as jointly learning to select infor- mative reviews and summarize the opinions, and their solution SELSUM based on reinforcement learning (REINFORCE; Williams, 1992). Aiming at avoiding the challenges brought by reinforce- ment learning, we decouple the process of selection and summarization in this work.
We present (I) review sampling strategy set based sentiment analysis and contrastive in- formation valuation and (2) two-stage train- ing scheme promoting the digestion and ab- sorption of the massive input.
We substantiate the effectiveness of the proposed opinion summarization framework SUBSUMM with sufficient experiments and in- depth analysis on two opinion summarization datasets from different domains.
Contrastive Learning Contrastive learning in tomatic summarization (Cao and Wang, 2021; Xu et al., 2021; Sun and Li, 2021; Liu and Liu, 2021; Liu et al., 2022) also gives us much inspiration. CLIFF (Cao and Wang, 2021) creates negative samples with automatically generated erroneous summaries. SIMCLS (Liu and Liu, 2021) trains an extra model with contrastive learning to eval- uate and rank the candidate summaries. BRIO (Liu et al., 2022) introduces contrastive learning assign a dual role to the model, alleviating infer- ence performance degradation. In this work, we explore contrastive lcarning for multi-document summarization rather than single-document sum-
2 Related Work
Opinion Summarization As high-quality anno- tation for the large opinion corpora is expensive to obtain (Ge et ul., 2023), most works of opinion summarization are unsupervised, summarizing limited number of reviews. Among the abstrac- tive approaches, VAE-based and dynthetic-datasst- based models have the upper hand.
The VAE-based models (Chu and Liu, 2019; Bražinskas et al., 2020b; Li et al., 2020; Iso et al.,
'The code is available at https: thub. con/
Sa www.oeeeeeeee....o. Sateeeeeee/uuuSume SubSumm.

---

Review subset produced by Sentiment- Random
Review subset produced by Centiment-informatioo
Training stage
Training stage II
Sentiment Analysis
Information Valuation
Review subsets
L Lexant
In Exant + Yhar
Random
Sentiment-
Random
PLM
Sentiment-
Information
Candidate summaries
Review sampling strategy
Review set
Inference
(a) Review sampling strategy set
(b) Two-stage progresive training scheme
Figure I: The framework of SUBSUMM. (a) The review sampling strategy set based on sentiment analysis and information valuation. The three strategies, Random Sampling, Centiment-Randonn Sampling, and Sentiment- Information Ranking are abbreviated as Random, Sentiment Randomm, and Sentiment Sentiment--nfomationn (b) The two-stage wwo-stage training scheme, under which the PLM (BART (Lewis et al., 2020) in this work) learns summarize sub-optimal and optimal subsets successively.
marization, and a PLM is fine-tuned contrastively for the information valuation.
contrary sentiment orientations. On the other hand, the sentiment tendency of the summary can be con- trolled by adjusting the proportion of input reviews with different sentiments; in this way, multiple an- gles are formed.
3 Methodology
We introduce SUBSUMM, supervised framework for large-scale and pusti-perspective opinion sum- marization, as illustrated in Fig. SUBSUMM is composed of review sampling strategy set regard- ing sentiment orientation and information value, as elucidated in Sec. 3.1; and a two-stage training scheme, where contrastive learning with candidate summaries is extra performed, see Sec. 3.2.
We formulate sentiment analysis text clas- sification task. The sentiment tags of reviews in RI.N are computed as:
p SLM
(2)
Sen; arg max P. sen (J)
(3)
where SLM(()) is a PL.M with linear classification head, and p. refers to the probability distribution over the sentiment classes of the review R, The class with the highest probability is taken as the sentiment tag Sen, We use the rating of each re- view in the dataset as the sentiment label and apply negative log likelihood loss w.r.t. the sentiment distribution pee" After fine-tuning, the sentiment tags of all the reviews can be obtained.
Given the entity set and the corresponding sam- ple set, for every sample { the goal of opinion summarization is learn function f that takes the review set as input and outputs a summary as close as possible to the reference:
S + f(I:N)
(1)
where RINN is the original review set, and S is the reference opinion summary. This paper mainly discusses the situation where R1:N is too large to be processed with most language models. For re- view set RI,N, let REKK be the review subset where K N.
Contrastive Information Valuation Information valuation has finer granularity than sentiment analy- sis. Intuitively, once subset of reviews selected for summarization, the closer the generation is to the reference, the more valuable the information in the subset may be.
3.1 Review Sampling Strategy Set
Given the reference summary, the information value of an input review is tied its similar- ity with the reference; ROUGE (Lin, 2004) is an appropriate metric to estimate such similar- ity. Thus we fit the ROUGE score of a review
Sentiment Analysis We leverage sentiment anal- ysis to filter the reviews roughly. It supposed that the reviews with similar sentiment orientations are less likely to conflict in content than those with

---

R₁ (ri ..... (.Rli by modeling the correla- tion between the review and the whole review set:
Sentiment-Rnddom Sampling: Firstly, di- vide all reviews into positive and negative types according to their sentiment tags Sen,, Secondly, the number of reviews of each type R1:K determined by the type of the refer- ence summary:
41
(1) h ([RA)) Enc(r: (1)
(
(4)
1
(K)
h4
h
(5)
R
Corr: N
1
E
hęhj
(6)
(K,0),
pros
(K,,
(0, K),
cons
(KN- KN
verdict
where Enc(-) is a transformer (Vaswani et al., 2017) encoder, and h( denotes the last hidden state of token v/(). The review representation h, is computed by averaging the last hidden states of the tokens Rg. Corr; is the correlation score be- tween R2 and the review set RI.N. We refer to the leave-one-out setting in unsupervised opinion sum- marization, computing Corr; by the dot product of h4 and the mean representation of the others.
N N
(8)
where (K+, K-),, (N+,
NN) stands for the numbers of positive and negative reviews in R1.K- RIN;; K K, N N. Finally, randomly sample K+ K reviews from the positive and negative types respec- tively for RI.K-
Sentiment-Information Ranking: Firstly, compute (-) likewise. Secondly, sort the reviews in descending order by the estimated information value Corr; in two types separately. Finally, take the top-K positive reviews and the top- K negative reviews for Ry.K.
Mindful of the volume of the original review set, it is unfcasible to fit the distribution of the ROUGE score directly employ list-wise loss. Therefore. we resort contrastive margin loss:
C r(})r()
max(0, Corry Corr; + Xij)
The quality of the corresponding review subsets should improve in sequence.
(7) where rii) accounts for the ranking of R, when sorted by ROUGE(R,, S) in descending order, and
3.2 Two-Stage Training for Large-Scale Opinion Summarization
Aij X(r(j) r(i)) is margin varying with the rankings, defined following Zhong et al. (2020); Liu and Liu (2021); Liu et al. (2022). The pair- wise loss allows the model learn the ROUGE rankings of large review set. After fine-tuning, we can get the estimated information values of all the reviews.
SURSUMM embodies a two-stage training scheme encouraging the summarizer to learn from the sub- optimal and optimal review subsets successively.
In stage we choose the sub-optimal strategy, Sentiment-Raud Randoom Sampling to re-sample the re- view subset RiKk at each training epoch and train the model with standard maximum likelihood esti- mation (MLE):
Multi-level Review Sampling Strategies Drawing support from the sentiment analysis and the contrastive information valuation, diverse review sampling strategies can be formed to select RI-K out of R1.N We find that not ideal to accomplish the task with single optimal subset, which will be explained in Sec. 4.3. To tackle the problem, we introduce stochastic factors to develop sampling strategies of multiple quality levels.
0" arg max Σ log Po (S
(9)
where 0 denotes the parameters of the abstract model, and po represents the probability derived by the parameters. The cross entropy loss is defined over the reference sequence of length as:
The sampling strategy set consists of the follow- ing three strategies:
Li Luent
Random Sampling: Randomly sample K re- views from the original review set RE.N as the subset R
ΣΣ p" (s) [s| Sei) (o) Ses)
1=1 8EV
(10)

---

where can be any token in the vocabulary V. and p" refers an one-hot distribution. Se stands for a pre-defined start token and the first i tokens of the reference summary. However, Standard MLE is prone to exposure bias since it heavily relies on the ground-truth sequence (Zhang et al., 2019). Mean- while, whichever strategy is adopted, the reviews sampled are only part of the original review set, where the information
Finally, to maintain the generation ability of the pre-trained model, we follow Edunov et al. (2018) to use the multi-task loss:
L = Crent + -
(13)
where y is the weight of the contrastive loss term. By involving the candidate summaries in training, stage II raises the utilization rate of the original review set and alleviates the problem of exposure bias; it acts as a complement to stage consider- ing the addition of the review subsets with higher quality and the contrastive loss term.
be further exploited. In stage II, we take cue from the practice of assigning probability mass to candidate summaries during training (Liu et al., 2022). Theoretically, assigning probability mass to a summary means an opportunity for the summary to pass on knowledge to the model through backpropagation. Hence the range of probability mass allocation is essentially the range of model learning, and better candidate summaries ought to compete for more probability mass. We plan to reuse the original review set the candidate summaries.
During inference, given review set RI.N. SUB- SUMM predicts the sentiment tag and information value each review with the fine-tuned PLMs in Sec. 3.1, then selects the optimal review sub- set Ri.K according to the iontiimenn-Informaati Ranking strategy and summarizes the subset using the summarization model from stage II.
To start with, we slightly modify the optimal strategy (i.e., Gentiment-Information Ranking), as some perturbations are required to obtain various candidate summaries for contrastive learning:
4
Experiments
4.1 Experimental Settings
Datasets We choose two opinion summarization datasets with large review sets as our testbed. The statistics are shown in Appendix A.
Sentimenn--uformation Ranking (modified): After computing (K+, in Eq. 8, take the estimated information value Corr; of each review as weight to sample K++ K" reviews from the positive and negative types severally.
AmaSum? (Bražinskas et al., 2021) is a prod- uet review dataset where each sample contains large number of reviews and reference summaries written by professional reviewers. Unlike other datasets, AmaSum provides reference summaries from three perspectives, namely verdict, which is equivalent to general opinion summary; pros and cons, which summarize the most important posi- tive and negative details. As shown in Table 6, with 4.2k tokens average, the combined reviews in AMASUM are too long to summarize with most summarizers. We refer to the preprocessing in SEL- SUM but split the dataset into three partitions with different targets.
Next, the modified optimal strategy is repeatedly conducted to get M review subsets, with which M candidate summaries $1, $2, .... SM are generated by the model from stage The review subset pro- duced by the original optimal strategy, denoted by RI:K, will be the training input. We again calculate the ROUGE scores of the reviews in RiE with the reference summary to derive the rankings and apply a contrastive loss term similar to Eq. 7:
M
Lar Σ
Σ
max(0, Lhj Lhs + Xig)
Rotten Tomatoes (RT; Wang and Ling, 2016) is large-scale movie review dataset. For each movie, one-sentence critic consensus is con- structed by editor to summarize the opinions in professional critics, which is treated the refer- ence summary. We follow Amplayo al. (2021b) to preprocess the dataset; the data in RT and the verdict partition of AmaSum are equally treated in our experiments.
i= r()
(11)
where Lhş is the length-normalized likelihood of the candidate summary S,, which is defined follow- ing Liu et al. (2022):
LN(S)= = Σn log po Ri.K Sei)
1o
(12)
Here a is a length penalty hyperparameter. This term enforces the model to assign more probability mass to better candidate summaries.
hhttps:/ggihhul com/ Ptttps://wee /wwbb eecs. umi miiichedd www.ed//wwale uxy/ data.html

---

Pros
Cons
Verdict
Method
R-1
R-2
R-L
R-1
R-2
R-L
R-1
R-2
R-L
Unsupervised
MEANSUM
10.44
0.63
9.55
5.95
0.45
5.29
13.78
0.93
11.70
LEXRANK
14.12
1.50
12.81
8.28
0.82
7.24
15.12
1.84
12.60
COPYCATI
15.12
1.48
13.85
6.81
0.82
5.89
17.05
1.78
14.50
EXTSUM!
19.06
2.47
17.49
11.63
1.19
10.44
18.74
3.01
15.74
Supervised
SELSUMI
21.29
4.00
19.39
14.96
2.60
13.07
24.33
5.29
18.84
LONGFORMER
22.40
4.71
15.36
14.68
2.53
11.62
22.56
4.83
17.08
SELSUM
23.17
4.77
21.13
15.12
2.83
13.07
22.87
4.85
18.05
BRIO
25.48
4.58
23.50
16.65
2.94
14.60
24.93
4.78
19.44
SUBSUMM
26.25
4.96
24.18
16.72
3.00
14.80
25.36
5.04
19.58
Zero-Shot
QG CHATGPT 18.53
18.40
2.21
16.02
13.27
1.39
11.61
14.75
1.23
12.06
2.37
14.68
13.92
1.78
11.93
22.88
3.50
19.79
Table I: Automatic evaluation results on AnaSum dataset. Best models are shown in bold and 2nd best models are underlined; i means that the results are copied from Bražinskas al. (2021). retrained SE.SUM* on pros, cons, and verdict separately for fair comparison.
unsupervised extractive models LEXRANK (Erkan and Radev, 2004) and EXTSUM (Bražinskas et al., 2021); (2) unsupervised abstractive models MEAN- SUM (Chu and Liu, 2019) and COPYCAT (Bražin- skas et al., 2020b); (3) supervised abstractive mod- els SELSUM, LONGFORMER (Beltagy al., 2020), and BRIO; and (4) zero-shot solutions related LLMs, including GPT-3.5-turbo (CHATGPT) as well as QG (Bhaskar et al., 2023) based on QF- Summ (Ahuja al., 2022) and GPT-3 (Brown et al., 2020).
Method R-1
R-2
R-L
Unsupervised
W2VCENT
13.93
2.10
10.81
LEXRANK
14.88
1.94
10.50
OPINOSIS
14.98
3.07
12.19
MEANSUM
15.79
1.94
12.26
SNCENTi
15.90
2.01
11.74
BERTCENT!
17.65
2.78
12.78
DENOISESUM!
21.26
4.61
16.27
Weakly Supervised
PLANSUM
21.77
6.18
16.98
On RT dataset, the extra baselines are (1) unsu- pervised extractive models W2VCENT (Rossiello et al., 2017), SNCENT (Amplayo and Lapata, 2020), and BERTCENT (Amplayo al., 2021b); (2) unsupervised abstractive models OPINOSIS (Ganesan al., 2010) and DENOISESUM (Am- playo and Lapata, 2020); (3) weakly supervised model PLANSUM (Amplayo et al., 2021b). We classify PLANSUM as weakly-supervised summa- rizer since it uses additional information other than review text.
Supervised
BRIO
23.72
5.16
18.05
LONGFORMER
24.96
6.34
18.40
SUBSUMM
24.96
6.66
19.08
Zero-Shot
QG CHATGPT 22.73
18.14
2.34
14.28
4.21
17.52
Table 2: Automatic evaluation results on RT dataset. Best models shown in bold and 2nd best models are underlined; means that the results are copied from Amplayo et (2021b).
A detailed introduction to the baselines is in Appendix B.
Implementation Details We used RoBERTa-base (Liu et al., 2019) for the sentiment analysis, a BART-base (Lewis et al., 2020) encoder for the contrastive information valuation, and BART-base
Baselines Concerning the baselines, we select a series of competitive models for the two datasets. On AmaS Sum dataset, the baselines include (1)

---

Pros
Cons
Verdict
Summary
Infot
Coht
N-RT
Infot
Coht
N-R1
Infot
Coht
N-R1
GOLD
19.7
5.7
6.4
0
0
4
6.9
5.2
10.2
SELSUM
-20.2
2.9
11.8
-6.3
-0.6
-0.9
-7.2
6.5
0.6
BRIO
-7.5
-25.6
-23.5
-5.1
-7.3
-10.7
-8.4
-17.2
-14.7
SUBSUMM
17
5.3
11.4
7.9
7.6
8.6
5.5
3.9
Table 3: Human evaluation results SmaSum. Info, Coh, and N-R abbreviations of Informativeness, Coherence, and Ron-Reednddncy.
two observations:
AmaSum
(1) SUBSUMM excels in generating summary with obvious emotion tendency, i.e., pros and cons. We notice that the three targets in AmaSum equally treated by all the baselines, indicating the lack of exploration into the difference between the perspectives. SUBSUMM samples only posi- tive/negative reviews for pros/cons summary. As depicted in Fig. 2, the reviews sampled for pros and cons are distributed in different zones of the semantic space, with the reviews for verdict evenly scattered between them. It not only reduces incon- sistencies, but also adds valuable information the input, since positive reviews always point out more advantages of the product, and vice versa.
Method
Pros
Cons
Verdict
RT
SUBSUMM 0.66
0.67
0.89
0.70
CHATGPT
0.34
0.33
0.11
0.30
Table 4: Win rates of SUBSUMM and CHATGPT in the pair-wise comparisons.
IMS 251 AND SEAUIT crewn MONNEERNA
Futn coft tychated.
Tree ald witter abi...
(2) The supervised methods score generally higher than the LLM-related methods, and SUB- SUMM has edge over the congener supervised systems. Although LLMs possess strong versatility in text generation, fine-tuning standard PLMs on annotated data seems non-trivial for opinion sum- marization. Compared with the other supervised methods, SUBSUMM reuses the reference sum- maries through contrastive learning in both infor- mation valuation and the training stage II, thus com- prehensively utilizing the annotations. Especially, LONGFORMER'S sparse attention mechanism plays an implicit selection role, while the review sam- pling strategies of SUBSUMM consider the sen- timent tendency and information value, which is more sophisticated and task-specific.
From
Gota
Vertee
Figure 2: A t-SNE (van der Maaten and Hinton, 2008) plot of the embeddings of most reviews from a product in AmaSum, where the dots of different colors represent the reviews selected for different perspectives. A few outliers are omitted.
the backbone of our summarizer and its vari- ants. In all of the review sampling strategies, selected K- 10 reviews for every subset, which is explained Appendix D. All the following ex- periments were conducted on 2 Geforce RTX 3090 GPUs. For the hyperparameters and more details, please refer to Appendix C.
Human Evaluation As supplement to auto- matic evaluation, we conducted a user study using Best-Wors Scaling (BWS; Louviere et al., 2015) detailed Appendix E. The four evaluated sum- maries were GOLD (reference) summary and sum- maries generated by SELSUM, BRIO, and SUB- SUMM. The three criteria were Informativeness. Coherence, and Non- Non-Redundancy. Results in Table demonstrate the considerable
4.2 Results
Automatic Evaluation We used ROUGE-1/2/L the evaluation metrics and reported the F-1 scores. For AmaSum, we evaluated pros, cons, and verdict separately. As shown in Table and Ta- ble 2, SUBSUMM significantly outperforms other methods on both datasets. Specifically, there are

---

Pros
Cons
Verdict
Method
R-1
R-2
R-L
R-1
R-2
R-L
R-1
R-2
R-L
Sampling Strategy
RAND
21.83
4.28
19.91
14.06
2.55
12.35
22.67
4.66
17.81
SENTI-RAND
22.33
4.41
20.45
14.92
2.75
13.13
23.06
4.79
17.98
SENTI-INFO
22.82
4.60
20.80
14.42
2.48
12.59
23.51
5.03
18.16
SENTI-RAND-IE RAND-INFO
23.01
4.76
21.02
15.46
2.92
13.60
23.51
5.04
18.32
Training Scheme
SUBSUMM
26.25
4.96
24.18
16.72
3.00
14.80
25.36
5.04
19.58
w/o Stage
25.48
4.58
23.50
16.65
2.94
14.60
24.93
4.78
19.44
w/o Stage II
23.01
4.76
21.02
15.46
2.92
13.60
23.51
5.04
18.32
RAND Stage
26.03
4.93
23.96
16.66
2.97
14.78
24.71
4.93
19.43
RAND in Stage II
25.95
4.95
24.02
16.50
2.77
14.54
24.60
4.96
19.45
Table 5: Results of analysis experiments on Amašum. R-1/2/L are the ROUGE-1/2/. F-1 scores, and the highest scores in both blocks are shown bold.
practical value of our model. Regarding Informa- liveness, summaries from SUBSUMM display com- parable, even more information than GOLD sum- maries. As for Coherence, SUBSUMM leaves the users the best reading experience with correct gram- mar and straightforward expression. In terms of Kon-Redundancy, SUBSUMM does not present the most succinct summaries, but considering the first two criteria, the redundancy is still acceptable.
iments on AmaSum, using the same metrics as automatic evaluation. We also reported the results on RT in Appendix F.
Comparison between Sampling Strategies As aforementioned, the quality of the three strate- gies in the review sampling strategy set would el- evate sequentially. To confirm, compare sum- marizers from the training stage with different sampling strategies in the upper block of Table 5. RAND, SENTI-RAND, and SENTI-INFO apply Ran- dom Sampling, CentimeentRRandonn Sampling, and Sentiment Sentimennt-nformatiinn Ranking in training and in- ference respectively; SENTI- AND-INFO is trained with Sentiment Sentiment-Randon Sampling but infers on re- view subsets produced by Centiment-Informalt Ranking.
We further compared our model with the LLM via 50 head-to-head tests between SUBSUMM and CHATGPT. The test cases were randomly sampled from the two datasets (15 samples from each parti- tion in AmaSum's test set and 5 samples from RT's test set), and the annotators were asked to make pair-wise comparisons without the reference sum- maries. The results are shown in Table 4. It seems that the users prefer the summaries generated by SUBSUMM. An obvious issue of CHATGPT that it cannot control the output length within few calls when the input overlong. Consequently, most of the summaries generated are either exces- sively long or abruptly truncated with the maxi- mum length argument fixed. In addition, though CHATGPT is qualified to produce fluent text, it suffers from more severe hallucination than our model, which may compromise its ROUGE scores. In Table 10 are some supporting cases.
By comparing RAND with SENTI-RAND, it can be seen that with the aid of sentiment analysis. the review subsets sampled appear more useful for the summaries with emotion tendencies. There is no clear improvement from SENTI-RAND SENTI-INFO, so we add SENTI-RAND-INFO to as- and SENTI-
certain the reason. RAND only differ in the test input, while the former wins with a clear margin, suggesting Sentiment- Information Ranking produces better review sub- sets.
shares the same test in- put with IENTI-INFO but results in higher ROUGE scores, possibly because the stochastic factor pre- vents the potential over-fitting problem. It also drops hint that employing diverse review subsets might promote the model performance.
4.3 Analysis
With the purpose of deeper insights into our pro- posal, we carried out some in-depth analysis exper-

---

Insight into Two-Stage Training Scheme We in- vestigate the gains from the two-stage training scheme through an ablation study. The variants in the bottom block of Table share the same test input with SUBSUMM.
designs for aspect learning in SUBSUMM, which causes the model miss more 2-gram aspect terms than SELSUM (We noticed that SELSUM empha- sizes aspect learning). Secondly, ROUGE-L is com- puted based on the longest common subsequence, which has something to do with the flueney of the generation. We find that there are some errors, like repetitions and incomplete first words in the sum- maries from SUBSUMM. Compared the LLMs with extensive parameters, our proposal still has room for improvement in language modeling.
Our experiments evidence that both stage and stage Il are significant model performance, while the latter plays greater role. We suppose that the two stages are complementary
each other: the standard MLE training in stage functions as task- specific initialization, and the multi-task learning in stage II passes on more knowledge to the model, mitigating the exposure bias problem.
Ethics Statement
Moreover, we explore how the two-stage train- ing scheme contributes to the summary quality by replacing the sub-optimal and optimal strategies in the two training stages with Random Sampling. It leads to observable performance decreases, yet they are slighter than those when stage or II is directly removed. It can be inferred that besides the complementary training objectives and additional training steps, the sensible selection of the review subsets is also conducive to model training.
We used only publicly available datasets, artifacts, and figures. Nevertheless, realize that the pro- posed framework may produce fabricated and po- tentially harmful contents, for the PL.Ms used are pre- tre-trained on heterogeneous web corpora. There- fore, we recommend that the users cautiously apply the proposal and its by-products.
Acknowledgements
The work is partially supported by the National Na- ture Science Foundation of China (No. 61976160, 6 61906137, 61976158, 620761 84, 62076182) and Shanghai Science and Technology Plan Project (No. 21DZ1204800) and Technology Research Plan Project of Ministry of Public and Security (Grant No. 20201SYJD01).
5 Conclusion
In this paper, we put forward supervised sum- marization framework for large-scale and multi- perspective opinion summarization, SURSUMM. SUBSUMM supports two-stage training scheme based on set of review sampling strategies of multiple quality levels. Our model surpasses the state-of-the-art models and LL.M-related systems on AmaSum and RT, manifesting its superiority in dealing with plentiful reviews and displaying various points of view. The analysis experiments verify that both components of SUBSUMM help the summarizer achieve better results.
References
Ojas Ahuja, Jiacheng Xu, Akshay Gupta, Kevin Horecka, and Greg Durrett. 2022. ASPECTNEWS: Aspect-oriented summarization news documents. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6494 6506. Dublin, Ireland. Association for Computational Linguistics.
In the future, we are planning (1) explore more review sampling strategies to fully learn the aspect information and (2) combine the proposed framework with LLMs and generalize it to other large-scale multi-input tasks.
Reinald Kim Amplayo, Stefanos Angelidis, and Mirella Lapata. 202 Aspect-controllabbe opinion summa- rization. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 6578-6568 6693, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Limitations
There are also some limitations in SUBSUMM. In Table 1, the verdict partition, the ROUGE-2 F1- score of our model does not outweigh that of SEL- SUM; the ROUGE-L Fl-score of our model is slightly lower than that of CHATGPT.
Reinald Kim Amplayo, Stefanos Angelidis, and Mirella Lapata. 2021b. Unsupervised opinion summarization with content planning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35. pages 12489-12497
Firstly, since ROUGE-2 reflects 2-gram recall, we suspect that this is due to the absence of explicit
Reinald Kim Amplayo and Mirella Lapata. 2020. Un- supervised opinion summarization with noising and

---

denoising. In Proceedings the 58th Anmual Meet- ing of the Association for Computational Linguistics, pages 1934-1945, Online. Association for Computa- tional Linguisties.
Eric Chu and Peter Liu. 2019. MeanSum: A neural model for unsupervised documed abstractive summarization. In Proceedings of the 36th Interna- rional Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 1223-1232, PMLR.
Iz Beltagy, Matthew E. Peters, and Arman Cohan, 2020. Longformer: The long- aongdooumeett transformer.
Jacob Devlin, Ming-Wo Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Adithya Bhaskar, Alexander R. Fabbri, and Greg Dur- rett. 2023. Prompted opinion summarization with gpt-3.5.
Arthur Bražinskas, Mirella Lapata, and Ivan Titov. 2020a. Few-shot learning for opinion summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 41 19-4135, Online. Association for Computa- tional Linguistics.
Sergey Edunov, Myle Out, Michael Auli, David Grang- ier, and Mare' Aurelio Ranzato. 2018. Classical struc- tured prediction losses for sequence to sequence learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long Papers), pages 355-364, New Orleans, Louisiana. Association for Computa- tional Linguistics.
Arthur Bražinskas, Mirella Lapata, and Ivan Titov. 2020b. Unsupervised opinion summarization as copycat-review generation. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 5151-5169, Online. Asso- ciation for Computational Linguistics.
Arthur Bražinskas, Mirella I Lapata, and Ivan Titov. 2021. Learning opinion summarizers by selecting informa- tive reviews. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 9424-9442, Online and Punta Cana. Do- minican Republic. Association for Computational Linguistics.
Günes Erkan and Dragomir R. Radev. 2004. Lexrank: Graph-basee based lexical centrality salience in text sum- marization. J. Artif. Int. Res,, (755774574
Kavita Ganesan, ChengXiang Zhai, and Jiawei Han. 2010. Opinosis: A graph based approach to abstrac- tive summarization of highly redundant opinions. In Proceedings of the 23rd International Conference Computational Linguistics (Coling 2010), pages 340-346 308, Beijing, China. Coling 2010 Organizing Committee.
Arthur Brazinskas, Ramesh Nallapati, Mohit Bansal, and Markus Dreyer. 2022. Efficient few-shot fine- tuning for opinion summarization. Findings of the Association for Compulational Linguistics: NAACL 2022, pages 1509-1523, Seattle, United States. Asso- ciation for Computational Linguistics.
Suyu Ge. Jiaxin Huang, Yu Meng, and Jiawei Han. 2023. Finesum: Target-oriented, fine-grained opinion sum- marization. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, WSDM '23, page 1093-1101, New York, NY, USA. Association for Computing Machinery.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu. Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei, 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Jinbae Im. Moonki Kim, Hoyeop Lee, Hyunsouk Cho, and Sehee Chung. 2021. Self-supervised multimodal opinion summarization. Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 388-403, Online. Association for Computational Linguistics.
Proceedings of the 59th
Hayate Iso, Xiaolan Wang. Yoshihiko Suhara, Stefanos Angelidis, and Wang- Sann-Chiew'' Tan. 2021. Convex Ag- gregation for Opinion Summarization. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3885-3903, Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Shuyang Cao and Lu Wang. 2021. CLIFF: Contrastive learning for improving faithfulness and factuality in abstractive summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural
Language Processing, pages 6633-6649, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Masaru Isonuma, Junichiro Mori, Danushka Bollegala, and Ichiro Sakata, 2021. Unsupervised abstractive

---

opinion summarization by generating sentences with tre-strructrre topic guidance. Transactions of the A Association for Computational Linguistics, 9:945- 961.
Yixin Liu and Pengfei Liu. 2021. SimCLS: A sim- ple framework for contrastive learning of abstractive summarization. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference Natural Language Processing (Volume 2: Short Papers), pages 1065-1072, Online. Association for Computational Linguistics.
Wenjun Ke. Jinhua Gao, Huawei Shen, and Xueqi Cheng. 2022. Consistsum: Unsupervised opinion summarization with the consistency of aspect, senti- ment and semantic. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining. WSDM '22, page 467-475, New York, NY, USA. Association for Computing Machinery.
Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022. BRIO: Bringing order to abstractive summarization, In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 2890-2903, Dublin, Ireland. Association for Computational Lin- guistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: method for stochastic optimization. In 3rd Inter- national Conference for Learning Representations. ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings.
Ilya Loshchilov and Frank Hutter. 2017. Sgdr: Stochas- tic gradient descent with restarts. In 5th Inter- national Conference for Learning Representations, ICLR 2017, Palais des Congrès Neptune, Toulon, France, 2017, Conference Track Proceedings.
Svetlana Kiritchenko and Saif Mohammad. 2017. Best- worst scaling more reliable than rating scales; A case study on sentiment intensity annotation. In Proceed- ings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), pages 465-470, Vancouver, Canada. Association for Computational Linguistics.
Jordan Louviere, Terry Flynn, and A. J. Marley. 2015. Worst Scaling: Theory, Methods and Applica- lions. Cambridge University Press.
Ziming Mao, Chen Henry Wu, Ansong Ni, Yusen Zhang,
Rui 7hang, Tao Yu, Budhaditya Deb, Chenguang Zhu, Ahmed Awadallah, and Dragomir Radev. 2022. DYLE: Dynamic latent extraction for abstractive long-input summarization. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 1687-1698, Dublin, Ireland. Association for Compu- tational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-seqqeence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7888 Online. Association for Computa- tional Linguisties.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013. Distributed representa- tions of words and phrases and their compositionality. In Advances in Neural Information Processing Sys- tems, volume 26. Curran Associates, Inc.
Chunyuan Li. Xiang Gao, Yuan Li, Baolin Peng, Xiujun Li, Yizhe Zhang, and Jianfeng Gao. 2020. Optimus: Organizing sentences via pre-trained modeling of latent space. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 4678-4699, Online. Association for Computational Linguistics.
OpenAl. 2023. Gpt-4 technical report.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2079: Demonstrations.
Chin-Yew Lin. 2004. ROUGE: package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Nadav Oved and Ran Levy. 2021. PASS: Perturb and- select summarizer for product reviews. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Natural Language Processing (Volume 1: Long Papers), pages 351-365, Online. Association for Computational Linguisties.
Yang Liu and Mirella Lapata. 2019. Text summariza- tion with pretrained encoders. In Proceedings of the 2019 Conference Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLLL pages 3730-3740, Hong Kong, China, Association for Computational Linguistics.
Joint Conference
Bo Pang, Erik Nijkamp, Wojeiech Kryscinski, Silvio Savarese, Yingbo Zhou, and Caiming Xiong. 2023. Long document summarization with top-down and bottom-up inference. In Findings of the Associ- ation for Computational Linguistics: EACL 2023, pages 1267-1284. Dubrovnik, Croatia. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1 11692.

---

Romain Paulus, Caiming Xiong, and Richard Socher. 2017. deep reinforced model for abstractive sum- marization. CoRR, abs/1 705.04304.
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, and Amr Ahmed. 2020. Big bird: Transformers for longer sequences. In Proceedings of the 34th Interna- tional Conference Neural Information Processing Systems, NIPS' 20, Red Hook, NY, USA. Curran As- sociates Inc.
Ofrr Press and Lior Wolf. 2017. Using the output em- bedding improve language models. In Proceedings of the I 15th Conference of the European Chapter of the Association for Computational Linguistics: Vol- ume 2, Short Papers, pages 157-163, Valencia, Spain. Association for Computational Linguistics.
Wen Zhang, Yang Feng, Fandong Meng, Di You, and Qun Liu. 2019. Bridging the gap between training and inference for neural machine translation. In Pro- ceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4334- 4343, Florence, Italy, Association for Computational Linguistics.
Alec Radford, Rafal Józefowicz, and Ilya Sutskever. 2017. Learning to generate reviews and discovering sentiment. CoRR, abs/1704.01444
Gaetano Rossiello, Pierpaolo Basile, and Giovanni Se- meraro. 2017. Centroid- based text summarization through compositionality of word embeddings. In Proceedings of the MultiLing 2017 Workshop Sum- marization and Summary Evaluation Across Source Types and Genres, pages 12-21, Valencia, Spain. As- sociation for Computational Linguistics.
Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang. Xipeng Qiu, and Xuanjing Huang. 2020. Extractive summarization as text matching. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6197-6208, Online. Association for Computational Linguisties.
Ori Shapira and Ran Levy. 2020. Massive multi- document summarization of product reviews with weak supervision.
Shichao Sun and Wenjie Li. 2021. Alleviating expo- sure bias via contrastive learning for abstractive text summarization. CoRR, 111..1888166
Laurens der Maaten and Geoffrey E. Hinton. 2008. Visualizing high-dimensional data using t-sne, Jour- nal of Machine Learning Research, 9:2579-2605.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Ke Wang and Xiaojun Wan. 2021. TransSum: Trans- lating aspect and sentiment embeddings for self- supervised opinion summarization. In Findings of the Association for Computational Lingwistics: ACI- IJCNLP 2021, pages 729-742, Online. Association for Computational Linguistics.
Lu Wang and Wang Ling. 2016. Neural network-based abstract generation for opinions and arguments. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 47-57, San Diego. California. Association for Computational Linguistics.
Ronald Williams. 1992. Simple statistical gradient- following algorithms for connectionist reinforcement learning. Machine Learning, 8(3):229-22) 256.
Shusheng Xu, Xingxing Zhang, Yi Wu, and Furu Wei. 2021. Sequence level contrastive learning for text summarization. CoRR, abs/2109.03481.

---

AmaSum
Train
Dev
Test
AmaSum
Entity
26,660
3,302
3,362
Hyperparameter
Pros
Cons
Verdict
RT
Reviews per Ent.
77.78
77.76
76.80
Tokens per Rev.
53.98
54.13
53.97
bsz
16
16
16
8
Tokens per Sum.
bsz I1
16
16
16
16
Pros
39.13
39.21
38.81
warmup
5,000
5,000
5,000
500
Cons
18.34
18.31
17.81
warmup 11
3,000
3,000
3,000
300
Verdict
21.72
21.97
21.59
7 (Eq. 13) lenpen
0.1
1.0
0.1
0.1
Rotten Tomatoes
Train
Dev
Test
0.5
0.5
1.0
1.0
minlen
35
25
25
30
Entity
2,453
536
737
Reviews per Ent.
74.39
74.85
73.26
Table 7: Hyperparameter setting. bsz 1, II denote the batch sizes in the training stages. minlen stands for the minimum generation length in the beam search algorithm.
Tokens per Rev.
26.52
26.32
26.41
Tokens per Sum.
23.80
23.59
23.82
Table 6: Statistics of the preprocessed datasets. The average numbers are taken except for "Entity".
2020), and BERTCENT (Amplayo al., 2021b), which take encodings from word2vec (Mikolov et al., 2013), LSTM-based model (Radford et al.. 2017). and BERT (Devlin et al., 2019) as the in- put representations; unsupervised abstractive mod- els (m) OPINOSIS (Ganesan et al., 2010), a graph- based model that leverages token-level redundancy to summarize text, (n) DENOISESUM (Amplayo and Lapata, 2020), which re-formulates the sum- marization task denoising task; and weakly supervised model (o) PLANSUM (Amplayo et al., 2021b), which constructs the synthetic dataset with a Dirichlet distribution parametrized by content planner.
A Dataset Statisties
The statistics of the datasets after preprocessing shown in Table 6. The average numbers taken except for "Entity".
B Baselines
On AmaSum dataset, the baselines are unsuper- vised extractive models (a) LEXRANK (Erkan and Radev, 2004), PageRank-like algorithm that ex- tracts sentences based on graph centrality, (b) EXTSUM (Bražinskas et al., 2021), which uses the same ROUGE greedy heuristic as in Liu and Lapata (2019); unsupervised abstractive models (c) MEANSUM (Chu and Liu, 2019), which gener- ates opinion summary by reconstructing the mean review embeddings, (d) COPYCAT (Bražinskas et al., 2020b), a VAE summarizer with hierarchical continuous latent representations to model products and individual reviews; supervised abstractive mod- els (e) SELSUM, model jointly learns to select in- formative subsets of reviews and summarizing the opinions, (f) LONGFORMER, long-range model with an attention mechanism that scales linearly with sequence length, (g) BRIO, the state-of-the- art model of general abstractive summarization; and LLM-related solutions (h) GPT-3355-tubb (i) QG (Bhaskar et al., 2023), pipeline where reviews are summarized by QFSumm (Ahuja et al., 2022) and GPT-3 (Brown al., 2020), specifically the text-curie-001 model successively.
C Implementation Details
The codes we used for fine-tuning the PLMs sentiment analysis, contrastive information valu- ation, and the training stage were implemented with Fairseq (Ott et al., 2019) library. For senti- ment analysis, we set the learning rate to le-05 and updated the model parameters with the Adam optimizer. For contrastive information valuation, the margin À in Eq. 7 le-02. We set the learn- ing rate 3e-05 and adopted the Adam optimizer with the cosine learning rate scheduler (Loshchilov and Hutter, 2017). The minimum and maximum learning rates were le-08 and 3e-05.
During the training stage I, the input reviews were independently encoded, and the concatenated hidden states of the reviews were attended by the decoder predict the summary. Following Press and Wolf (2017), the token embeddings were shared across the encoder and decoder for regular- ization. We used the learning rate of 3e-05 and
On RT dataset are some other baselines: unsuper- vised extractive models (j) W2VCENT (Rossiello et al., 2017), (k) SNCENT (Amplayo and Lapata,

---

Here are some reviEws seporated by </5> of product.
Here ore some reviews separated by </5> of product.
<
review
</1>
review;
review,


review;
</<>
review,
Summarize the pros of the product.
Summiorie the cons of the product
Hete are some reviews seporated by </5> of product.
Here are some reviews seporated by 
 of movie MOV.
nevlew,
<
review,
4/5>
review;
review,
</S>
review
neview,
Summorize the reviews.
Summarizz the reviews.
Figure 3: Prompts for LLMs in our experiments. The four prompts were used for pros, cons, and verdict in AmaSun and Rotten Tomatoes order.
the Adam optimizer (Kingma and Ba, 2015) with 5,000 warmup steps for model optimization. The decoding strategy was beam search with beam size of 5 and trigram blocking (Paulus et al., 2017).
Value
R-1
R-2
R-L
K 6
21.36
4.30
15.87
K 8
22.16
4.89
16.62
K 10
23.20
5.56
17.28
In the training stage II, the reviews in the subset were joined with the separator 
 before being fed to the model, and M 16 candidate sum- maries were collected for every training sample. We used the Adam optimizer with the same learn- ing rate scheduler as BRIO and changed the max- imum learning rate to le-03. We used the margin À of le-03 in Eq. for all experiments. For sum- mary generation, we used beam search of size 5. Particularly, We distinguished between the length penalty hyperparameter a in Eq. 12 and lenpen in the beam search algorithm: the former was fixed at 2.0, and the latter differed across the targets. Other detailed hyperparameters
K 12 22.54
5.40
16.58
Table 8: Results of the experiments with varying K and random review subsets on RT dataset. The highest scores are shown in bold.
From the results Table 8, it can be inferred that a too-small value of K can cause information de- ficiency, and a too-large one may introduce the sparsity problem even after the review selection, so we didn't change the value of K.
listed in Table 7. For the baselines, we adapted BRIO to the task of large-scale opinion summarization. Concretely. while preparing the input, we always sampled R1-K out of BI.N and joined the K reviews with the separator 
, The input of BRIO and CHATGPT the optimal review subset in Sec. 3.2 due to the maximum input length of 4096; LONGFORMER (LED-base- 16384) and QG received the original review set as input. The prompts for the LLMs are presented in Fig. 3.
E Human Evaluation
BWS is known to produce more reliable results than raking scales (Kiritchenko and Mohammad, 2017) and widely used in opinion summarization studies. We randomly selected 30 samples from the pros, cons, and verdict partition of AmaSum's 's test set severally and recruited 6 volunteers. The volunteers were asked to choose one best and one worst summary from four summaries for three cri- teria and report the confidence of their choices. For each volunteer's response, the best model received +1, the worst model received -1, and the rest of the models received scores. Taking the confidence as weight, the scores of volunteers were weighted and summed to get the final scores.
D Hyperparameter Selection
The hyperparameter K has significant impact on the performance of SUBSUMM. In this paper, followed the setting of the baseline model SEL- SUM and inherited K- 10 on AmaSum dataset for comparability. Before working on RT dataset, we had conducted experiments with varying and random review subsets to explore the best value.
About the criteria, Informativeness tells if the summary presents opinions about specific aspects of the entity in the round, Coherence measures how casy the summary is to read and reflects if the summary follows a natural ordering of facts,

---

Method
R-1
R-2
R-L
Sampling Strategy
RAND
22.73
5.27
16.80
SENTI-RAND
22.76
5.36
16.81
SENTI-INFO
23.44
5.84
17.25
BANTI-RAND-INFO
23.99
6.09
17.54
Training Scheme
SUBSUMM
24.96
6.66
19.08
w/o Stage
23.72
5.16
18.05
w/o Stage II
23.99
6.09
17.54
RAND in Stage
25.40
6.74
19.13
RAND in Stage 11
25.46
6.77
18.90
Table 9: Results of analysis experiments on RT. R-1/2/L. are the ROUGE- 1/2/L F-1 scores, and the highest scores in both blocks are shown in bold.
and Non- Mon-Redundancy measures the repetitions and unnecessary contents in the summary.
F Experiment Results
The results of the analysis experiments on RT reported in Table 9. We list set of example sum- maries from SUBSUMM and other baselines on the AmaS Sun dataset in Table

---

Method
Pros
Cons
Verdict
GOLD
A quality pumice stone crafted from 100% pure volcanic lava. Has good shape, making it easy to hold and apply pressure hard skin. Use the handy rope loop to hang the stone out to dry after use.
Some users find the coarse- ness rough and takes time to get used to.
Comfortable to hold and use, this all natural vol- canic pumice stone works hard calluses and tough skin.
SELSUM
Made of natural materi- als that are easy to clean. Comes with rope at- tached to hang in the shower for easy use.
Some users find difficult to remove dead skin while it's in use. May not re- move all dead skin on the feet, but it's good choice for those with dry skin.
good choice you're looking for natural pumice stone that's easy to hold and use.
BRIO
A before and after, it was a bit embarrassing, and my feet were few deep cracks for few scrubs my feet are gone and couple scrubs
Users report that the pumice stone is not as smooth as they would like. Some reports of the stone breaking.
Pumice stone is the real deal and appreciate that. using it for the first time is a great experience.
CHATGPT
Pumiice Valley's natural pumice stone effective in removing dead skin, comes with a handy rope for easy use, and has a fine grain that doesn't
The stone is not sharp or rough enough to re- move dry skin, may be chipped or in poor shape, it can
The pumice stone effec- tive in removing dead skin and smoothing heels, but may not work well on ex- isting calluses.
SUBSUMM
Natural lava stone that's easy to use and easy to clean. Comes with a rope for hanging in the shower. Comes in variety of sizes. Made from lava.
May be too rough for some users. Some users find it difficult to remove the calluses from the stone. May not remove all cal- luses.
You're looking for a pumice stone that's easy use and clean, this is the one to buy.
Table 10: Example summaries from SUBSUMM and other baselines on AmaSum. Contents that coincide with the reference summaries or appear erroneous for opinion summarization are highlighted.


Reviews:
[{"review_id": "GXVGbMtFfA", "reviewer": "Reviewer_RBcE", "summary": "", "questions": "- Clarification on K Value in Experiments:  The paper would benefit from explicitly stating the value of K used in the experiments within the main body of the text. If this information is currently relegated to the appendix, it should be brought forward to the main part of the paper for better accessibility and understanding.\n- The paper lacks clarity on how different subsets of reviews are chosen from Rotten Tomatoes data. It's also unclear which category the summary belongs to—pros, cons, or verdict. These aspects require further explanation for better understanding.\n- Experiments involving retrieve and generate models, while not strictly mandatory, have become increasingly pertinent in the current age. The reviewer is understandably curious about your paper's positioning within the framework of \"Retrieve relevant context and generate.\" To address this curiosity, it would be valuable if you could furnish experiments demonstrating the application of your retrieval strategy on a Language Model (LM) like LLaMa and observe the resulting outcomes. Such experiments would serve to validate the relevance and effectiveness of your retrieval strategies independently of the specific language model employed", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments!\n***\n***Responses to the reasons to reject***\n\n* *Response to R1*\n\n&emsp; (1) Our work’s main contribution is leveraging explicit information selection mechanisms to tackle extensive input reviews. To emphasize the selective ability of the proposed framework, we prioritize *SelSum* and *BRIO*, which follow a similar select-then-summarize pipeline, for comparison.\n\n&emsp; (2) We had considered LED-base-16384 before, but we are sorry for not having sufficient computational resources then. \n\n&emsp; (3) In an effort to make up for this limitation, we rented additional GPUs and fine-tuned a `LED-base-16384` model. The results are as follows. \n\n&emsp;&emsp;(i) Rotten Tomatoes\nMethod|R-1|R-2|R-L\n-|-|-|-\nLongformer|24.96|6.34|18.40\nSubSumm|24.96|6.66|19.08\n\n&emsp;&emsp;(ii) AmaSum\n&emsp;|&emsp;|Pros|&emsp;|&emsp;|Cons|&emsp;|&emsp;|Verd|&emsp;\n-|-|-|-|-|-|-|-|-|-\nMethod|R-1|R-2|R-L|R-1|R-2|R-L|R-1|R-2|R-L\nLongformer|22.40|4.71|15.36|14.68|2.53|11.62|22.56|4.83|17.08\nSubSumm|26.25|4.96|24.18|16.72|3.00|14.80|25.36|5.04|19.58\n\nIt can be seen that as the input text gets longer (from *RT* to *AmaSum*), the edge of *SubSumm* over *Longformer* appears more remarkable. Intuitively, *Longformer*’s sparse attention mechanism plays an implicit selection role, while the sampling strategies of *SubSumm* consider the sentiment tendency and information value, which is more explicit and task-related. Therefore, our method is better than *Longformer* in handling massive reviews. We will supplement the results and the analysis in our revision.\n\n* *Response to R2*\n\n&emsp; (1) Considering the comparability, we follow the setting of the baseline model *SelSum* and inherit K=10 on the *AmaSum* dataset.\n\n&emsp; (2) Before working on the *RT* dataset, we had conducted a group of experiments with varying K and random review subsets to explore the best value: \nValue|R-1|R-2|R-L\n-|-|-|-\nK=6|21.36|4.30|15.87\nK=8|22.16|4.89|16.62\nK=10|23.20|5.56|17.28\nK=12|22.54|5.40|16.58\n\nIt can be inferred that a too small value of K can cause information deficiency, and a too large one may introduce the sparsity problem even after the selection, so we didn’t change the value of K. We will add the results to our revised paper.\n\n&emsp; (3) While designing the review sampling strategies, both criteria are calculated in a point-wise manner. The loss terms used for fine-tuning the LMs are either point-wise or pair-wise. Hence our method has better scalability than other list-wise approaches theoretically.\n\n&emsp; (4) To our knowledge, we have selected the two datasets with largest N among all the opinion summarization datasets. After preprocessing, the value of N varies with the samples from 10 to 100 in both datasets, which has already covered most situations. The SOTA performance can reflect acceptable scalability of our approach.\n***\n***Responses to the questions***\n* *Response to Q1*\n\n&emsp; Thanks a lot for your suggestion. We will move the clarification of the value of K (K=10) from the appendix (L937) to the implementation details in Sec. 4.1.\n\n* *Response to Q2*\n\n&emsp; Thank you! We have stated that the *verdict* summary is similar to general opinion summary in the dataset introduction (L390), and we will modify it to clarify that the data in *RT* and the *verdict* partition of *AmaSum* was equally treated in our experiments.\n\n* *Response to Q3*\n\n&emsp; (1) So far as we know, the most cutting-edge retrieve-and-generate application is the combination of advanced search engine and Large Language Model (LLM). Limited by the research conditions, we are not able to experiment with such foundation models. However, our proposal provides an idea of orientally retrieving information according to multiple criteria, then continuously improving model performance by manipulating different information, which we think to be applicable and promising in this framework.\n\n&emsp; (2) We have justified the effectiveness of the review sampling strategy itself in our experiments.\n\n&emsp; &emsp; (i) In the comparison experiments (Table 1 & 2), *SubSumm*, *SelSum*, and *BRIO* employ the same LM (i.e., *BART*), yet our method outperforms the other two, evidencing that the performance gains from the framework are model-agnostic.\n\n&emsp; &emsp; (ii) In the ablation study (Table 4 & 7), we first demonstrate that the proposed sampling strategies differ in quality (top half of the tables), then verify the contribution of the optimal strategy in both training stages (the last two rows of the tables).\n\n&emsp; &emsp; (iii) Moreover, the supplementary experiments of *Longformer* also show that equipped with our sampling strategy set and the corresponding training scheme, the smaller LM can defeat the larger LM.\n***\n***Others***\n* We will cite the recent papers studying opinion summarization with a large number of reviews.\n* We are going to simplify the first sentence of the introduction in our revision. Thanks for reminding us!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- ***Refreshing idea***\nThis idea presents a refreshing and novel approach by combining various techniques to achieve opinion summarization.\n- ***Good empirical analysis***\n    \n    The paper conducts a robust empirical analysis, considering a substantial number of baselines and recent advances for comparison. Furthermore, the authors conduct thorough ablation studies, effectively justifying their choice for the final system design.", "reasons_to_reject": "- Mismatch between motivation and experiments - One notable aspect where the paper falls short is the absence of comparisons with models capable of handling long context, such as the Longformer or any other recently released model with extended context length. Including such comparisons would have enriched the analysis and strengthened the overall contribution of the study.\n- Experiemnts with different values of K and N - The paper lacks experiments for different values of K, which is a significant limitation. By exploring various values of K, the authors could have gained valuable insights into the performance and behavior of their method under different settings. Additionally, it is essential to assess how well the proposed method performs when N (the number of samples or data points) is higher compared to other existing methods, as this was a primary motivation behind the research. By conducting experiments with varying N, the authors could have demonstrated the scalability and effectiveness of their approach and provided a more comprehensive evaluation of its performance in real-world scenarios. This absence of experimentation limits the paper's ability to fully support its main claims and conclusions. Including experiments with different values of K and diverse N settings would significantly enhance the paper's scientific rigor and strengthen its contribution to the field.", "questions_to_authors": "- Clarification on K Value in Experiments:  The paper would benefit from explicitly stating the value of K used in the experiments within the main body of the text. If this information is currently relegated to the appendix, it should be brought forward to the main part of the paper for better accessibility and understanding.\n- The paper lacks clarity on how different subsets of reviews are chosen from Rotten Tomatoes data. It's also unclear which category the summary belongs to—pros, cons, or verdict. These aspects require further explanation for better understanding.\n- Experiments involving retrieve and generate models, while not strictly mandatory, have become increasingly pertinent in the current age. The reviewer is understandably curious about your paper's positioning within the framework of \"Retrieve relevant context and generate.\" To address this curiosity, it would be valuable if you could furnish experiments demonstrating the application of your retrieval strategy on a Language Model (LM) like LLaMa and observe the resulting outcomes. Such experiments would serve to validate the relevance and effectiveness of your retrieval strategies independently of the specific language model employed", "paper_topic_and_main_contributions": "In the realm of opinion summarization, one of the primary challenges lies in creating comprehensive summaries that encompass diverse perspectives. The task becomes even more complex when dealing with a large number of reviews. However, this paper introduces a novel approach to address this issue by leveraging sentiment analysis and contrasting information from various reviews to identify the most suitable ones for each type of summary. The proposed model undergoes two stages of training. In the first stage, it is trained using Maximum Likelihood Estimation (MLE) to generate the initial summaries. Subsequently, in the second stage, the model is further refined by utilizing its capability to produce reasonable summaries, and it is trained on its own generated outputs. This innovative method shows promise in enhancing opinion summarization and streamlining the selection of reviews for different perspectives, ultimately leading to more effective and insightful summaries.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Line 130 - “These methods work well…” - Is this an opinion of the paper. Can you add a previous work that shows that the previous works dont work when the number of reviews increase", "Typos_Grammar_Style_and_Presentation_Improvements": "The first line in the introduction is too cumbersome to read. I think you can remove the heavy sounding words and simplify it. "}, "extra_text": {}}, {"review_id": "shikfCDLTR", "reviewer": "Reviewer_DG5P", "summary": "", "questions": "- I think this study is a really nice attempt. I'm really sad this is not an ARR submission. However, if you can address all the concerns raised in the reasons to reject during the rebuttal, I would consider raising the score.\n\n# after rebuttal\nThanks for taking your time! I updated the score.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback!\n***\n***Responses to the reasons to reject***\n\n* *Response to R1-R3*\n\n&emsp; (1) Our work’s main contribution is leveraging explicit information selection mechanisms to tackle extensive input reviews. To emphasize the selective ability of the proposed framework, we prioritize *SelSum* and *BRIO*, which follow a similar select-then-summarize pipeline, for comparison.\n\n&emsp; (2) We had considered LED-base-16384 before, but we are sorry for not having sufficient computational resources then. \n\n&emsp; (3) Thanks for your attention to our experiment data! In an effort to make up for this limitation, we rented additional GPUs and fine-tuned a `LED-base-16384` model. The results are as follows. \n\n&emsp;&emsp;(i) Rotten Tomatoes\nMethod|R-1|R-2|R-L\n-|-|-|-\nLongformer|24.96|6.34|18.40\nSubSumm|24.96|6.66|19.08\n\n&emsp;&emsp;(ii) AmaSum\n&emsp;|&emsp;|Pros|&emsp;|&emsp;|Cons|&emsp;|&emsp;|Verd|&emsp;\n-|-|-|-|-|-|-|-|-|-\nMethod|R-1|R-2|R-L|R-1|R-2|R-L|R-1|R-2|R-L\nLongformer|22.40|4.71|15.36|14.68|2.53|11.62|22.56|4.83|17.08\nSubSumm|26.25|4.96|24.18|16.72|3.00|14.80|25.36|5.04|19.58\n\nIt can be seen that as the input text gets longer (from *RT* to *AmaSum*), the edge of *SubSumm* over *Longformer* appears more remarkable. Intuitively, *Longformer*’s sparse attention mechanism plays an implicit selection role, while the sampling strategies of *SubSumm* consider the sentiment tendency and information value, which is more explicit and task-related. Therefore, our method is better than *Longformer* in handling massive reviews. We will supplement the results and the analysis in our revision.\n\n* *Response to R4*\n\n&emsp; (1) As in L968, the input of *ChatGPT* is the same as our summarizer, which is the optimal review subset mentioned in Sec. 3.2, L376.\n\n&emsp; (2) Unfortunately there are no other oracle subsets of the *AmaSum* or *RT* released. We trained *BRIO* on the optimal review subsets in our paper, and the results are as follows, which are better but still surpassed by our method. We are going to report the results in our revision.\n\n&emsp;&emsp;(i) Rotten Tomatoes\nMethod|R-1|R-2|R-L\n-|-|-|-\nBRIO (oracle)|23.72|5.16|18.05\nSubSumm|24.96|6.66|19.08\n\n&emsp;&emsp;(ii) AmaSum\n&emsp;|&emsp;|Pros|&emsp;|&emsp;|Cons|&emsp;|&emsp;|Verd|&emsp;\n-|-|-|-|-|-|-|-|-|-\nMethod|R-1|R-2|R-L|R-1|R-2|R-L|R-1|R-2|R-L\nBRIO (oracle)|25.48|4.58|23.50|16.65|2.94|14.60|24.93|4.78|19.44\nSubSumm|26.25|4.96|24.18|16.72|3.00|14.80|25.36|5.04|19.58\n\n* *Response to R5*\n\n&emsp; (1) Although ROUGE has a long history and apparent limitations, it is very frequently adopted for evaluation of the opinion summarization methods.\n\n&emsp; (2) Before the human evaluation, we compared the summaries generated by different systems preliminarily. The summaries of *ChatGPT* are fluent, yet they do suffer from more severe hallucination than our model, which may compromised their ROUGE scores.\n\n&emsp; (3) We have conducted an extra human evaluation to compare *SubSumm* with *ChatGPT*. 50 cases were randomly sampled from the two test sets (15 cases from each partition in *AmaSum* and 5 cases from *RT*). The recruited 6 volunteers were asked to conduct pair-wise comparisons without the reference summaries. The results are as follows:\n\nData|SubSumm vs. ChatGPT\n-|-\nAmaSum - cons|66.67% vs. 33.33%\nAmaSum - pros|65.56% vs. 34.44%\nAmaSum - verd|88.89% vs. 11.11%\nRotten Tomatoes|70.00% vs. 30.00%\n\nIt seems that the raters prefer the summaries generated by *SubSumm*. An obvious issue of *ChatGPT* is that it can’t control the output length within a few calls when the input is overlong. Consequently, most of the summaries generated are either excessively long or abruptly truncated with the maximum length argument fixed. We will report the results and the corresponding analysis in our revision. For (2) and (3) you may refer to Table 8 to see some supporting cases. \n\n***\n***Others***\n* We will supplement the references with all the mentioned papers. Thanks!\n* Again, we are grateful to receive your affirmation, and we would appreciate it if you can reconsider your score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- It shows improvement compared to the strong supervised opinion summarization baselines.\n- It performs human evaluation using best-worst scaling (which is preferable) and demonstrates the effectiveness of the proposed system.", "reasons_to_reject": "- The biggest issue with this study is the lack of comparison with long-document summarization studies, such as sparse attention-based methods and reduce-then-generate approaches.\n- I calculated the dataset statistics for your specific datasets and found that sparse attention-based encoder-decoder models, which often take 16k tokens as input, can handle all the reviews as input.\n- So, you should try the long-document summarization methods for a more comprehensive comparison.\n\n- I feel the BRIO and ChatGPT baselines are a bit unfair. Randomly sampled input is used for them. For training BRIO, I believe you should use an oracle set of reviews as input and perform a pipeline-based method for inference. For ChatGPT, you should use better retriever for the sampling stage.\n- As of 2023, I feel it is hardly acceptable to show a ROUGE score and claim that it is a superior model to ChatGPT. It is evident that ChatGPT is not aligned with the training set distribution, making it hard to achieve high ROUGE scores. However, with human evaluation, it can perform better. So, it feels a bit suspicious to claim that the proposed method is better than ChatGPT.", "questions_to_authors": "- I think this study is a really nice attempt. I'm really sad this is not an ARR submission. However, if you can address all the concerns raised in the reasons to reject during the rebuttal, I would consider raising the score.\n\n# after rebuttal\nThanks for taking your time! I updated the score.", "paper_topic_and_main_contributions": "- This paper performs supervised opinion summarization with a long input setting. The study proposes a two-stage approach to perform the task. The experimental results show that the proposed method improves over the strong supervised opinion summarization baselines.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- [1] [Longformer: The Long-Document Transformer](https://arxiv.org/abs/2004.05150) (Beltagy et al., arXiv)\n- [2] [Big Bird: Transformers for Longer Sequences](https://proceedings.neurips.cc/paper/2020/file/c8512d142a2d849725f31a9a7a361ab9-Paper.pdf) (Zaheer et al., NeurIPS 2020)\n- [3] [DYLE: Dynamic Latent Extraction for Abstractive Long-Input Summarization](https://aclanthology.org/2022.acl-long.118) (Mao et al., ACL 2022)\n- [4] [Long Document Summarization with Top-down and Bottom-up Inference](https://aclanthology.org/2023.findings-eacl.94) (Pang et al., Findings 2023)"}}, {"review_id": "m0usOksTlX", "reviewer": "Reviewer_AegH", "summary": "", "questions": "1. Have you tested SUBSUMM on any other datasets beyond online reviews to prove its generalization ability?\n2. The sampling strategies use sentiment analysis and ROUGE with reference summaries. How would the framework adapt to a purely unsupervised setting?\n3. What is the inference time of SUBSUMM on large review sets? Is it applicable for real-time usage?\n4. What were the main challenges faced in training the 2-stage model? Did you try any other scheduling or curriculum strategies?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments!\n***\n***Responses to the reasons to reject***\n\n* *Response to R1*\n\n&emsp; Thanks a lot for your attention to our research problem. Though the most commonly-used online review datasets are *Yelp* and *Amazon* with 8 reviews per product, we still observe plentiful products with a large number of reviews on some e-commerce platforms. For example, a fidget spinner (ASIN: B08HWDXFMS) on Amazon has 126 reviews, and a restaurant on Yelp could have thousands of comments. We think our method with information selection design can better facilitate reading in such scenarios.\n\n* *Response to R2*\n\n&emsp; (1) Our work is an attempt to utilize the evaluation metrics to take the place of the aspect and sentiment terms when guiding the summarizer. Specifically, we integrate ROUGE-related information into the review sampling strategies and the training loss. In Table 1 & 2, *SubSumm* establishes new SOTA on both benchmarks, outperforming the baseline *SelSum* which explicitly models the aspect information, which suggests the rationality of this trial.\n\n&emsp; (2) As mentioned in our conclusion and limitations, we agree that the model performance would be further improved if the evaluation metrics and traditional aspect information can be integrated for guidance. We plan to explore it in the future.\n\n* *Response to R3*\n\n&emsp; (1) As the number of the input reviews increases, a large part of the reviews would become dispensable due to data sparsity. Therefore, selecting valuable subsets of reviews could mitigate the sparsity problem and reduce the computation cost.\n\n&emsp; (2) We found that diversity is not frequently concerned in the field, both general summarization, for example, \n\n> Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization (He et al., ACL 2023)\n\n> Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (Liu et al., ACL 2023)\n\nand opinion summarization, for example,\n\n> Attributable and Scalable Opinion Summarization (Hosking et al., ACL 2023)\n\n> From Key Points to Key Point Hierarchy: Structured and Expressive Opinion Summarization (Cattan et al., ACL 2023)\n\n&emsp; (3) From the results, no serious issues of repetition or diversity are found. You may refer to the human evaluation results of *Informativeness* and *Non-Redundancy* in Table 3. Besides, we computed the *Distinct-N* scores of our method and part of the baselines (supervised and LLM-related) on *AmaSum*. *Distinct-N* is a metric that measures the diversity of a sentence by counting distinct n-grams in it and penalizing the sentence with lots of repeated words. It was proposed by the following paper:\n> A Diversity-Promoting Objective Function for Neural Conversation Models (Li et al., NAACL 2016)\n\nAnd here are the results:\n\n&emsp;|Pros|&emsp;|Cons|&emsp;|Verd|&emsp;\n-|-|-|-|-|-|-\nMethod|D-1|D-2|D-1|D-2|D-1|D-2\nSelSum|**22.05**|**67.22**|23.57|68.25|**24.91**|**73.70**\nBRIO|$\\underline{\\text{16.70}}$|$\\underline{\\text{57.77}}$|20.88|$\\underline{\\text{65.16}}$|22.07|$\\underline{\\text{69.21}}$\nSubSumm|17.33|59.15|21.05|65.83|22.52|70.18\nQG|20.68|65.17|**23.87**|69.41|22.52|69.48\nChatGPT|17.13|65.05|$\\underline{\\text{20.23}}$|**69.83**|$\\underline{\\text{20.94}}$|70.20\n\nwhere the highest scores are shown in bold and the lowest scores are underlined. *SubSumm* shows comparable diversity among the powerful baselines. Especially, it performs better than *BRIO* which has no special selection mechanism.\n\n&emsp; (4) When training the summarizer, we introduce stochastic factors to the optimal sampling strategy (i.e., *Sentiment-Information Ranking (modified)* in L340), where we perform weighted sampling to balance the diversity and quality of the selected review subsets.\n***\n***Responses to the questions***\n* *Response to Q1*\n\n&emsp; Since the current target of opinion summarization is mainly epitomizing the online reviews, we have not tried other datasets yet. We are planning to scale our study to other domains.\n\n* *Response to Q2*\n\n&emsp; The rankings of the reviews in the “information valuation” step should rely on some reference-free metric rather than ROUGE. Moreover, the training scheme is to be replaced as it is fully supervised.\n\n* *Response to Q3*\n\n&emsp; The average inference speed is less than 10 seconds per sample  (sampling + summarization) on our devices, which is similar to the baseline *SelSum* and acceptable for practical use. \n\n* *Response to Q4*\n\n&emsp; (1) During training, the biggest challenge was the dilemma between the limited model scale and the massive input reviews. We come up with a set of flexible review sampling strategies to reduce the input volume and maintain valuable information, which forms one of the contributions of our paper.\n\n&emsp; (2) We haven’t tried other training strategies yet, but we are going to experiment with more training schemes and model architectures in our future work.\n\n***\n***Others***\n* We are going to cite the *FineSum* paper in our revision. Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It addresses an important problem of summarizing large volumes of opinion text from different perspectives, which is useful for many real-world applications like marketing analysis and decision making.\n2. The proposed framework SUBSUMM achieves new state-of-the-art results on two benchmark datasets, and the in-depth analysis offers insights into the contribution of different components of the framework.", "reasons_to_reject": "1. The problem setup itself is somewhat artificial - summarizing a very large set of reviews (average 76-74 reviews per product) when most real products would not have that many reviews.\n2. There is no component in the framework that specifically models aspect information which is crucial for opinion summarization. It is mainly driven by ROUGE scores.\n3. The two-stage training technique feels a bit ad-hoc. Ideally, the model should be able to learn from all data rather than just optimal subsets in later stages. Also, no mechanisms are introduced to reduce repetition and ensure diversity in the generated summaries.", "questions_to_authors": "1. Have you tested SUBSUMM on any other datasets beyond online reviews to prove its generalization ability?\n2. The sampling strategies use sentiment analysis and ROUGE with reference summaries. How would the framework adapt to a purely unsupervised setting?\n3. What is the inference time of SUBSUMM on large review sets? Is it applicable for real-time usage?\n4. What were the main challenges faced in training the 2-stage model? Did you try any other scheduling or curriculum strategies?", "paper_topic_and_main_contributions": "1. This paper proposes a framework called SUBSUMM for large-scale and multi-perspective opinion summarization. It introduces a review sampling strategy set based on sentiment analysis and contrastive information valuation to select high-quality review subsets from a large set of reviews.\n2. It proposes a two-stage training scheme that allows the model to learn from sub-optimal and optimal review subsets successively. The second stage uses contrastive learning to assign higher probability to better candidate summaries.\n3. Experiments on two datasets AmaSum and Rotten Tomatoes show SUBSUMM outperforms previous state-of-the-art models in generating pros, cons and verdict summaries from hundreds of reviews.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "1. Suyu Ge, Jiaxin Huang, Yu Meng, Jiawei Han. FineSum: Target-Oriented, Fine-Grained Opinion Summarization. WSDM 2023."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a new supervised opinion summarization model for a large number of reviews. A new sampling strategy based on sentiment analysis and contrastive information evaluation is introduced for the selection of high-quality review subsets from a large input set.  The study suggests a two-stage process for carrying out the task: training using MLE to generate the initial summaries and then refining the model with contrastive learning to assign a higher probability to better candidate summaries. The results of the experiments demonstrate that the suggested method outperforms the robust supervised opinion summarization baselines. 

Pros:
1. New sampling strategy for selecting the best candidates for review summarization
2. A novel two-stage training scheme 
3. The experimental results, including human evaluation using best-worst scaling, show the superiority of the proposed system compared to the strong supervised opinion summarization baselines.

Cons:
Some reviewers noticed missing comparisons with LLMs designed for long inputs. The authors ran experiments and reported the results in their rebuttals. 
The authors responded to every reviewer with detailed letters. Two reviewers increased their scores.
Decision: Accept-Findings