Synthesize, if you do not have: Effective Synthetic Dataset Creation Strategies for Self-Supervised Opinion Summarization in E-commerce
Authors: Tejpalsingh Siledar
Suman Banerjee
Amey Patil
Sudhanshu Shekhar Singh
Muthusamy Chelliah
Nikesh Garera
Pushpak Bhattacharyya
Venue: EMNLP 2023 Findings (2023)
Keywords: opinion summarization
summarization
ecommerce
nlp

Abstract: In e-commerce, opinion summarization is the process of condensing the opinions presented in product reviews. However, the absence of
large amounts of supervised datasets presents challenges in generating both aspect-specific and general opinion summaries. Existing
approaches have attempted to address these challenges through synthetic dataset creation (SDC). However, general opinion summarization models struggle to generate summaries faithful to the input reviews whereas aspect-specific opinion summarization models are limited due to their reliance on human-specified aspects and seed words. To address this, we propose SDC strategies tailored for general and aspect-specific opinion summarization. We experimented on three e-commerce test sets: Oposum+, Amazon, and Flipkart. For general opinion summarization, pre-trained language model (PLM) fine-tuned on our general synthetic dataset surpass the SOTA on average by 2.3 R1 points. Faithfulness evaluation metrics and human evaluations indicate that our model-generated summaries are more faithful to the input compared to others. For aspect-specific opinion summarization, PLM fine-tuned on our aspect-specific synthetic dataset surpass SOTA by ∼ 1 R1 point without the aid of any human-specified aspects or seed words.

Synthesize, if you do not have: Effective Synthetic Dataset Creation Strategies for Self-Supervised Opinion Summarization in E-commerce
Tejpalsingh Siledar', x Suman Banerjee Amey Patil", Sudhanshu Shekhar Singh",
Muthusamy Chelliah Nikesh Garera Pushpak Bhattacharyya
Department of Computer Science and Engineering, IIT Bombay, India
Flipkart, India
Abstract
MultimodalSum
In e-commerce, opinion summarization is the process of condensing the opinions presented in product reviews. However, the absence of large amounts of supervised datasets presents challenges generating both aspect-specific and general opinion summaries. Existing approaches have attempted to address these challenges through synthetic dataset creation (SDC). However, general opinion summariza- tion models struggle to generate summaries faithful the input reviews whereas aspect- specific opinion summarization models are lim- ited due to their reliance on human-specified aspects and seed words. To address this, we propose SDC strategies tailored for general and aspect-specific opinion summarization. We experimented three e-commerce test sets: Oposum+, Amazon, and Flipkart. For general opinion summarization, pre-trained language model (PLM) fine-tuned on our general syn- thetic dataset surpass the SOTA on average by 2.3 RI points. Faithfulness evaluation metrics and human evaluations indicate that model- generated summaries are more faithful to the input compared to others. For aspect-specific opinion summarization, PL.M fine-tuned on our sppect-specific synthetic dataset surpass SOTA by R1 point without the aid of any human- specified aspects seed words.
This is nice teapot, but the color is not bright the picture. It is more of a dark lurquoise than a light blue. was hoping would be more of an aqua bluc. but more like dark aqua. It still looks nice, hut I would have preferred the color to be more like the photo.
Our Model
This is a beautiful teapot, the color not the same as shown the picture. is more of a dark turqunise. The color is a little darker than the picture, but still a beautiful color. The only thing I don't like about it that the handle is very hard to open.
Table General Opinion Summarization. Example of faithfulness issue in MultimodalSum compared to our model. There is support for the information in red parts in the corresponding input reviews.
Bražinskas et al. (2020); Amplayo and Lapata (2020) proposed creating synthetic pairs by sam- pling one of the reviews as pseudo-summary. However, careful curation of synthetic datasets is essential in ensuring that the models generate sum- maries that are faithful the input as shown in Table 1. In this regard, we propose a synthetic dataset creation (SDC) approach for general opin- ion summarization that leverages lexical (R-1 F1 (RI) (Lin, 2004)) and semantic (cosine) similarities for pseudo-summan selection.
1 Introduction
Amplayo et al. (2021) proposed constructing synthetic datasets for aspect-specific opinion sum- marization. They trained multiple instance Iearn- ing (MIL) module (Keeler and Rumelhart, 1991) with silver-standard labels obtained using seed words filter out aspect-containing sentences. Shen al. (2023) proposed two solutions for gen- erating synthetic datasets without the MIL module. Their NLI-LOO method eliminated the need for seed words but still relied on human- -specfiee as- pects. This reliance on numan-specified aspects or
In the c-commerce domain, customer reviews are essential for making informed buying decisions. However, going through all the product reviews before making a decision is overwhelming. Opin- ion summarization is the process of summarizing the opinions presented in the reviews (Hu and Liu, 2006; Wang and Ling, 2016; Angelidis and Lap- ata, 2018). However, it is infeasible and expensive obtain reference summaries at a large scale for general and aspect-specific opinion summarization.

---

AceSum
Pre-delined aspoct-seed words mapping
Pre-defined mapping
looks: looks color stylish prety
Ports the hilm ports stopped working and the tv would black for about three to five seconds every few minutes, it hus all of the inputs for us, and more.
quality; quality material poor broke dunable
big space
Antomatic mapping
Pine-gralned clustess
sellie camera, camen,, camera quality, back camer,... display, display, quality.... batlery life. battery backup. battery capacity, battery... processor, intel processet....
Our Model
Ports i have had this tv for few months now and have had no issues with it, has all the hdmi ports i need and the picture quality is great.
Course-grateed clusters
disglay, display
camen,, comerrs, camera, ....
battery, hatlery
processor
Table 2: Aspect-specific Opinion Summarization. AceSum vs. Our Model generated summaries for the aspect Ports Our model does not rely on any human- specified aspects and seed words for training, making our approach generalizable to other domains where such aspects and seed words are defined.
Aspeet muppleg
display; screen, display, quality. cament selfie cammen, camera, cimenquality builery: battery
backup.... ...
processor. procossor, insel processor....
Table 3: An example the pre-defined aspect-seed words mapping and our automatic aspect mapping.
seed words limits aspect-specific opinion summa- rization only a few domains where such aspects or seed words are defined. In contrast, our pro- posed solution overcomes this limitation by not relying on any human-specified aspects or seed words and outperforms all existing approaches. Ta- ble 2 shows an example aspect-specific summary. In general opinion summarization, the input con- sists of reviews, and the output is a general sum- mary, whereas, in aspect-specific opinion summa- rization, the input consists of reviews and aspects. and the output is an aspect-specific summary. Our contributions are:
Synthetic Dataset Creation (SDC)
2.1 General Opinion Summarization
Following Bražinskas et al. (2020), we assume that a review ry can serve as a summary for a set reviews D;- This lets create training points (D2, similar to what the model will experience during inference. D; is limited to size k; enabling comparison with previous works where the number of input reviews is fixed (Chu and Liu, 2018).
M a * Msim X2 * Mrouge
(1)
T1
22
TN
T2
1. For general opinion summarization, matrix- based synthetic dataset creation strategy that leverages lexical (RI) and semantic (cosine) similarities for pseudo- pseudo-summary selection. Pre-trained language model (PLM) fine-tuned on our synthetic dataset of 945K instances outperforms the SOTA average by 2.3 RI points and generates summaries that are more faithful to the input reviews than alternatives, as evidenced by automatic and human evalua- tions (Section 1, Table 5, 7, 9).
0
M =
0
(2)
maj
0
0
TN
8; mean( topk( M[r, :)))
(3)
We encode reviews using sentence-transfoon (Reimers and Gurevych, 2019) and construct two matrices M and Mrouge E R NxN by comput- ing cosine similarity and R1 scores between them. We obtain a unified matrix M by adding the two matrices. A cell mi of the matrix M corresponds to a score between reviews r; and ry. To prevent self-comparisons, we assign zero to the diagonal entries of the matrix. Here, each row corresponds to a potential greudo-summan and its compari- son with all other reviews. For each such pseudo- summary r, we select the top-k (first k with the highest scores) reviews D; and compute the mean scores 8; across the selected reviews D;-
2. For sspect-specific opinion summarization, an aspect-specific synthetic dataset creation strat- egy that does not rely on any human-specified aspects seed words making our approach generalizable to any opinion summarization domain. Pre-trained language model (PLM) fine-tuned on our synthetic dataset of 350K instances outperforms the state-of- of-the-art by RI point (Section 2.2, Table 6).

---

Dataset
Oposum+
Amazon
Flipkart
synthetic datasets also enable the use of multiple aspects for generating a summary. Let a review E R (R is the set of all reviews for a product) consist of sentences (42, and the sentence- level aspects are {a.1: d.2,..) where any dij is a list of aspects. For each review we remove sentences Tith where ash Φ and concatenate sen- tences Bi and Fy on the condition that azy Na, # Φ. A review Tv, instead of sentences, can now be rep- resented consisting of portions (concatenation of 2/2, sentences) ( ...} with corresponding aspect-
No. of domains
6
3
No. of aspects
18
38
No. of lest set
60
32
147
No. of reviews per product
10
8
10
No. of summaries per product
3
1
No. of general summaries
180
96
No. of aspect summaries
540
676
Table 4: Data statistics for Oposum+, Amazon, and Flipkart test sets, Extractive summaries are underlined.
synthetic pair (Ds, r:) is considered only if the score 8; 2 threshold Tg. For our experiments, we set the value of λι, 12 Ty and k to be 0.5, 0.5, 0.38 and respectively.
lists (d:1 4:2 ...). We replace each aspect with its coarse saanse-rriied aspect using the aspect mappings. Next, we create synthetic triplets. For a portion, say, Tim belonging to r; we concatenate all por- tions Tith R_ (where R_; R (rx)) to form Com such that dim n Otk # Φ. We also keep track of the reviews Rim from R_; that contribute to the for- mation of Cim- Now, for each pair (Cim: Zim)) we compute a cosine similarity s8im and rouge score r8 and add them to form a unified score us as u.Sm X1 58 + X2 * 18 We consider only those pairs where the unified score um8 z threshold Ta Finally, (Dim: Gim: Fim) forms a syn- thetic triplet, where Dim is equal to Rim or a com- bination of Ram and some random reviews from R_; - Rim so that the final review set Dim is lim- ited to size k. For our experiments, we set the values of λ1, Tas k to 0.5, 0.5, 0.4, and 10 spectively. Appendix E.
2.2 Aspect-specific Opinion Summarization
Synthetic dataset creation for aspect-specific opin- ion summarization can be divided into two phases: aspect mapping and synthetic triplets creation. Aspect Mapping Prior methods relied on pre- defined mappings of seed words to aspects in order initially filter sentences and assign them to corre- sponding aspects. However, these pre-defined map- pings (Table 3) are seldom available. To address this limitation, we propose the automatic genera- tion of such aspect mappings. Our approach be- gins with the utilization of an InstructABSA mod- ule (Scaria et al., 2023) to extract aspects. Subse- quently, we employ sentence transformers and the fast clustering al algorithm' (Reimers and Gurevych, 2019) encode and cluster these aspects. We apply two distinct thresholds 0.7 and 0.9, to create coarse- grained and fine-grained clusters. The coarse- grained aspects function similarly seed words, serving as filters for sentence selection, whereas the fine-grained aspects correspond to the aspects which sentences will be mapped. Say, for product, we get coarse-grained clusters (cr,C,,... C2, and fine-grained clusters (f1, f2..... We map the coarse-grained cluster c; to fine-gran tieeggrined cluster f using the condition e n fy # & to create the auto- matic aspect mapping (Table 3). Appendix D. Synthetic Triplets Creation In the synthetic triplet creation phase, we construct triplets of the form /reviews, aspects, ascudo-summary to facili- tate supervised training of models that take reviews and aspects as input and the seeudo-summary as the output. In contrast to previous approaches, our
3 Experiments
3.1 Datasets
We conducted experiments using three e-commerce datasets: Oposum+ (Amplayo et al., 2021), Ama- zon (He and McAuley, 2016; Bražinskas al., 2020), and Flipkart (Siledar et al., 2023). The Oposum+ evaluation set contains extractive gen- eral summaries and aspect-specific abstractive sum- maries, Amazon contains general abstractive sum maries, and Flipkart contains only aspect-specific abstractive summaries. Statistics are Table Us- ing our SDC strategy, we generated 945K instances from Amazon for general opinion summarization whereas 350K instances from Oposum+ for aspect- specific opinion summarization. We test our mod- els on Oposum+ and Amazon for general whereas Oposum+ and Flipkart for sspect- opinion summarization. Appendix G.

---

Oposum+
Amazon
Opesum+
Flipkart
Model
axp?
R11
R21
RLT
RIT
R21
RLT
Model
exp?
RI1
R21
RLT
RIT
R21
RLT
Clustroid
33.44
11.00
20.54
29.27
4.41
17.78
LexRank
22.51
3.35
17.27
10.41
0.93
8.72
LexRank
x
35.42
10.22
20.92
29.46
5.53
17.74
Eutrrrii
QT
23.99
4.36
16.61
14.11
1.71
9.56
QT
37.72
14.65
21.69
34.04
7.03
18.08
Acesumau
26.16
5.75
18.55
Acesumee
38.48
15.17
22.82
SW-L000a
40.45
19.13
23.20
SW-LOO
28.14
6.10
19.51
gL
NLI-L.00
26.78
6.48
18.07
MeanSum
26.25
4.62
16.49
29.20
4.70
18.15
MeanSum
X
24.63
3.47
17.53
10.64
1.33
9.78
CopyCar
27.98
5.79
17.07
31.97
5.81
16
Absteactiv
CopyCat
26.17
4,30
18.20
13.48
1.92
10.35
Acesum
32.98
20.27
Acesum
29.53
6.79
21.06
Aborwstte
SW-LOD
36.19
12.17
21.11
SW-LOO
30.00
6.92
20.76
NLI-L.0
31.22
9.93
19.08
x
x
NLI-LOO
20.11
PlanSum
30.26
17.48
32.87
19.05
ASBOS
23.45
4.37
16.85
11:56
ConsistSum
X
33,32
5.94
21.41
MulimodaaSum
33.08
7.46
19.75
34,19
7.05
20.81
Model
30.95
21.73
15.99
TransSum
34.29
7.24
20.49
COOP
36.57
7.23
21.24
Table 6: Evaluation for aspect summaries Oposum+ and Flipkart test sets. asp? indicates systems that use nouaan-specifie aspects. Bold and underline indicate best and second-best scores using abstractive systems. Our model without relying on any human-specified as- pects or seed words learns the task of espect-speciffs opinion summarization and outperforms alternatives.
Our Model
X
36.57'
8,79
21.36
35.46
7.30
21.50
Table 5: Evaluation for general summaries on Opo- sum+ and Amazon test sets. asp? indicates systems that use human-specified aspects. Bold and underline indi- cate best and second-best scores using abstractive sys- tems. indicates pvalue < 0.05 on paired t-test against MultimoaaSumm. Our model outperforms existing mod- els on the task general opinion summarization.
-neg -ee--neut-ombinee (Scaria et al., 2023) model. For encoding reviews we use the sentence-tr (Reimers and Gurevych, 2019). Appendix J.
3.2 Baselines
General Opinion Summarization. Extractive approaches: Clustroid (Bražinskas et al., 2020), LexRank (Erkan and Radev, 2004), QT (Angelidis
Results and Analysis
et al., 2021), AceSumexi (Amplayo et al., 202 1), SW-LOO,.t ext and NLI-LOOes ex (Shen et al., 2023). For abstractive approaches we use MeanSum (Chu and Liu, 2019), CopyCat (Bražinskas al., 2020).
General Opinion Summarization. We use the ROUGE- (1,2,L F1 score (Lin, 2004) (R1, R2 & RL) to assess the generated summary quality. Ta- ble 5 reports results on general opinion summariza- tion. On both Oposum+ and Amazon, our model outperforms MultimodalSum. Overall, model achieves the best R1 and RL on Oposum+ and the best R2 and RL whereas second-best RI on the Amazon test set. Our synthetic dataset creation approach plays an essential role in our models per- forming better. By meticulously selecting (input reviews, pscudo-summary pairs that are lexically and semantically most similar one another, we enable our models to effectively learn the task of opinion summarization. Additionally, this also en- sures that the model generates summaries that are most faithful to the input reviews.
Acesum, SW-LOO, NLI-LOO, Plansum (Amplayo and Lapata, 2020), ConsistSum (Ke et al., 2022),
MultimodalSum (Im et al., 2021), TransSum (Wang and Wan, 2021), and COOP (Iso et al., 2021). Aspect-specific Opinion Summarization. We compare extractive approaches: LexRank, QT, AceSumest. SW-LOOex> exr. and NLI-LO0 For ab- stractive approaches
MeanSum, CopyCat, Acesum, SW-LOO, NLI-LOO, and ASBOS (Siledar et al., 2023). Appendix H.
3.3 Implementation Details
We used bart-large (Lewis et al., 2019) as our Transformer model to fine-tune on our synthetic datasets. We used lcarning rate of 2e 6, batch size of 8 for 5 epochs. We performed manual hyperparameter tuning on dev sets to select the best model to report scores the test set. During inference, we set the beam size to 2 and no repeat ngram to 3. For aspect-extraction, we use the
Aspect-specific Opinion Summarization. Table presents results on aspect-specific opinion sum- marization. Without using any human-specified as- pects and seed words, our model is able to achieve the best scores on both datasets across all metrics. As a result of automatic aspect mappings, our syn-

---

Model
SummaC 1
CTC 1
FacICC 1
FactGraph
OPOSUM+
Amazon
Flipkart
Our Model
Aspect
General
General
Aspect
PlanSum
0.33
0.81
0.16
0.21
Multimodalsam
0.38
0.46
0.53
w/ Both metrics
21.73
21.35
21.50
15.99
Our Model
0.40
0.85
0,68
0.66
w/ Cosine Similarity
20.25
20.07
19.48
15.60
of RI w/ Training Random
20.50
20.23
19.90
15.63
17.85
16.06
16.18
7.04
Table 7: Faithfulness Evaluation. Our model out- performs alternatives three faithfulness measuring metries:
Table 8: Ablation Study. Cosine Similarity or RI indicates the of only that metric. Training Ran- dom means randomly selecting reviews and pseudo- summary. We report the RI. scores on the test sets.
SummaC, FactCC, and FactGraph.
thetic datasets contain training points with a signif- icantly broader range of aspects compared to the pre-defined ones. This diversity in aspects ensures that the models trained on our synthetic datasets are better equipped to effectively Iearn the nuances of aspect-specific opinion summarization. Finally, as Flipkart contains only test set, we use models trained using the Oposum+ dataset for testing.
Amazon
Faithfulness
Coherent
Cancise
Flaency
PlanSum
-0.92
-0.91
-0.84
-0.72
Multimodalsum
0.41
0.45
0.40
0.31
Our Model
0.51
0.45
0.41
0.41
Table 9: Worst Scaling. Our model receives bet- ter scores on three criteria. Best values are in bold.
Analysis. For general opinion summarization, our model-generated summaries usually contain infor- mation that is verifiable from the input reviews, c.g., the color of the teapot being dark turquoise (Table 1). This we attribute to our SDC strategy for general synthetic datasets. For aspect-specific opinion summarization, our model is able to gener- ate aspect-specteee speciffc summaries without ever being explicitly trained aspects, e.g., ports (Table 2). This we intuit is possible as our model is trained on a wide range of automatically identified aspects instead of a few pre-defined ones. Finally, we do observe that aspect-specific summaries have a ten- dency to generate information irrelevant to the as- pect under consideration.
training models with random reviews and pseudo- summary and evaluating its performance the test set. Table 8 demonstrates our choice of metrics achieving improved performance.
Scal- ing (Louviere et al., 2015) to assess the quality of general opinion summaries. Three participants evaluated model summaries on faithulnnss, coher- ence, conciseness, and fluncc. They compared the model summaries to human-written summaries from the Amazon test set and selected the best and worst summaries accordingly. Table 9 shows that our model outperformed alternatives across all cri- teria. Appendix L.
Human Evaluation. We used
Faithfulness Evaluation. We use four automatic faithfulness evaluation metrics: SummaC (Laban et al., 1). CTC (Deng et al., 2021), ,FactCC (Kryscinski et al., 2019), and FactGraph (Ribeiro al., 2022) on the Amazon test set for general opinion summarization. Table 7 shows that our model scores higher than the other models on all the metrics indicating that the summaries generated from our models are much more faithful to the input reviews compared to others. Appendix K.
Conclusion and Future Work
We proposed synthetic dataset creation approaches for general and aspect-specific opinion summariza- tion. For general opinion summarization, our SDC approach enables models to generate summaries that are more faithful to the input than existing approaches, as indicated through automatic and hu- man evaluations. For aspect-specific opinion sum- marization, our SDC approach enabled models to outperform existing models without relying on any human-specified aspects or seed words, making our approach generalizable other opinion sum- marization domains where such human- -pean-sd aspects or seed words are not defined.
Ablation Study. We conduct ablation studies to analyze the impact of using different metrics and filtering strategies. Specifically, we compare re- sults using either cosine similarity or RI alone, finding that the scores decrease when only one met- ric is used. Furthermore, we assess the effective- ness of our pseudo-summary selection approach by
We plan to further extend our work domains other than e-commerce for opinion summarization.

---

Limitations
Papers), pages 1209-1225, Dublin, Ireland. Associa- tion for Computational Linguistics.
We showed results only on e-commerce datasets. However, the synthetic dataset creation approaches proposed here are not domain specific and are gen- eralizable to domains other than e-commerce. The other challenge specific opinion summarization is to summarize a large number of reviews. It is infeasible to input all the reviews to train model for summarization, limited by modern hardware capacity. As a result, existing approaches either resort to a hybrid approach of extractive filtering followed by abstractive summarization smaller set of input reviews for summarization. We resort to the second approach to make comparisons with existing approaches. Finally, we did not make comparisons with recent large language models for opinion summarization as our focus was on pushing for improvement in the synthetic dataset creation approaches to train smaller models and also per- form a fair comparison to existing approaches.
Arthur Bražinskas, Mirella Lapata, and Ivan Titov. 2020. Unsupervised opinion summarization as copycat- review generation. In Proceedings of the 58th Anmual Meeting of the Association, for Computational Lin- guisties, pages 5151-5169, Online. Association for Computational Linguistics.
Giuseppe Carenini, Jackie Chi Kit Cheung, and Adam Pauls, 2013. Multi-document summarization of eval- uative text. Computational Intelligence, 24())))) 29(4):545- 576.
Subhajit Chaudhury, Sarathkrishna Swaminathan, Chu- laka Gunasekara, Maxwell Crouse, Srinivas Rav- ishankar, Daiki Kimura. Keerthiram Murugesan, Ramón Fernandez Astudillo, Tahira Naseem, Pavan Kapanipathi, and Alexander G. Gray. 2022. X-factor: A cross-metric evaluation of factual correctness in ab- stractive summarization. In Conference on Empirical Methods in Natural Language Processing.
Eric Chu and Peter Liu. 2019. Meansum: A neural model for unsupervised multi-document abstractive summarization. In International Conference on Ma- chine Learning, pages 1223-1232. PMLR.
Eric Chu and Peter J. Liu. 2018. Meansum: A neural model for unsupervised multi-document abstractive summarization. In International Conference on Ma- chine Learning.
References
Reinald Kim Amplayo, Stefanos Angelidis, and Mirella Lapata. 2021. Aspect-controllable opinion summa- rization. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 6578-6593, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Mingkai Deng, Bowen Tan, Zhengzhong Liu, Eric P. Xing. and Zhiting Hu. 2021. Compression, transduc- tion, and creation: A unified framework for evaluat- ing natural language generation. In Conference on Empirical Methods in Natural Language Processing.
Reinald Kim Amplayo and Mirella Lapata. 2020. Un- supervised opinion summarization with noising and denoising. Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 1934 1945, Online. Association for Computa- tional Linguistics.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Stefanos Angelidis, Reinald Kim Amplayo, Yoshihiko Suhara, Xiaolan Wang, and Mirella Lapata. 2021. Extractive opinion summarization in quantized trans- former spaces. Transactions of the Association for Computational Linguistics, 9:277-293.
Giuseppe Di Fabbrizio, Amanda Stent, and Robert Gaizauskas. 2014. hybrid approach to multi- document summarization of opinions in reviews. In Proceedings of the 8th International Natural Lan- guage Generation Conference (INLG), pages 54-63.
Stefanos Angelidis and Mirella Lapata. 2018. Sum- marizing opinions: Aspect extraction meets senti- ment prediction and they are both weakly supervised. In Proceedings the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 3675-3684 Brussels, Belgium. Association for Com- putational Linguistics.
Bogdan Dumitrescu and Paul Trofti. 2018. Dictionary learning algorithms and applications. Springer.
Hady Elsahar, Maximin Coavoux, Jos Rozen, and Matthias Gallé. 2021. Self-supervised and controlled multi-documaen opinion summarization. ings of the 16th Conference of the European Chap- ter of the Association for Computational Linguistics: Main Volume, pages 1646-1661 Online. Association for Computational Linguistics.
Somnath Basu Roy Chowdhury, Chao Zhao, and Snigdha Chaturvedi. 2022. Unsupervised extractive opinion summarization using sparse coding. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume I: Long
Proceed-

---

Günes Erkan and Dragomir R. Radev. 2004. Lexrank: Graph- based lexical centrality as salience in text sum- marization. J. Artif. Intell. Res., 22:457-479.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad. Abdel rahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising weqeeeceeeeeeeeeeeeee
pre-training for nat- ural language generation, translation, and compre- hension. In Annual Meeting of the Association for Computational Linguistics.
requence-to-sequene
Kavita Ganesan, ChengXiang Zhai, and Jiawei Han. 2010. Opinosis: A graph based approach to abstrac- tive summarization of highly redundant opinions. In Proceedings of the 23rd international conference on computational linguistics (Coling 2010), pages 340-
Chin-Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Sunmariza- fion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguisties.
348.
Ruining He and Julian McAuley. 2016. Ups and downs:
Modeling the visual evolution of fashion trends with one-class collaborative filtering. Proceedings of the 25th International Conference on World Wide Web.
Jordan Louviere, Terry N. Flynn, and Anthony A. J. Marley. 2015. Best-worst scaling: Theory, methods and applications.
Minqing Hu and Bing Liu. 2006. Opinion extraction and summarization the web. In Aaai, volume 7, pages 1621-1624.
Rajdeep Mukherjee, Hari Chandana Peruri, Uppada Vishnu, Pawan Goyal, Sourangshu Bhattacharya, and Niloy Ganguly. 2020. Read what you need: Control- lable aspect-based opinion summarization of tourist reviews. In Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval, pages 1825-1828.
Jinbae Im, Moonki Kim, Hoyeop Lee, Hyunsouk Cho, and Sehee Chung. 2021. Self-supervised multimodal opinion summarization. ArXiv, abs/2105.
Hayate Iso. Xiaolan Wang, Yoshihiko Suhara. Stefanos Angelidis, and Wang-Chiew Tan. 2021. Convex Ag- gregation for Opinion Summarization. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3885-3903, Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese nert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguistics.
Wenjun Ke. Jinhua Gao, Huawei Shen, and Xueqi Cheng. 2022. Consistsum: Unsupervised opinion summarization with the consistency of aspect, sen- timent and semantic. Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining.
Leonardo F. R. Ribeiro, Mengwen Liu, Iryna Gurevych, Markus Dreyer, and Mohit Bansal. 2022. Factgraph: Evaluating factuality in summarization with semantic graph representations. ArXiv, abs/2204.06508..
Kevin Scaria, Himanshu Gupta, Saurabh Arjun Sawant, Swaroop Mishra, and Chitta Baral. 2023. In- structabsa: Instruction learning for aspect based sen- timent analysis. ArXiv, abs/2302.08824..
Jim Keeler and David Rumelhart. 1991. A self- organizing integrated segmentation and recognition neural net. Advances in neural information process- ing systems, 4.
Ming Shen, Jie Ma, Shuai Wang, Yogarshi Vyas, Kalpit Dixit, Miguel Ballesteros, and Yassine Benajiba. 2023. Simple yet effective synthetic dataset construc- tion for unsupervised opinion summarization. In Findings of the Association for Computational Lin- guistics: EACL 2023, pages 1898-1911, Dubrovnik, Croatia. Association for Computational Linguistics.
Hyun Duk Kim, Kavita Ganesan, Parikshit Sondhi, and ChangXiang Zhai. 201 Comprehensive review of opinion summarization.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. CoRR, abs/1412.6980.
Tejpalsingh Siledar, Jigar Makwana, and Pushpak Bhat- tacharyya. 2023. sonnect-eetimment-baseeee opinion summarization using multiple information sources. Proceedings of the 6th Joint International Confer- ence on Data Science & Management of Data (10th ACM IKDD CODS and 28th COMAD).
Diederik P Kingma and Max Welling. 2013. Auto- encoding variational bayes. arXiv preprint
131.6:1312.6114.
Wojciech Kryscinski. Bryan McCann, Caiming Xiong. and Richard Socher. 2019. Evaluating the factual consistency of abstractive text summarization. ArXiv, abs/1910.12840.
Aaron Van Den Oord, Oriol Vinyals, et al. 2017. Neural discrete representation learning. Advances in neural information processing systems, 30.
Philippe Laban. Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2021. Summac: Re-visiting nli-
based models for inconsistency detection in summa- rization. Transactions of the Association for Compu- lational Linguistics, 10:165 163-177.
Ke Wang and Xiaojun Wan. 2021. Transsum: Trans- lating aspect and sentiment embeddings for self- supervised opinion summarization. In Findings.

---

Lu Wang and Wang Ling. 2016. Neural network-based abstract generation for opinions and arguments. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- lional Linguistics: Human Language Technologies, pages 47-57, San Diego, California. Association for Computational Linguistics.
to extract aspects and manually create mapping between fine-grained and coarse-grained aspects using Integer Linear Programming (ILP) based ex- tractive subset of opinions. Our work is closest to (Shen et al., 2023) in that we also build syn- thetic datasets without the use of complex MIL models. However, synthetic datasets as our approach does not rely on any buman-specifin aspects or seed words. We use off the-thhehhfff aspect extraction and clustering techniques to build an automatic mapping of as- pects. Moreover, our approach uses two metrics: cosine similarity and rouge scores to form synthetic datasets which achieves better performance. Our approach of aspect mapping is similar to Mukherjee et al. (2020) where we create mapping between fine-grained and coarse grarsee aspects. However, our approach is fully automatic and does not need any human intervention to perform the mapping.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Transformers: State-of-the- art natural language processing. In Conference on Empirical Methods in Natural Language Processing.
different in generating
A Related Work
Earlier research in the field of opinion summariza- tion has focused on extractive approaches (Hu and Liu, 2006; Kim al., 201 I; Angelidis and Lapata, 2018) as well as abstractive approaches (Ganesan Carenini et al., 2013; Di Fabbrizio et al., 2014). The absence of large amounts of annotated opinion summarization datasets, recent works consider the unsupervised setting for opinion summarization where only review corpus avail- able without the corresponding summary (Chu and Liu, 2019; Bražinskas et al., 2020).
et al., 2010;
A.2 General Opinion Summarization
Chu and Liu (2019); Bražinskas et al. (2020) use autoencoders (Kingma and Welling, 2013) and its variants to learn a review decoder through recon- struction which is then used to generate summaries using the averaged representations of input reviews. Another approach is to curate synthetic datasets using one of the reviews as pseudo-summan and pair with input reviews using different strategies. Bražinskas et al. (2020) uses random sampling. Amplayo and Lapata (2020) generates noisy ver- sion of the pseudo-summarry Elsahar et al. (2021) ranks reviews using similarity and relevance, and Amplayo and Lapata (2020) uses content plans to generate synthetic datasets. Im al. (2021) ran- domly selects a review as aseudo-summant and proposes a pipeline to generate summaries using multimodal input sch text, image, and meta-data. Ke et al. (2022) captures the consistency of as- pects and sentiment between reviews and summary, whereas Wang and Wan (2021) learns aspect and sentiment embeddings to generate relevant pairs. Iso et al. (2021) searches for convex combinations of latent vectors to generate summaries. Our ap- proach uses cosine similarity and rouge scores be- tween reviews to filter highly relevant synthetic pairs that enable models to generate summaries more faithful to input reviews compared altemna tive approaches.
A.1 Aspect-specific Opinion Summarization Angelidis et al. (2021) proposed the first approach to generate both aspect-speeific and general sum- maries. They utilize Vector Quantized Variational Autoencoder (Van Den Oord al., 2017) for clus- tering review sentences followed by a popularity- driven extraction algorithm to summarize. (Basu Roy Chowdhury et al., 2022) utilizes dictionary learning (Dumitrescu and Irofti, 2018) to acquire representations of texts based on latent seman- tie units. Amplayo al. (2021) proposed the first abstractive approach for generating aspect- specific and general summaries. They generate synthetic datasets by identifying aspect-bearing el- ements (words, phrases, sentences) using a mul- tiple instance learning (MIL) (Keeler and Rumel- hart, 1991) model trained silver-labeled data obtained through seed words. Shen et al. (2023) proposes two simple solutions for generating syn- thetic datasets that do not rely complex MIL modules. The SW-LOO simply matches the aspect seed words to construct synthetic datasets, whereas NLI-LOO uses an off-the-shelf NLI model to do so using only aspects and no seed words. Mukher- jee et al. (2020) takes an unsupervised approach

---

"Three of us stayed at the Hotel Nevena and shared room.)
forming clusters two levels: fine-grained (us- ing a low threshold) and coarse-grained (using a high threshold). For instance, fine-grained aspects like camera, selfie camera, camera quality, back camera should all be mapped to the coarse-grae saurse-rrined aspect camera by associating them with the cluster that contains camera. The aspect mapping in Ta- ble demonstrates the final mapping for a sample product. Now, we can utilize this automatically generated mapping to filter review sentences and create synthetic datasets.
had two twins pushed tegether
seporate cot-lixe bed."
"We found
room to more than large enough and very comfortable
"I slept on the cot-lise bed and slept very well", "Showar worked well."
"Our room was on courlyard with large, heavy ook door." "You connot beat the atmosphere and location.,,
very helpful and friendly:"
"Lacation, Location, Location!"
Figure 1: Sample reviews for a hotel domain.
B Generalizability
this work, we focus on only e-commerce datasets and show results on them. However, our ap- proaches are generalizable and could be used for any other domain. Because our approaches do not rely on human- -peean-pe aspects or seed words, it be extended to any domain. Figure shows example of reviews for a hotel. Using our approach, we will be able to identify aspects such as room, atmosphere, location, staff. ... and create synthetic pairs corresponding from the training data.
E
SDC for Aspect-specific Opinion Summarization
To begin, we identify the sentence-level aspects for each product using aspect mappings. We then eliminate sentences that do not contain any aspects. Within review, if there are sentences that share common aspects, merge them together and refer to them as portions. The objective is determine if a specific portion is of sufficient quality to be con- sidered pseudo- summary.. We combine all remain- ing portions that share the same aspects with the current portion under consideration and calculate cosine similarity and rouge scores between them. This process is repeated for all portions across all reviews associated with particular product.
C Opinion Summarization Model
For general opinion summarization, only concate- nated reviews are provided input in the following format:

 review,
review2


For sspect-specific opinion summarization, we transform the aspects and reviews into the follow- ing format:
F
Why our models perform better?
General Opinion Summarization opinion summarization, synthetic dataset curation is im- portant due the absence of any large-scale pervised datasets (reviews-summary pairs). Such synthetic datasets enable supervised training for models. In Appendix A, discussed how existing approaches create synthetic datasets followed by the generation of opinion summaries. We, however, hypothesize that set of reviews and pseudo- summary can act as potential pair only the pseudo-summary is lexically (RI) and semantically (cosine similarity) similar to the review set. We use the matrix construction approach find such pairs which ensures that only highly relevant pairs selected in synthetic dataset. The simple intuition behind this that during training we want to show the model pseudo-summary that is most relevant to the input reviews. This will enable the model to learn the task of opinion summarization much more accurately. This is evidenced by results

 aspect,, aspect,, 
 


 review; review2
where 
 and 
 are special tokens for BART and aspects are comma-
Aspect Mapping
The primary idea behind this approach is gener- ate something similar to human-speeified aspects and seed words. As illustration, let's consider the laptop bag product from the Oposum+ dataset as shown in Table 3, where the aspect looks is asso- ciated with seed words such as looks, color, stylish, looked, and pretry. Sentences from reviews are fil- tered for the looks aspect if any of these seed words appear in them. We establish similar mappings by employing aspect extraction and clustering tech- niques as shown in Table 3. Our approach involves

---

in Table 5 as well as faithfulness scores in Tables 7 and the paper.
summary consists of multiple aspect-specific summaries. There are around 676 aspect- specific summaries in total. The original test set contains around 1000 reviews per prod- uct on average. We downsample this to 10 reviews per product to compare different mod- els. We first remove all the reviews with less than 20 and more than 100 words. For filter- ing out 10 reviews we use simple approach of first checking if the reviews contain the peets for which summaries need to be created. After the filtering step, we randomly selected 10 reviews to form input for our test set.
Appect- Opinion Summarization Exist- ing approaches are dependent on human-specified (pre-specified) aspects and seed words for creating synthetic datasets during training and filtering sen- tences from reviews before generating summaries during inferencing. Our approach, however, crc- ates aspect mappings automatically which enables the creation of synthetic datasets for wide range of aspects. Here as well we use the combina- tion of lexical and semantic scores for selecting a pseudo-summary (set of sentences about as- pect) and reviews pair. This ensures that the most relevant review-pseudo summary pair is shown to the model during training enabling better learning for the model to generate summaries. Our synthetic datasets are not restricted by the human-specified aspects and seed words which makes the model more robust generating summaries for any rel- evant aspect of a specific product. This we intuit is causing improved scores of our models over ex- isting models even though existing models have this extra aid of human- suman--peccffed aspects and seed words for training and inference.
H Baseline Details
General Opinion Summarization Clustroid is simple extractive model that selects the view with the highest RL score with respect to other reviews, For abstractive approaches, we use AceSun which generates synthetic datasets with the help of a multiple-instance learning model. SW-LOO and NLI-LOO use the leave-one-out strategy construct syn- thetic datasets. PlanSum used content plans to generate synthetic datasets by extracting a representative review the pseudo sum- mary. ConsistSum uses aspect and sentiment distribution to generate review-summary pairs. Multimoda/Stun proposes pipeline to gener- ate summaries using multimodal data such text, images, and meta-data. TransSum uses aspect and sentiment embeddings to construct synthetic datasets to train supervised opin- ion summarization model. COOP searches for convex combinations of latent vectors to generate summaries.
G Dataset Details
Oposum+ Oposum+ contains product reviews from six different domains: laptop bags, blue- tooth headsets, boots, keyboards, televisions from the Amazon Product Dataset (He and McAuley, 2016). Evaluation set contains four summaries per product: three aspect-specific abstractive summaries and one general extrac- tive summary. Each product has 10 reviews in the evaluation set. The training set contains 4.1 reviews over 95K products.
Aspect-specific Opinion Summarization LexRank selects the most salient sentences using BERT (Devlin al., 2019) encodings to represent sentences. QT is neural clustering method that represents opinions in quantized space. AceSument the extractive version that uses sentences ranked by their con- troller induction model as input to LexRank. Similarly, SW-LO0 ext and NLI-LOO ext also input aspect-related sentences to LexRank. Abstractive approaches such as MeanSum generate summaries by reconstructing the mean of the encoded reviews. CopyCat is
Amazon Amazon dataset contains product reviews from four domains: electronics, home and kitchen, personal care, and clothing, shoes and jewelry. The evaluation set contains three general abstractive summaries per product. Each product has reviews the evaluation set. The training set contains ~ 1M reviews over 90K products.
Flipkart Flipkart dataset contains product reviews from three domains: laptops, mobiles, and tablets. The test set contains around 147 prod- ucts with one summary per product. Each

---

hierarchical variational autoencoder that learns a latent code of the summary. ASBOS uses hybrid approach of filtering review sentences the basis of aspects and then summarizing them.
evance. A higher CTC score indicates higher faithfulness.
FactCC: FactCC is BERT-based classification model, with the objective of ascertaining the consistency or inconsistency between a pro- vided text or summary and its corresponding source article. A higher FactCC score indi- cates higher faithfulness.
I Training Details
Table 6, results for the Flipkart test set are ob- tained from models trained using the Oposum+ training data due the unavailability of any train- ing data. In Table and 6, results that couldn't be produced due to the unavailability of public code certain issues have been marked with x.
Frathraph: FactGraph uses both the text and their structured meaning representations computed using graph encoder with structure-aware adapters to enhance the factuality of the sum- maries with respect to the source document. A higher FactGraph score indicates higher faith- fulness.
J Implementation Details
L Best Worst Scaling
We use BART implementation from HuggingFace (Wolf et al., 2019). We use Adam (Kingma and Ba, 2015) optimizer with eps of le - and linear weight decay to optimize our models. We use [le 6, 2e 6, le 5, 2e 5] and batch size [8, 16] as our hyperparameters. We observe that a learning rate of 2e- 6 and batch size of 8 performs the best, All experiments were performed using NVIDIA A100-SXM4- 80GB GPUs. Refer Section 3.3.
We evaluated our model-generated general opinion summaries on four different criteria: faithfulness (how consistent are the opinions compared to refer- ence summaries?), coherence (is the summary well organized and casy to read?), conciseness (is the summary concise yet informative?), and fluency (is the summary fluent and grammatical?) (Am- playo et al., 2021). Each participant's choice of the best model earned score of +1, while the worst model received score of -1. The remain- ing models were assigned score of O. To calcu- late the final scores, we averaged the scores from all participants' responses. Table 9 indicates that our model gives the best scores all the crite- ria. Specifically our model scores around 0.1 point higher than the second-best model indicating that our model--ennraate general opinion summaries are more faithful than alternatives.
Faithfulness Metries Details
We measure the faithfulness of summaries using four faithfulness measuring metrics: SummaC, CTC, FactCC, and FactGraph. We chose these metrics for evaluation as they correlate better with human judgments (Chaudhury et al., 2022).
SummaC: SummaC (Summary Consistency) aims to tackle the granularity in NLI models. Specifically, SummaC focuses on identifying inconsistencies in summarization, taking into account the diverse levels of granularity that can exist between sentences and documents A higher SummaC score indicates higher faith- fulness.
M Example Summaries
Table 10 shows sample model-generated sum- maries for both general opinion summarization and aspect-specific opinion summarization. For gen- eral opinion summarization, we show three human- written summarics followed by model-genereated summaries compared to basclines for product from the Amazon test set. For aspect-specific opin- ion summarization, we show human-written sum- maries followed by moderl-gnnrrttdd summaries compared baseline different aspects for product from the Oposum+ test set.
CTC: CTC (Compression Transduction Creation) presents framework that considers various natural language generation (NLG) tasks, in- cluding compression (such summarization), transduction (like text rewriting), and creation (such as dialog generation). The CTC metric evaluates information alignment, with spe- cific emphasis on gauging consistency and rel-

---

General Opinion Summarization
Human written summaries
These look very nice on. The mesh in them run little tight though so you may want to go in size. They are roomy in the pockets and can hold onto lot of items. They are great price for the quality and would highly recommend them to anyone looking for new pair of swim trunks.
These swim trunks look great and the waist fits as expected for the size. They are also a good deal for the price. However, the inner mesh liner much smaller than the size of the outer trunks. You will probably have to out the mesh liner wear them comfortably.
These nice fitting shorts but the mesh liner felt tight and cut it out. You might want to size up if you unsure. The deep pockets come in handy. These shorts stylish and they look good. They are more comfortable the inner mesh liner removed.
PlanSum
bought this for my husband and she loves it. il is very good product and the fit good. the only reason didn't give it 5 stars because it does fit the middle of the back of my shirt. would recommend this anyone who looking for a good quality product.
MultimodalSum
bought these for my husband wear the beach. They fit well and look good. The only thing don't like is that the mesh liner is a little tight. He has to pull up every time he goes and of the water.
Our Model
These shorts are great. They fit well and look great. The only thing would change is the liner. is little small and had to Other than that, would recommend these shorts.
Aspect-specific Opinion Summarization
Human written summaries
Ports in some cases the output headphones and the hdmi ports have stopped working, however has enough inputs and for the price it is pretty solid option.
Picture the quality is good and the resolution is decent. sharpness generally good althouug blacks are somewhat inconsistent in the imagee.
Sound the sound quality is acceptable for the price the equipment, the sound can descrihed above average may not need sound bar.
AceSum
Ports the hdmi ports stopped working and the tv would go black for about three to five seconds every few minutes. has all of the inputs for us, and more.
Picture the picture quality was better than my 5 year old led samsung. the picture is decent, but not as good as the hipe. Sound i bought the tv fry's after salesperson assured me headphones would work with this. picture is acceptable and sound above average for the price.
Our Model
Ports i have had this tv for few months now and i have had issues with it, it has all of the hdmi ports i need and the picture quality great.
Picture the picture is great, the tv is easy to set up, and all the inputs need, is not a smart tv.
Sound the sound is decent, but i have a surround sound system so don't use the tv's speakers much, but great tv for the price.
Table 10: General Opinion Summarization: Three human- human-written summaries followed by summaries from different models for a product from the Amazon test for general opinion summarization. Aspect-specitic Opinion Summarization: Human written summaries followed by summaries for product aspects: Ports, Picture, and Sound from the Oposum+ test set for aspect- sspecffc opinion summarization.


Reviews:
[{"review_id": "dJtaD5RUsJ", "reviewer": "Reviewer_UMd3", "summary": "", "questions": "1. Clustering Process:\nThe paper lacks specific details regarding the clustering algorithm employed for both fine-grained and coarse clusters. Including a description of the clustering algorithms utilized would enhance the readers' understanding of the methods employed to group data points into meaningful clusters.\n2. Intersection of Coarse-Grained and Fine-Grained Aspects:\nThe paper briefly mentions an intersection between coarse-grained and fine-grained aspects at line 110. However, it does not elaborate on how this intersection is measured. Clarifying whether it is based on word overlap or some other metric would provide clarity and reinforce the validity of the findings.\n3. Purpose of Paragraph at Line 117:\nThe purpose of creating these triplets in the paragraph starting at line 117 requires further explanation. Providing an introduction or overview of the intention behind this approach before delving into the technical details would be beneficial, as it would set the context and enable readers to grasp the significance of the triplet creation process more effectively.\n4. Definition of the term D:\nAt line 138, the paper references the term \"D\" without providing a clear definition. To ensure clarity and avoid confusion, it is crucial to explicitly define \"D\" within the paper, preferably in the section where it is first introduced. This would enable readers to comprehend the role and significance of \"D\" in the context of the study.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable suggestions and feedback. We have tried our best to resolve all the doubts and provide clarifications. We request the reviewer to kindly consider increasing the scores positively if they find the rebuttal satisfactory.\n\n### A. Additional Results\n\n**1. General Opinion Summarization:** Due to certain issues with our code, our results in the paper are with beam size 1. We re-ran our updated code with different beam sizes and observed the best performance using beam size 5 (decided using the dev set). Currently, we achieve the best R2 and RL score and second-best R1 compared to baselines. \n\n| *Amazon* | R1 | R2 | RL |\n| ------------- | ---- | ---- | ---- |\n| Multimodalsum | 34.19 | 7.05 | 20.81 |\n| ConsistSum | 33.32 | 5.94 | _21.41_ |\n| TransSum | 34.23 | _7.24_ | 20.49 |\n| COOP [1] | **36.57** | 7.23 | 21.24 |\n| Ours | _35.46*_ | **7.30** | **21.50** |\n\n**Rebuttal Table 1:** **Bold** and _italics_ indicate best and second-best results. *indicates results that are significantly better than MultimodalSum with a pvalue<0.05 on paired t-test computed over the Amazon test set for general opinion summarization (couldn't be compared with others due to unavailability of code/results).\n\n\n**2. Aspect-specific Opinion Summarization:** Due to certain issues with our code, our results in the paper are with beam size 1. We re-ran our updated code with different beam sizes and observed the best performance using beam size 2 (decided using the dev set). Currently, we achieve the best R1, R2, and RL scores compared to baselines. \n\n| *Oposum+* | R1 | R2 | RL |\n| ------------- | ---- | ---- | ---- |\n| Acesum | 29.53 | 6.79 | _21.06_ |\n| SW-LOO | _30.00_ | **6.92** | 20.76 |\n| NLI-LOO | 28.90 | 6.60 | 20.11 |\n| Ours | **30.95*** | **6.92** | **21.73** |\n\n**Rebuttal Table 2:** **Bold** and _italics_ indicate best and second-best results. *indicates results that are significantly better than AceSum with a pvalue<0.05 on paired t-test computed over the Oposum+ test set for aspect-specific opinion summarization (couldn't be compared with others due to unavailability of code/results).\n\n\n| Acesum |  |\n| --------- | ---------------- |\n| Ports | the hdmi ports stopped working and the tv would go black for about three to five seconds every few minutes. it has all of the inputs for us, and more. |\n| Picture | the picture quality was better than my 5 year old lcd samsung. the picture is decent, but not as good as the htpc.|\n| Sound | i bought the tv at fry's after salesperson assured me headphones would work with this. picture is acceptable and sound is above average for the price. |\n| **Ours** |  |\n| Ports | i have had this tv for a few months now and i have had no issues with it, it has all of the hdmi ports i need and the picture quality is great. |\n| Picture | the picture is great, the tv is easy to set up, and it has all of the inputs i need, but it is not a smart tv. |\n| Sound | the sound is decent, but i have a surround sound system so i don't use the tv's speakers much, but it is a great tv for the price.|\n\n**Rebuttal Table 3:** Comparative analysis of our model-generated summaries with Acesum for aspect-specific opinion summarization on different aspects such as _ports, picture,_ and _sound_. Our model does not rely on any human-specified aspects and seed words for training, making our approach generalizable to other domains where such aspects and seed words are not defined.\n\n**3. Faithfulness Evaluation:**  We additionally show results of general opinion summarization on FactGraph [3] which is a metric that correlates the best with human judgment on faithfulness evaluation [2]. \n\n|  | FactGraph | \n| ------------- | ---- | \n| Plansum | 0.16 | \n| Multimodalsum | 0.53 | \n| Ours | **0.66** | \n\n**Rebuttal Table 4:** **Bold** indicate the best result. We achieved the highest score on FactGraph for faithfulness evaluation for general opinion summarization.\n\n| Models | Summaries |\n| --------- | ---------------- |\n| Multimodalsum | This is a good receiver for the price. I have it **hooked up to a pair of Sony speakers** and it sounds great. The only problem I have with it is that the **remote control is a little difficult to use**. I would recommend this to a friend. |\n| Ours | This is a great stereo for the price. The only thing I don't like about it is the fact that you can't adjust the volume on the radio. I have to turn it up all the way to hear it. Other than that, it's a great product. |\n| Multimodalsum | This is a nice teapot, but the color is not as bright as the picture. It is more of a dark turquoise than a light blue. **I was hoping it would be more of an aqua blue, but it is more like a dark aqua**. It still looks nice, but I would have preferred the color to be more like the photo. |\n| Ours | This is a beautiful teapot, but the color is not the same as shown in the picture. It is more of a dark turquoise. The color is a little darker than the picture, but it is still a beautiful color. The only thing I don't like about it is that the handle is very hard to open.|\n\n**Rebuttal Table 5:** Example of faithfulness issue in baseline models compared to ours. There is no support for the information in **bold** parts in the corresponding input.\n\n\n### B. Reason why our models are performing better than baselines\n**1. General Opinion Summarization:**\n\nIn opinion summarization, synthetic dataset curation is important due to the absence of any large-scale supervised datasets (reviews-summary pairs). Such synthetic datasets enable supervised training for models. \n\nIn Appendix A, we discussed how existing approaches create synthetic datasets followed by the generation of opinion summaries. We, however, hypothesize that a set of reviews and a pseudo-summary can act as a potential pair only if the pseudo-summary is lexically (rouge 1) and semantically (cosine similarity) similar to the review set. We use the matrix construction approach to find such pairs which ensures that only highly relevant pairs are selected in our synthetic dataset. The simple intuition behind this is that during training we want to show the model a pseudo-summary that is most relevant to the input reviews. This will enable the model to learn the task of opinion summarization much more accurately. This is evidenced by results in Table 4 as well as faithfulness scores in Tables 5 and 6 in the paper.\n\n**2. Aspect-specific Opinion Summarization:**\n\nExisting approaches are dependent on human-specified (pre-specified) and seed words for creating synthetic datasets for training and filtering sentences from reviews before generating summaries for inferencing.\n\nOur approach, however, creates aspect mappings automatically which enables the creation of synthetic datasets for a wide range of aspects. Here as well we use the combination of lexical and semantic scores for selecting a pseudo-summary (set of sentences about an aspect) and reviews pair. This ensures that the most relevant review-pseudo summary pair is shown to the model during training enabling better learning for the model to generate summaries. Our synthetic datasets are not restricted by the human-specified aspects and seed words which makes the model more robust in generating summaries for any relevant aspect of a specific product. This we intuit is causing improved scores of our models over existing models even though existing models have this extra aid of human-specified aspects and seed words for training and inference.\n\n### C. Clustering Process\n\nWe used the fast clustering library provided by SBERT [4]. We did not find the specific details of the algorithm. Specific details regarding its implementation will be added to the Appendix.\n\n### D. Clarification on coarse-grained and fine-grained aspects\nThe intersection is on the aspect level. We will add this specific detail in the paper.\n\n\n### E. Comment on the suggestions\n1. Purpose of Paragraph at Line 117: Thanks for the suggestion. Will add appropriate reasoning at the start of the section.\n\n2. Definition of the term D: We apologize for this mistake and will update it in the paper. We had made some last-minute section shifting which caused this lapse.\n\n3. Presentation improvements: Noted. Will be updated in the paper.\n\n### F. References\n[1] [Convex Aggregation for Opinion Summarization](https://aclanthology.org/2021.findings-emnlp.328/) (Iso et al., Findings 2021)\n\n[2] [X-FACTOR: A Cross-metric Evaluation of Factual Correctness in Abstractive Summarization](https://aclanthology.org/2022.emnlp-main.478/) (Chaudhury et al., EMNLP 2022)\n\n[3] [FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations](https://aclanthology.org/2022.naacl-main.236/) (Ribeiro et al., NAACL 2022)\n\n[4] [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://aclanthology.org/D19-1410/) (Reimers & Gurevych, EMNLP-IJCNLP 2019)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "In this research endeavor, the authors harness the power of recent advances in machine learning models to generate aspects. They showcase their ingenuity by developing clever methodologies for creating pseudo summaries. By leveraging cutting-edge techniques, they aim to improve the quality and accuracy of aspects and summaries in their work.", "reasons_to_reject": "Analysis of results - The results section provides a verbose representation of the result tables. However, it lacks an in-depth analysis of why certain results are as they are. To enhance the section, it would be beneficial to include a detailed analysis explaining the underlying reasons behind the observed outcomes. This would provide valuable insights and a better understanding of the significance and implications of the results obtained.\n\nComparisons to baselines that uses human specified aspects? - In comparison to a baseline that relies on human-specified aspects (AceSum), the authors have not included any comparative analysis in their paper with respect to their closest competitor model concerning the number of added data points. The paper lacks a discussion on the usefulness of automatically generating these aspects using InstructABSA. Consequently, certain pertinent questions remain unanswered, potentially limiting a comprehensive understanding of the research findings. A more thorough investigation into these aspects would significantly strengthen the paper's contributions and provide valuable insights for the readers.", "questions_to_authors": "1. Clustering Process:\nThe paper lacks specific details regarding the clustering algorithm employed for both fine-grained and coarse clusters. Including a description of the clustering algorithms utilized would enhance the readers' understanding of the methods employed to group data points into meaningful clusters.\n2. Intersection of Coarse-Grained and Fine-Grained Aspects:\nThe paper briefly mentions an intersection between coarse-grained and fine-grained aspects at line 110. However, it does not elaborate on how this intersection is measured. Clarifying whether it is based on word overlap or some other metric would provide clarity and reinforce the validity of the findings.\n3. Purpose of Paragraph at Line 117:\nThe purpose of creating these triplets in the paragraph starting at line 117 requires further explanation. Providing an introduction or overview of the intention behind this approach before delving into the technical details would be beneficial, as it would set the context and enable readers to grasp the significance of the triplet creation process more effectively.\n4. Definition of the term D:\nAt line 138, the paper references the term \"D\" without providing a clear definition. To ensure clarity and avoid confusion, it is crucial to explicitly define \"D\" within the paper, preferably in the section where it is first introduced. This would enable readers to comprehend the role and significance of \"D\" in the context of the study.", "paper_topic_and_main_contributions": "In this study, the researchers address the limitations of aspect-specific summarization, which currently relies heavily on human-specified aspects and seed words. Additionally, general summarization struggles to accurately capture the essence of input reviews. To overcome these challenges, the authors propose innovative techniques for generating synthetic datasets specifically tailored for both aspect-specific and general summarization. By utilizing cosine similarity and ROUGE-1 scores for pseudo-summary selection, they achieve notable improvements of 0.02 and 0.2 points on the SummaC and FactCC benchmarks, respectively. This approach represents a  step forward compared to previous methods that heavily depended on human-specified aspects and aspect-specific seed words for generating data for aspect-specific summarization.", "excitement": 3, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 35 can contain 2-3 recent papers rather than a list of references."}, "extra_text": {}}, {"review_id": "5O5yWQSRwA", "reviewer": "Reviewer_7Tmj", "summary": "", "questions": "- Is there a specific reason for focusing on the e-commerce domain? Other well-studied opinions summarization benchmarks, such as Yelp [4] and Space [5], could also be considered. Could you clarify the reason for this focus?\n- The synthetic dataset creation method bears similarities to an existing study [6]. While differences exist, it would be nice to acknowledge their work.\n\n* [4] [MeanSum: A Neural Model for Unsupervised Multi-document Abstractive Summarization](http://proceedings.mlr.press/v97/chu19b/chu19b.pdf) (Chu et al., ICML 2019)\n* [5] [Extractive Opinion Summarization in Quantized Transformer Spaces](https://aclanthology.org/2021.tacl-1.17/) (Angelidis et al., TACL 2021)\n* [6] [Self-Supervised and Controlled Multi-Document Opinion Summarization](https://aclanthology.org/2021.eacl-main.141) (Elsahar et al., EACL 2021)\n\n\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable suggestions and feedback. We have tried our best to resolve all the doubts and provide clarifications. We request the reviewer to kindly consider increasing the scores positively if they find the rebuttal satisfactory.\n\n### A. Additional Results\n\n**1. General Opinion Summarization:** Due to certain issues with our code, our results in the paper are with beam size 1. We re-ran our updated code with different beam sizes and observed the best performance using beam size 5 (decided using the dev set). Currently, we achieve the best R2 and RL score and second-best R1 compared to baselines. \n\n| *Amazon* | R1 | R2 | RL |\n| ------------- | ---- | ---- | ---- |\n| Multimodalsum | 34.19 | 7.05 | 20.81 |\n| ConsistSum | 33.32 | 5.94 | _21.41_ |\n| TransSum | 34.23 | _7.24_ | 20.49 |\n| COOP [1] | **36.57** | 7.23 | 21.24 |\n| Ours | _35.46*_ | **7.30** | **21.50** |\n\n**Rebuttal Table 1:** **Bold** and _italics_ indicate best and second-best results. *indicates results that are significantly better than MultimodalSum with a pvalue<0.05 on paired t-test computed over the Amazon test set for general opinion summarization (couldn't be compared with others due to unavailability of code/results).\n\n\n**2. Aspect-specific Opinion Summarization:** Due to certain issues with our code, our results in the paper are with beam size 1. We re-ran our updated code with different beam sizes and observed the best performance using beam size 2 (decided using the dev set). Currently, we achieve the best R1, R2, and RL scores compared to baselines. \n\n| *Oposum+* | R1 | R2 | RL |\n| ------------- | ---- | ---- | ---- |\n| Acesum | 29.53 | 6.79 | _21.06_ |\n| SW-LOO | _30.00_ | **6.92** | 20.76 |\n| NLI-LOO | 28.90 | 6.60 | 20.11 |\n| Ours | **30.95*** | **6.92** | **21.73** |\n\n**Rebuttal Table 2:** **Bold** and _italics_ indicate best and second-best results. *indicates results that are significantly better than AceSum with a pvalue<0.05 on paired t-test computed over the Oposum+ test set for aspect-specific opinion summarization (couldn't be compared with others due to unavailability of code/results).\n\n\n| Acesum |  |\n| --------- | ---------------- |\n| Ports | the hdmi ports stopped working and the tv would go black for about three to five seconds every few minutes. it has all of the inputs for us, and more. |\n| Picture | the picture quality was better than my 5 year old lcd samsung. the picture is decent, but not as good as the htpc.|\n| Sound | i bought the tv at fry's after salesperson assured me headphones would work with this. picture is acceptable and sound is above average for the price. |\n| **Ours** |  |\n| Ports | i have had this tv for a few months now and i have had no issues with it, it has all of the hdmi ports i need and the picture quality is great. |\n| Picture | the picture is great, the tv is easy to set up, and it has all of the inputs i need, but it is not a smart tv. |\n| Sound | the sound is decent, but i have a surround sound system so i don't use the tv's speakers much, but it is a great tv for the price.|\n\n**Rebuttal Table 3:** Comparative analysis of our model-generated summaries with Acesum for aspect-specific opinion summarization on different aspects such as _ports, picture,_ and _sound_. Our model does not rely on any human-specified aspects and seed words for training, making our approach generalizable to other domains where such aspects and seed words are not defined.\n\n**3. Faithfulness Evaluation:**  We additionally show results of general opinion summarization on FactGraph [3] which is a metric that correlates the best with human judgment on faithfulness evaluation [2]. \n\n|  | FactGraph | \n| ------------- | ---- | \n| Plansum | 0.16 | \n| Multimodalsum | 0.53 | \n| Ours | **0.66** | \n\n**Rebuttal Table 4:** **Bold** indicate the best result. We achieved the highest score on FactGraph for faithfulness evaluation for general opinion summarization.\n\n| Models | Summaries |\n| --------- | ---------------- |\n| Multimodalsum | This is a good receiver for the price. I have it **hooked up to a pair of Sony speakers** and it sounds great. The only problem I have with it is that the **remote control is a little difficult to use**. I would recommend this to a friend. |\n| Ours | This is a great stereo for the price. The only thing I don't like about it is the fact that you can't adjust the volume on the radio. I have to turn it up all the way to hear it. Other than that, it's a great product. |\n| Multimodalsum | This is a nice teapot, but the color is not as bright as the picture. It is more of a dark turquoise than a light blue. **I was hoping it would be more of an aqua blue, but it is more like a dark aqua**. It still looks nice, but I would have preferred the color to be more like the photo. |\n| Ours | This is a beautiful teapot, but the color is not the same as shown in the picture. It is more of a dark turquoise. The color is a little darker than the picture, but it is still a beautiful color. The only thing I don't like about it is that the handle is very hard to open.|\n\n**Rebuttal Table 5:** Example of faithfulness issue in baseline models compared to ours. There is no support for the information in **bold** parts in the corresponding input.\n\n### B. Clarification on hyper-parameters\nWe do agree that there are some extra hyper-parameters that would change according to the domain under consideration. However, having these hyper-parameters is essential for the proper selection of review-pseudo summary pairs to fine-tune models. \n\n### C. COOP baseline\nWe apologize for missing this out. Our current results in Rebuttal Table 1 do show that our model surpasses Coop on both R2 and RL. Thanks for pointing this out.\n\n### D. LLM discussion\nWe couldn't compare our models with [3], and [4] as they are very recent papers. As mentioned in the limitations section, we were investigating better synthetic dataset approaches and used BART to have a fair comparison with previous models. We do agree the applicability of these for LLMs needs to be investigated.\n\n### E. Reason for focus on e-commerce datasets\nThe main aim of the paper was to show the applicability of our approach not restricted to just one e-commerce dataset but others as well. We specifically chose e-commerce as there were multiple datasets available. This ensured that the better results were not observed only on a specific dataset but on the whole e-commerce domain. Appendix B does talk about the same approach being applicable to other domains but this needs to be demonstrated empirically.\n\n### E. Missing References\nThank you for pointing out the missing references. Will incorporate them in the paper.\n\n### F. References\n[1] [Convex Aggregation for Opinion Summarization](https://aclanthology.org/2021.findings-emnlp.328/) (Iso et al., Findings 2021)\n\n[2] [X-FACTOR: A Cross-metric Evaluation of Factual Correctness in Abstractive Summarization](https://aclanthology.org/2022.emnlp-main.478/) (Chaudhury et al., EMNLP 2022)\n\n[3] [FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations](https://aclanthology.org/2022.naacl-main.236/) (Ribeiro et al., NAACL 2022)\n\n[4] [Prompted Opinion Summarization with GPT-3.5](https://aclanthology.org/2023.findings-acl.591/) (Bhaskar et al., Findings 2023)\n\n[5] [Attributable and Scalable Opinion Summarization](https://aclanthology.org/2023.acl-long.473/) (Hosking et al., ACL 2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed method demonstrates improvements across two different tasks: aspect-based and general opinion summarizations, outperforming strong baselines.\n- The authors conducted a human evaluation using best-worst scaling, a credible approach, which showed that the proposed method performs better or comparably to the baselines.\n- Overall, the experiments look solid and easily reproducible.\n", "reasons_to_reject": "- The proposed method introduces too many hyper-parameters, potentially limiting its generalizability to other domains.\n- An important baseline, Coop [1], for unsupervised general opinion summarization, which exhibits better performance than the author's model in terms of ROUGE-1 & L on Amazon, is missing.\n- The discussion of Language Model-based methods (LLMs) is lacking. Given the availability of data for training, it would be relevant to consider studies on opinion summarization with LLMs [2], [3]. It would be beneficial to differentiate why synthetic data creation remains relevant in the post-ChatGPT era.\n\n* [1] [Convex Aggregation for Opinion Summarization](https://aclanthology.org/2021.findings-emnlp.328/) (Iso et al., Findings 2021) \n* [2] [Prompted Opinion Summarization with GPT-3.5](https://aclanthology.org/2023.findings-acl.591/) (Bhaskar et al., Findings 2023)\n* [3] [Attributable and Scalable Opinion Summarization](https://aclanthology.org/2023.acl-long.473/) (Hosking et al., ACL 2023)", "questions_to_authors": "- Is there a specific reason for focusing on the e-commerce domain? Other well-studied opinions summarization benchmarks, such as Yelp [4] and Space [5], could also be considered. Could you clarify the reason for this focus?\n- The synthetic dataset creation method bears similarities to an existing study [6]. While differences exist, it would be nice to acknowledge their work.\n\n* [4] [MeanSum: A Neural Model for Unsupervised Multi-document Abstractive Summarization](http://proceedings.mlr.press/v97/chu19b/chu19b.pdf) (Chu et al., ICML 2019)\n* [5] [Extractive Opinion Summarization in Quantized Transformer Spaces](https://aclanthology.org/2021.tacl-1.17/) (Angelidis et al., TACL 2021)\n* [6] [Self-Supervised and Controlled Multi-Document Opinion Summarization](https://aclanthology.org/2021.eacl-main.141) (Elsahar et al., EACL 2021)\n\n\n", "paper_topic_and_main_contributions": "This study focuses on building a synthetic training dataset for opinion summarization. Creating such a dataset is crucial to developing a high-quality opinion summarization system without relying on a supervised training dataset.\n\nThe main contribution is the proposal of methods not only for general opinion summarization but also for aspect-based opinion summarization.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The list of above:\n* [1] [Convex Aggregation for Opinion Summarization](https://aclanthology.org/2021.findings-emnlp.328/) (Iso et al., Findings 2021) \n* [2] [Prompted Opinion Summarization with GPT-3.5](https://aclanthology.org/2023.findings-acl.591/) (Bhaskar et al., Findings 2023)\n* [3] [Attributable and Scalable Opinion Summarization](https://aclanthology.org/2023.acl-long.473/) (Hosking et al., ACL 2023)\n* [4] [MeanSum: A Neural Model for Unsupervised Multi-document Abstractive Summarization](http://proceedings.mlr.press/v97/chu19b/chu19b.pdf) (Chu et al., ICML 2019)\n* [5] [Extractive Opinion Summarization in Quantized Transformer Spaces](https://aclanthology.org/2021.tacl-1.17/) (Angelidis et al., TACL 2021)\n* [6] [Self-Supervised and Controlled Multi-Document Opinion Summarization](https://aclanthology.org/2021.eacl-main.141) (Elsahar et al., EACL 2021)"}}, {"review_id": "JBk2uL9nDI", "reviewer": "Reviewer_GYEt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable suggestions and feedback. We have tried our best to resolve all the doubts and provide clarifications. We request the reviewer to kindly consider increasing the scores positively if they find the rebuttal satisfactory.\n\n### A. Additional Results\n\n**1. General Opinion Summarization:** Due to certain issues with our code, our results in the paper are with beam size 1. We re-ran our updated code with different beam sizes and observed the best performance using beam size 5 (decided using the dev set). Currently, we achieve the best R2 and RL score and second-best R1 compared to baselines. \n\n| *Amazon* | R1 | R2 | RL |\n| ------------- | ---- | ---- | ---- |\n| Multimodalsum | 34.19 | 7.05 | 20.81 |\n| ConsistSum | 33.32 | 5.94 | _21.41_ |\n| TransSum | 34.23 | _7.24_ | 20.49 |\n| COOP [1] | **36.57** | 7.23 | 21.24 |\n| Ours | _35.46*_ | **7.30** | **21.50** |\n\n**Rebuttal Table 1:** **Bold** and _italics_ indicate best and second-best results. *indicates results that are significantly better than MultimodalSum with a pvalue<0.05 on paired t-test computed over the Amazon test set for general opinion summarization (couldn't be compared with others due to unavailability of code/results).\n\n\n**2. Aspect-specific Opinion Summarization:** Due to certain issues with our code, our results in the paper are with beam size 1. We re-ran our updated code with different beam sizes and observed the best performance using beam size 2 (decided using the dev set). Currently, we achieve the best R1, R2, and RL scores compared to baselines. \n\n| *Oposum+* | R1 | R2 | RL |\n| ------------- | ---- | ---- | ---- |\n| Acesum | 29.53 | 6.79 | _21.06_ |\n| SW-LOO | _30.00_ | **6.92** | 20.76 |\n| NLI-LOO | 28.90 | 6.60 | 20.11 |\n| Ours | **30.95*** | **6.92** | **21.73** |\n\n**Rebuttal Table 2:** **Bold** and _italics_ indicate best and second-best results. *indicates results that are significantly better than AceSum with a pvalue<0.05 on paired t-test computed over the Oposum+ test set for aspect-specific opinion summarization (couldn't be compared with others due to unavailability of code/results).\n\n\n| Acesum |  |\n| --------- | ---------------- |\n| Ports | the hdmi ports stopped working and the tv would go black for about three to five seconds every few minutes. it has all of the inputs for us, and more. |\n| Picture | the picture quality was better than my 5 year old lcd samsung. the picture is decent, but not as good as the htpc.|\n| Sound | i bought the tv at fry's after salesperson assured me headphones would work with this. picture is acceptable and sound is above average for the price. |\n| **Ours** |  |\n| Ports | i have had this tv for a few months now and i have had no issues with it, it has all of the hdmi ports i need and the picture quality is great. |\n| Picture | the picture is great, the tv is easy to set up, and it has all of the inputs i need, but it is not a smart tv. |\n| Sound | the sound is decent, but i have a surround sound system so i don't use the tv's speakers much, but it is a great tv for the price.|\n\n**Rebuttal Table 3:** Comparative analysis of our model-generated summaries with Acesum for aspect-specific opinion summarization on different aspects such as _ports, picture,_ and _sound_. Our model does not rely on any human-specified aspects and seed words for training, making our approach generalizable to other domains where such aspects and seed words are not defined.\n\n**3. Faithfulness Evaluation:**  We additionally show results of general opinion summarization on FactGraph [6] which is a metric that correlates the best with human judgment on faithfulness evaluation [2]. \n\n|  | FactGraph | \n| ------------- | ---- | \n| Plansum | 0.16 | \n| Multimodalsum | 0.53 | \n| Ours | **0.66** | \n\n**Rebuttal Table 4:** **Bold** indicate the best result. We achieved the highest score on FactGraph for faithfulness evaluation for general opinion summarization.\n\n| Models | Summaries |\n| --------- | ---------------- |\n| Multimodalsum | This is a good receiver for the price. I have it **hooked up to a pair of Sony speakers** and it sounds great. The only problem I have with it is that the **remote control is a little difficult to use**. I would recommend this to a friend. |\n| Ours | This is a great stereo for the price. The only thing I don't like about it is the fact that you can't adjust the volume on the radio. I have to turn it up all the way to hear it. Other than that, it's a great product. |\n| Multimodalsum | This is a nice teapot, but the color is not as bright as the picture. It is more of a dark turquoise than a light blue. **I was hoping it would be more of an aqua blue, but it is more like a dark aqua**. It still looks nice, but I would have preferred the color to be more like the photo. |\n| Ours | This is a beautiful teapot, but the color is not the same as shown in the picture. It is more of a dark turquoise. The color is a little darker than the picture, but it is still a beautiful color. The only thing I don't like about it is that the handle is very hard to open.|\n\n**Rebuttal Table 5:** Example of faithfulness issue in baseline models compared to ours. There is no support for the information in **bold** parts in the corresponding input.\n\n### B. Clarifications on the naming convention for General, Aspect-specific, human-specified aspects, and seed words\nSorry for any confusion this might have caused, but we did use standard terminologies following existing literature ([3], [4]) for general and aspect-specific opinion summarization. Seed-words is also a standard term in aspect-specific opinion summarization ([3], [4]). Human-specified aspects are the aspects defined for a product category (eg. Bags) by human annotators (eg. looks, quality, size) [1]. We agree that this point should have been made clearer in the paper.\n\n### C. Clarifications on our approach applicable to other domains\n\nBy other domains, we specifically meant domains within opinion summarization such as hotels (example in Appendix B), movies, businesses, etc. which contain opinions in the form of reviews. We do agree that domains falling under text summarization such as “news” have different behaviors and the applicability of these approaches needs to be investigated separately. (Will update this clarification in the paper)\n\n### D. Confusion regarding Datasets\n\nAmazon and Oposum+ although based on Amazon data [5] are named as such in existing literature. Hence separate mentions for Oposum+ and Amazon to keep things clear. The main difference between these two datasets is the domain they cover, mentioned in Appendix G. Specifically, Oposum+ covers _laptop bags, bluetooth headsets, boots, keyboards, televisions_ whereas the Amazon dataset [5] covers _electronics, home and kitchen, personal care, and clothing, shoes and jewelry_.\nTable 2. stats are for the Amazon, Oposum+, and Flipkart test sets. For training, we create synthetic datasets out of the review corpus, statistics for which have been mentioned in Appendix G. Specifically, we create 945k instances from ∼1M reviews over 90k products in the Amazon dataset for general opinion summarization. We create 345k instances from ∼4.13M reviews over 95k products in the Oposum+ dataset for aspect-specific opinion summarization. \n\nEach synthetic dataset instance for general opinion summarization contains the following:\n\n1. Input reviews (8 for Amazon, 10 for Oposum+ and Flipkart)\n\n2. Pseudo-summary (Selected through our synthetic dataset creation strategy)\n\nEach synthetic dataset instance for aspect-specific opinion summarization contains the following:\n\n1. Input reviews (8 for Amazon, 10 for Oposum+ and Flipkart)\n\n2. Aspects (Selected using aspect mapping)\n\n3. Pseudo-summary (Selected through our synthetic dataset creation strategy)\n\n### E. Clarification on Faithfulness\n\nAlthough Line 11 and Line 261 do mention that the faithfulness evaluation is done for general opinion summarization, we apologize for missing the inclusion of this vital information in some key places. (Will be added in the paper)\n\nRebuttal Table 5 gives examples showing the problem of faithfulness in Multimodalsum generated summaries (Will be added to the paper). In comparison, we observe that our model-generated summaries tend to be faithful to the input in most cases. This we quantitatively measure using the different faithfulness evaluation metrics and human evaluations (Tables 5 and 6, Rebuttal Table 4). \n\nAs mentioned in line 49, we specifically propose to curate synthetic datasets in such a way that it enables models to generate summaries that are more faithful to the input. Through experimentation, we show that our synthetic dataset creation strategy works better in generating faithful summaries compared to others. Line 769 mentions the reason for choosing SummaC, CTC, and FactCC. These metrics correlate better with human judgments for evaluating faithfulness [2]. Additionally, we show results for FactGraph in Rebuttal Table 4. Better results on these metrics do indicate that our model-generated summaries are more faithful, thereby indicating that our synthetic dataset creation strategy does aid in training the model to generate faithful summaries.\n\nAdditionally, we employ three Master’s students (will be added to the paper) to evaluate summaries on different criteria such as faithfulness, coherence, conciseness, and fluency. We observe that in human evaluations as well our models score higher in faithfulness thus proving our hypothesis.\n\n\n### F. Reason why our models are performing better than baselines\n**1. General Opinion Summarization:**\n\nIn opinion summarization, synthetic dataset curation is important due to the absence of any large-scale supervised datasets (reviews-summary pairs). Such synthetic datasets enable supervised training for models. \n\nIn Appendix A, we discussed how existing approaches create synthetic datasets followed by the generation of opinion summaries. We, however, hypothesize that a set of reviews and a pseudo-summary can act as a potential pair only if the pseudo-summary is lexically (rouge 1) and semantically (cosine similarity) similar to the review set. We use the matrix construction approach to find such pairs which ensures that only highly relevant pairs are selected in our synthetic dataset. The simple intuition behind this is that during training we want to show the model a pseudo-summary that is most relevant to the input reviews. This will enable the model to learn the task of opinion summarization much more accurately. This is evidenced by results in Table 4 as well as faithfulness scores in Tables 5 and 6 in the paper.\n\n**2. Aspect-specific Opinion Summarization:**\n\nExisting approaches are dependent on human-specified (pre-specified) and seed words for creating synthetic datasets for training and filtering sentences from reviews before generating summaries for inferencing.\n\nOur approach, however, creates aspect mappings automatically which enables the creation of synthetic datasets for a wide range of aspects. Here as well we use the combination of lexical and semantic scores for selecting a pseudo-summary (set of sentences about an aspect) and reviews pair. This ensures that the most relevant review-pseudo summary pair is shown to the model during training enabling better learning for the model to generate summaries. Our synthetic datasets are not restricted by the human-specified aspects and seed words which makes the model more robust in generating summaries for any relevant aspect of a specific product. This we intuit is causing improved scores of our models over existing models even though existing models have this extra aid of human-specified aspects and seed words for training and inference.\n\n### G. Clustering Process\n\nWe used the fast clustering library provided by SBERT [7]. We did not find the specific details of the algorithm. Specific details regarding its implementation will be added to the Appendix.\n\n### H. Synthetic Triplets Creation \nThank you for your comments. We will reframe this section to make the process easier to understand, with cleaner, understandable notations.\n\n### H. Relevant related work in Appendix\nWe do agree that some of the relevant work is in the appendix but it was primarily due to space limitations (4 pages for short paper). We will try to accommodate it in the main paper.\n\n### I. Grammar, Presentation, and Missing References\nThank you for pointing out the issues. Will incorporate all the suggestions in the paper.\n\n### J. References\n[1] [Convex Aggregation for Opinion Summarization](https://aclanthology.org/2021.findings-emnlp.328/) (Iso et al., Findings 2021)\n\n[2] [X-FACTOR: A Cross-metric Evaluation of Factual Correctness in Abstractive Summarization](https://aclanthology.org/2022.emnlp-main.478/) (Chaudhury et al., 2022)\n\n[3] [Aspect-Controllable Opinion Summarization](https://aclanthology.org/2021.emnlp-main.528/) (Amplayo et al., 2021)\n\n[4] [Simple Yet Effective Synthetic Dataset Construction for Unsupervised Opinion Summarization](https://aclanthology.org/2023.findings-eacl.142/) (Shen et al., 2023)\n\n[5] [Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering](https://arxiv.org/abs/1602.01585) (He and McAuley, 2016)\n\n[6] [FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations](https://aclanthology.org/2022.naacl-main.236/) (Ribeiro et al., NAACL 2022)\n\n[7] [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://aclanthology.org/D19-1410/) (Reimers & Gurevych, EMNLP-IJCNLP 2019)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A new way to create synthetic data for aspect-based summarization.\n- With a relatively simple model, the training data enables high scores against previous methods.", "reasons_to_reject": "There are several points that were hard for me to follow:\n- The reference to \"human-specified aspects\" and \"seed-words\" already in the abstract was not clear. Usually aspect-based summarization has a set of pre-specified aspects for a product-type. Case-specific aspects are known as \"open aspects\", and this is not what this paper seems to describe (e.g. for evaluation on Oposum+). It should be made clearer that by having a train set that is not aspect-specific, a model may be able to be trained robustly for any aspect.\n- The motivation that there are faithfulness issues in other synthetic datasets is not clear to me. There is nothing proving this or even giving an intuition for why it might be true. The fact that the paper's baseline gets higher scores on faithfulness doesn't imply that there is a problem in other synthetic datasets.\n- There's no explanation on the \"fast clustering algorithm\" and the thresholds for the granularity of aspects. It should at least be added in appendix E, since this is supposedly part of an important contribution of the paper.\n- The whole \"Synthetic Triplets Creation\" paragraph is very difficult to follow due to the way it is phrased and formalized with notations. E.g.: (1) Line 120: what is \"combined\"; (2) the term \"portion\" is not intuitive without deeply apprehending what is going on; (3) why are r_i1 and r'_t5 used, and not generic indices?; (4) Line 130: what is \"concatenate\"?; (5) Line 131: what is c in c_i1 notation?; (6) Line 134: Between what two instances is cossim and rouge computed?; (7) how are the hyperparameters chosen?; (8) Line 145: why are there two aspects? the standard aspect-based summary only considers one? If this is an extension somehow, it should be explained.\n- Section 2.2 also has some unclear points: (1) Line 168: I have to guess what top-k is; (2) Line 169: What are the \"selected\" reviews?; (3) Line 173: What is D_i?; (4) Again, how are hyperparams chosen? (5) What is the final dataset?; (6) Generally, the approach for synthesizing generic pseudo-summaries (using a combination of two classic similarity metrics) is a well-known method and was used in some of the papers referenced and others. If there is something new about the approach it should be explicitly mentioned.\n- Overall, I don't really understand what the two final datasets contain per instance.\n- In Section 3.1, it's not clear how there are so many instances created when the source datasets are so small. Table 2 talks about a few hundred instances, but the synthetic data is of hundreds of thousands. The appendix says something about it, but even there, it's not clear what the train set of these datasets are.\n- In section 3.3 or in Section 4, it doesn't say anything explicit about the baseline being trained on the synthetic datasets or how (e.g. when aspects are present).\n- Line 233: it says \"even though [the Oposum+ variant] is trained on Amazon dataset\", but Oposum+ *is* based on Amazon data, unless I'm missing something, so it is as expected.\n- Line 243: The automatic faithfulness metrics do not \"prove\" the claim that the summaries are more faithful, they only \"indicate\" it. These metrics are known to be very approximate. Also, is this on the aspect-based or generic summarization? it doesn't say.\n- While most of the relevant related work is in the introduction, some of it is in the appendix, and would be beneficial in the main paper.\n- I think the comparison to \"random training data\" in the ablation in Appendix D is important to further convince the reader that the training set is indeed worthwhile. It's hidden away in the appendix, and should be mentioned in the paper itself.\n- Some presentation points confused me (see in separate box)\n\nGenerally, it's hard to tell what actually causes the improvement against the baselines. Is it the BART-based model? Is it the abundance of the training data? Is it the quality of the data? The comparison is not so fair since the baseline models are not trained on the same data. If the point is to show that the proposed synthetic data is better and also improves faithfulness, the convincing way to do it would be to train several models with different training data separately (and taking quantity of training instances into account), and comparing the results on that. The storyline of this current version of the paper would be convincing that way much more. Otherwise, the paper can be motivated differently and then corresponding proper experiments would be required.\n\nAlso, since aspects are prepared on-the-fly, it says that the proposed strategy enables creating synthetic aspect-based summarization data for other domains as well. But I'm not sure about that because reviews and, e.g., news, have very different behavior when pseudo-summaries are the main player. This assumption needs to be checked.\n\nSorry for the many comments, I'm trying to provide constructive points to improve the paper, at least in my eyes.\n\n\n------\n\nUPDATE: I have raised my scores from 2/2 to 3/3, hoping that the authors will indeed update the paper as they have suggested in the rebuttals.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes methods for creating synthetic training sets for generic and aspect-based review summarization.\n- For aspect-based summarization, an aspect mapping is first prepared using a two-level clustering method (on SentenceBERT representations of aspects extracted by the off-the-shelf InstructABSA). The fine-grained \"aspects\" in review sentences can be mapped to more general coarse grained ones. Then \"portions\" of reviews are prepared for each coarse-grained aspect, and partial reviews are grouped together with related full reviews to form triplets of reviews, aspects and a pseudo-summary.\n- For generic summarization, reviews that are most similar to other reviews on ROUGE and cosine-similarity of sentenceBERT are set as pseudo-summaries for the others.\n- A BART-based model is trained on each of these datasets, and compared to several baselines on Amazon and Flipkart product reviews. The model is generally the best overall in ROUGE and in three faithfulness metrics. It is also best overall in a manual evaluation against two other relatively strong baselines on faithfulness, coherence, conciseness and fluency.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "- Some of the baseline systems in 3.2 don't have references in place."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- \"General\" and \"aspect-specific\" summarization are usually termed \"generic\" and \"aspect-based\" summarization. This confused me at first, that maybe this paper was regarding different tasks that I was unaware of.\n- Line 43: Text summarization does not \"contain\" reference summaries, rather a dataset does.\n- Figure 1: The \"filtered-reviews using scores\" diagram is not clear, even after reading the whole paper, and especially not when the figure is first referenced.\n- Second contribution bullet in intro: Explicitly say that this is for generic summarization.\n- Line 118: \"... *consist* of sentences and the *corresponding* sentence-level...\n- Paragraph titles are usually not in title case, and end with a period.\n- Line 161: \"unified\" is not so clear, just \"combined\""}}]

Author rebuttal:

Meta-review: The paper introduces methods for creating synthetic training data to improve opinion summarization, addressing both general and aspect-based summarization. The paper demonstrates improvements over strong baselines in various evaluation metrics, supported by human assessment. However, concerns were raised in the reviews about the introduction of numerous hyperparameters, potential limitations in generalizability, the absence of a baseline comparison with Coop in unsupervised general opinion summarization, and the need for a more comprehensive discussion of language model-based methods in the post-ChatGPT era. Additionally, issues related to presentation clarity, ambiguities, and details in various sections were noted, along with recommendations for a fairer comparison and a clearer motivation regarding faithfulness issues in other synthetic datasets. Despite these issues, reviewers were satisfied by most of the responses of the authors during the rebuttal period.
Decision: Accept-Findings