SlowGuess/NAACL_2022_OCR huggingface dataset:
For each Amazon test set entry and criterion, we asked three independent workers on Amazon Mechanical Turk (AMT) to select the best and worst summary. For each criterion, a system's score is computed as the percentage of times it was selected as best, minus the percentage of times it was selected as worst (Orme, 2009). The scores range from -100 (unanimously worst) to +100 (unanimously best). For more details, please refer to Appendix 10.2.
First, the results indicate that the summaries generated by ADAQSUM are substantially more preferred to ADASUM in terms of coherence and nonredundancy. Namely, +13.73 vs -30.91 and -1.96 vs -25.93 for coherence and non-redundancy, respectively. We also computed the percentage of unique n-grams in each generated summary for both datasets, as shown in Table 5. The results support that query-based summaries are less redundant. However, similar to findings in Oved and Levy (2021), we observe that more coherent summaries tend to get lower ROUGE scores. Nevertheless, our model outperforms PASS by a margin on both datasets – by 2.72 and 0.9 ROUGE-L points on Amazon and Yelp, respectively.
7Training sets are of similar sizes, i.e., 84 and 90 summaries on Amazon and Yelp, respectively
R1	R2	RL	unique 1-gram (%)	unique 2-gram (%)
Amazon	ADASUM (5%) + L1O	39.78	10.80	25.55	67.72	80.83
ADAQSUM (5%) + L1O	38.53	10.52	26.06	69.38	82.57
Yelp	ADASUM (5%) + L1O	38.82	11.75	25.14	62.26	76.55
ADAQSUM (5%) + L1O	36.79	10.06	23.99	65.74	79.88
Table 5: Comparison of the query-based and generic summarizers on test sets. Unique n-grams were computed in generated summaries.
Fluency	Coher.	Non-Red.
PASS	-21.74	+33.33	0.00
LEXRANK	-45.95	-52.38	-58.97
ADAQSUM (5%) + L1O	+26.67	+25.00	+26.67
GOLD	+46.67	+27.78	+55.56
Comparison to Baselines. To understand better how our query-based model compares to other models, we performed an additional human evaluation experiment. We used the following criteria: coherence, non-redundancy, and fluency. As previously, we used the Best-Worst scaling on the Amazon test set. We assigned three AMT workers to each tuple containing summaries from PASS, ADAQSUM  $(5\%)+$  L1O, LEXRANK, and human annotators (GOLD).
The results in Table 6 suggest that summaries produced by our model are more fluent and nonredundant than the ones produced by PASS. In general, PASS produces more diverse and detailed summaries yet with more semantic mistakes that make them harder to understand (hence lower fluency scores). However, summaries by both systems are similarly preferred in terms of coherence. Also, we note that PASS utilizes a separately trained classifier on human-annotated summaries (Fabbri et al., 2021) to rank candidate summaries, while our approach does not.
Input Content Fidelity. As was shown in Falke et al. (2019); Tay et al. (2019), the ROUGE metric can be insensitive to hallucinations (Maynez et al., 2020). However, hallucinations can lead to user aversion, and their reduction remains an open problem in summarization. To assess the input fidelity of generated summaries, we performed a human evaluation. Specifically, we used summaries produced by the adapter models (ADASUM  $(5\%)$  + L1O and ADAQSUM  $(5\%)$  + L1O), FEWSUM, PASS, and human-written (GOLD). In each task (HIT), we presented both reviews and all summary sentences. We asked three workers to assess how
Table 6: Human evaluation results in terms of the BestWorst scaling on the Amazon test set.
Full↑	Partial↑	No↓
FEWSUM	47.56	24.39	28.05
PASS	60.70	31.84	7.46
ADASUM (5%) + L1O	78.97	15.48	5.56
ADAQSUM (5%) + L1O	72.69	20.37	6.94
Table 7: Input fidelity on the Amazon test set, normalized by sentences.
well the content in summary sentences is supported by the reviews. The three following options were available. Full support: all the content is reflected in the reviews; Partial support: only some content is reflected in the reviews; No support: content is not reflected in the reviews. The results, normalized by sentences, are shown in Table 7.
First, we observe that FEWSUM hallucinates the most, potentially because it was not initialized with a pre-trained language model. Second, PASS improves input fidelity over FEWSUM yet substantially underperforms our adapter-based models. We also notice a slight decrease in input fidelity when the query is used. This is likely caused by more abstractive summaries generated by ADAQSUM, we discuss it in Sec. 5.3.
5.1 Query-based Pre-training
Query-based summarizers should generate summaries reflecting all aspects in user queries to be useful in practice. We investigated how summarizers learn this task in the few-shot regime with and without pre-training. We created test-time queries from gold summaries (indicated by *) and input reviews. Further, we calculated the aspect recall (AR) score by counting aspect keywords in queries present in generated summaries. The results are shown in Table 8.
As indicated by low AR scores, without pretraining, the models miss many aspects in queries. The increase to nearly  $100\%$  in AR suggests that pre-training is crucial for the task. The same trend remains when aspect keywords from reviews are used in queries.
R1	R2	RL	AR
FULL (100%) + Q*	40.52	10.96	25.06	59.84
FULL (100%) + L1O + Q*	42.65	11.53	26.82	96.39
ADAQSUM (5%)*	41.04	11.08	25.46	60.64
ADAQSUM (5%) + L1O*	43.84	13.41	27.31	97.19
ADAQSUM (5%)	38.58	10.10	24.19	69.14
ADAQSUM (5%) + L1O	38.53	10.52	26.06	98.78
Table 8: Amazon test set ROUGE F1 for query-based summarization. Here, * indicates that queries were created from gold summaries; AR stands for aspect recall.
PPL↓
FULL (100%) + L1O	21.51
FULL (100%) + L1O + FT	34.87 (+13.36)
ADASUM (5%) + L1O	19.69
ADASUM (5%) + L1O + FT	28.45 (+8.76)
5.2 Catastrophic Forgetting
ROUGE scores in Table 4 suggest that L10 pretraining is beneficial for the end task. However, fine-tuning on summaries can lead to the catastrophic forgetting of the acquired in-domain specifics from reviews. Because adapters have fewer parameters to optimize, we hypothesized that they might be more robust to this phenomenon.
To test the hypothesis, we evaluated two models on the pre-training L1O pairs where a review is used as a summary, before and after fine-tuning on human-written summaries. For the first model, we optimized only adapters (5%), both in pre-training and fine-tuning. And in the second case, we optimized the entire model. We used PPL to measure the model's confusion about the pre-training pseudo summaries, as shown in Table 9.
The results demonstrate that the adapter-based model better preserves information about reviews after they are fine-tuned on summaries, as indicated by lower PPL scores. Our findings are also supported by Yu et al. (2021).
5.3 Abstractiveness
Abstracting information in reviews is important for practical applications (Carenini and Cheung, 2008). To investigate how well the models abstract, we computed the number of novel n-grams in generated summaries with respect to input reviews on the Amazon test set. The results in percentages are shown in Table 10.
First, we observe that FewSum tends to produce
Table 9: Catastrophic forgetting evaluation on the Amazon pre-training task's validation set, before and after fine-tuning (FT).
2-gram	3-gram	4-gram
FEWSUM	78.63	95.59	98.74
PASS	70.72	86.32	93.24
ADASUM (5%) + L1O	55.47	78.24	86.78
ADAQSUM (5%) + L1O	56.27	79.18	88.48
Table 10: The abstractiveness of generated summaries in terms of novel n-grams on the Amazon test set.
the most abstractive summaries, followed by PASS. Second, ADAQSUM has higher abstractiveness than ADASUM. We also observe that abstractiveness is inversely proportional to input faithfulness in Table 7, in line with previous studies (Durmus et al., 2020; Dreyer et al., 2021).
5.4 Semantic Mistakes
When a pre-trained model (with and without adapters) is fine-tuned on a handful of annotated samples, it often results in summaries with subtle semantic mistakes; see examples in Table 11. For instance, a  $5\%$  adapter model generates a semantically contradicting fragment: 'This dead on arrival battery is of good quality and holds a charge well'.
We hypothesize that it is caused by the lack of indomain knowledge, which we propose to learn via L1O (see Sec. 2.4). During a manual investigation, we observed that L1O pre-training substantially reduces semantic mistakes. This is also reflected in higher ROUGE scores in Table 4.
Extractive opinion summarization has been an active area of research (Hu and Liu, 2004; Ganesan et al., 2010; Medhat et al., 2014; Isonuma et al., 2019; Angelidis et al., 2020). For example, a more recent extractive method of Angelidis and Lapata (2018) decouples the summarization procedure into multiple steps with separate models. Other earlier approaches (Gerani et al., 2014; Di Fabbrizio et al., 2014) relied on text planners and templates, which, however, restrict the output.
Abstractive opinion summarization is an emerging branch (Chu and Liu, 2019; Amplayo and Lapata, 2020; Bražinskas et al., 2020b, 2021a). Customer reviews were used to train unsupervised summarizers in Amplayo and Lapata (2020); Bražinskas et al. (2020b); Isonuma et al. (2021). The few-shot model FEWSUM (Bražinskas et al., 2020a) was also pre-trained on customer reviews before fine-tuning. In this work instead, we focus on pretraining adapters to avoid catastrophic forgetting
ADASUM
(5%)	This Thomas the Train costume is cute and functional. The size is perfect for a toddler or 2 year old and the candy pouch is large enough to carry a lot of candy. The costume can be customized with googly eyes, pumpkin patch, spiders, bats, or train tracks to make it more suitable for a child of any age. The fit is comfortable and the fit quality is great. The only complaint I have is the quality of the felt fabric.
ADASUM
(5%) + L1O	This Thomas the Train costume is very cute and the size is perfect for a 2-year-old. The hat is thin and flimsy and the face is not 3D sculpted. The candy pouch is a nice feature and it looks like it will grow with the child. Overall, it's a cute costume and will be used for Hal-loween next year.
ADASUM
(5%)	This dead on arrival battery is of good quality and holds a charge well. It is easy to install and is a great value for the money. However, it may not hold a charge as advertised due to the plas-tic case bulging. Overall, this product is highly recommended.
ADASUM
(5%) + L1O	This battery is a great value for the price and works great. It is a good quality battery that can be used to replace a dead battery in an alarm system. The price is great and the quality of the product is good. The shipping was fast and the customer service was excellent.
Table 11: Adapter-based models (5%) and their generated outputs with and without L1O pre-training. Semantic mistakes and disfluencies are highlighted in bold.
and reduce computational and memory overheads. OPINIONDIGEST (Suhara et al., 2020) proposes to aggregate opinions in a pipeline framework. We approach the problem end-to-end and rely on aspect keywords (e.g., price) instead of opinion phrases (e.g., good location). Controllability using input fragments (e.g., entities) and meta information (e.g., coarse-grained aspects) has received recent attention in various NLP domains (Frermann and Klementiev, 2019; Liu and Chen, 2021; Narayan et al., 2021; Elsahar et al., 2021). In contrast to a related work, SELFSUM (Elsahar et al., 2021), we use aspect keywords instead of generic tokens and consider a few-shot setup instead of unsupervised and a different model architecture. Also, planning was tackled in opinion summarization in Amplayo et al. (2021b). However, their approach is substantially less flexible, as the summary plan consists of an aspect and sentiment classes only. Query-based settings have received recent attention in the news
domain (Xu and Lapata, 2020, 2021). Compared to a concurrent work on opinion summarization ACESUM (Amplayo et al., 2021a), our approach does not require a trained aspect induction model, is few-shot instead of self-supervised, and benefits from a large collection of automatically created fine-grained aspects (a couple of thousands) instead of human annotated coarse-grained aspects (up to 18). Concurrently with our work, Poth et al. (2021) support our findings on the benefits of pre-training adapters for other tasks.
In this work, we improve few-shot learning for opinion summarization with adapters pre-training on customer reviews in the end task-specific manner. In this way, the model learns in-domain specifics, which reduces semantic mistakes in generated summaries. We show that our approach leads to more than 2.0 and 1.3 ROUGE-L points improvement over the entire model's fine-tuning on the Amazon and Yelp datasets, respectively. Further, we propose a simple method for few-shot query-based summarization. The queries consist of aspect keywords reflecting potential user interests. We create these queries automatically and show that pre-training is crucial for the end task, significantly improving performance. Finally, in human evaluation, we demonstrate that the query-based model generates more coherent and less redundant summaries.
We would like to thank Jonathan Pilault, Leonardo Ribeiro, Sandeep Atluri, and the Amazon AI teams for their useful feedback and discussions.
We used only publicly available datasets. For human evaluation, we used a publicly available service (Amazon Mechanical Turk) to hire voluntary participants, requesting native speakers of English. The participants were fairly compensated, above the minimum hourly wage in their self-identified countries of residence.
In this work, we explicitly focus on multi-document abstractive opinion summarization. However, our pre-training self-supervised method
and fine-tuning techniques can be applied to a broader set of multi-document summarization domains (e.g., news) and can be considered in the future work. Also, while we tested our approach only with BART, we believe that it would work with other pre-trained encoder-decoder models, like PE-GASUS (Zhang et al., 2020).
Reinald Kim Amplayo, Stefanos Angelidis, and Mirella Lapata. 2021a. Aspect-controllable opinion summarization. arXiv preprint arXiv:2109.03171.
Reinald Kim Amplayo, Stefanos Angelidis, and Mirella Lapata. 2021b. Unsupervised opinion summarization with content planning. In AAAI.
Reinald Kim Amplayo and Mirella Lapata. 2020. Unsupervised opinion summarization with noising and denoising. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1934-1945, Online. Association for Computational Linguistics.
Stefanos Angelidis, Reinald Kim Amplayo, Yoshihiko Suhara, Xiaolan Wang, and Mirella Lapata. 2020. Extractive opinion summarization in quantized transformer spaces. In In Transactions of the Association for Computational Linguistics (TACL).
Stefanos Angelidis and Mirella Lapata. 2018. Summarizing opinions: Aspect extraction meets sentiment prediction and they are both weakly supervised. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3675-3686, Brussels, Belgium. Association for Computational Linguistics.
Ankur Bapna, Naveen Arivazhagan, and Orhan First. 2019. Simple, scalable adaptation for neural machine translation. arXiv preprint arXiv:1909.08478.
Arthur Bražinskas, Mirella Lapata, and Ivan Titov. 2020a. Few-shot learning for opinion summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4119-4135, Online. Association for Computational Linguistics.
Arthur Bražinskas, Mirella Lapata, and Ivan Titov. 2020b. Unsupervised opinion summarization as copycat-review generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5151-5169, Online. Association for Computational Linguistics.
Arthur Bražinskas, Mirella Lapata, and Ivan Titov. 2021a. Learning opinion summarizes by selecting informative reviews. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9424-9442, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Arthur Bražinskas, Mengwen Liu, Ramesh Nallapati, Sujith Ravi, and Markus Dreyer. 2021b. Transductive learning for abstractive news summarization. arXiv preprint arXiv:2104.09500.
Giuseppe Carenini and Jackie C. K. Cheung. 2008. Extractive vs. NLG-based abstractive summarization of evaluative text: The effect of corpus controversiality. In Proceedings of the Fifth International Natural Language Generation Conference, pages 33-41, Salt Fork, Ohio, USA. Association for Computational Linguistics.
Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. 2020. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7870-7881, Online. Association for Computational Linguistics.
Eric Chu and Peter Liu. 2019. Meansum: a neural model for unsupervised multi-document abstractive summarization. In Proceedings of International Conference on Machine Learning (ICML), pages 1223-1232.
Giuseppe Di Fabbrizio, Amanda Stent, and Robert Gaizauskas. 2014. A hybrid approach to multi-document summarization of opinions in reviews. In Proceedings of the 8th International Natural Language Generation Conference (INLG), pages 54-63.
Markus Dreyer, Mengwen Liu, Feng Nan, Sandeep Atluri, and Sujith Ravi. 2021. Analyzing the abstractiveness-factuality tradeoff with nonlinear abstractiveness constraints. CoRR, abs/2108.02859.
Esin Durmus, He He, and Mona Diab. 2020. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055-5070, Online. Association for Computational Linguistics.
Hady Elsahar, Maximin Coavoux, Jos Rozen, and Matthias Galle. 2021. Self-supervised and controlled multi-document opinion summarization. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1646-1662, Online. Association for Computational Linguistics.
Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of artificial intelligence research, 22:457-479.
Alexander R Fabbri, Wojciech Krysciński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391-409.
Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2214-2220, Florence, Italy. Association for Computational Linguistics.
Lea Frermann and Alexandre Klementiev. 2019. Inducing document structure for aspect-based summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6263-6273, Florence, Italy. Association for Computational Linguistics.
Kavita Ganesan, ChengXiang Zhai, and Jiawei Han. 2010. Opinosis: A graph based approach to abstractive summarization of highly redundant opinions. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 340-348, Beijing, China. Coling 2010 Organizing Committee.
Shima Gerani, Yashar Mehdad, Giuseppe Carenini, Raymond T. Ng, and Bita Nejat. 2014. Abstractive summarization of product reviews using discourse structure. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1602-1613, Doha, Qatar. Association for Computational Linguistics.
Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. 2013. An empirical investigation of catastrophic forgetting in gradient-based neural networks. In Proceedings of International Conference on Learning Representations (ICLR).
Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jiawei Low, Lidong Bing, and Luo Si. 2021. On the effectiveness of adapter-based tuning for pretrained language model adaptation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2208-2222, Online. Association for Computational Linguistics.
Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In Proceedings of the 25th international conference on world wide web, pages 507-517. International World Wide Web Conferences Steering Committee.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790-2799. PMLR.
Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth
ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168-177. ACM.
Xinyu Hua and Lu Wang. 2019. Sentence-level content planning and style specification for neural text generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 591-602, Hong Kong, China. Association for Computational Linguistics.
Masaru Isonuma, Junichiro Mori, Danushka Bollegala, and Ichiro Sakata. 2021. Unsupervised abstractive opinion summarization by generating sentences with tree-structured topic guidance. Transactions of the Association for Computational Linguistics, 9:945–961.
Masaru Isonuma, Junichiro Mori, and Ichiro Sakata. 2019. Unsupervised neural single-document summarization of reviews via learning latent discourse structure and its ranking. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2142-2152, Florence, Italy. Association for Computational Linguistics.
Ronald Kemker, Marc McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan. 2017. Measuring catastrophic forgetting in neural networks. arXiv preprint arXiv:1708.02072.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Svetlana Kiritchenko and Saif M. Mohammad. 2016. Capturing reliable fine-grained sentiment associations by crowdsourcing and best-worst scaling. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 811-817, San Diego, California. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Zhengyuan Liu and Nancy Chen. 2021. Controllable neural dialogue summarization with personal named entity planning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 92-106, Online and Punta