SlowGuess/ACL_2020_OCR huggingface dataset:
Best-Worst Scaling We performed human evaluation using the AMT platform. We sampled 50 businesses from the human-annotated Yelp test set and used all 32 test products from the Amazon set. We recruited 3 workers to evaluate each tuple containing summaries from MeanSum, our model, LexRank, and human annotators. The reviews and summaries were presented to the workers in random order and were judged using BestWorst Scaling (Louviere and Woodworth, 1991; Louviere et al., 2015). BWS has been shown to produce more reliable results than ranking scales (Kiritchenko and Mohammad, 2016). Crowdworkers were asked to judge summaries according to the
For experiments on Yelp, we used the checkpoint provided by the authors, as we obtained very similar ROUGE scores when retraining the model.
Fluency	Coherence	Non Red.	Opinion Cons.	Overall
Copycat	0.5802	0.5161	0.4722	-0.0909	0.3818
MeanSum	-0.5294	-0.4857	0.0270	-0.6235	-0.7468
LexRank	-0.7662	-0.8293	-0.7699	0.3500	-0.5278
Gold	0.6486	0.8140	0.6667	0.3750	0.8085
Table 5: Human evaluation results in terms of the Best-Worst scaling on the Yelp dataset.
Fluency	Coherence	Non Red.	Opinion Cons.	Overall
Copycat	0.4444	0.3750	0.0270	-0.4286	-0.1429
MeanSum	-0.6410	-0.8667	-0.6923	-0.7736	-0.8305
LexRank	-0.2963	-0.3208	-0.3962	0.4348	0.1064
Gold	0.3968	0.7097	0.7460	0.6207	0.7231
Table 6: Human evaluation results in terms of the Best-Worst scaling on the Amazon dataset.
criteria listed below (we show an abridged version below, the full set of instructions is given in Appendix A.5). The non-redundancy and coherence criteria were taken from Dang (2005).
Fluency: the summary sentences should be grammatically correct, easy to read and understand; Coherence: the summary should be well structured and well organized; Non-redundancy: there should be no unnecessary repetition in the summary; Opinion consensus: the summary should reflect common opinions expressed in the reviews; Overall: based on your own criteria (judgment) please select the best and the worst summary of the reviews.
For every criterion, a system's score is computed as the percentage of times it was selected as best minus the percentage of times it was selected as worst (Orme, 2009). The scores range from -1 (unanimously worst) to +1 (unanimously best).
On Yelp, as shown in Table 5, our model scores higher than the other models according to most criteria, including overall quality. The differences with other systems are statistically significant for all the criteria at  $p < 0.01$ , using post-hoc HD Tukey tests. The difference in fluency between our system and gold summaries is not statistically significant.
The results on Amazon are shown in Table 6. Our system outperforms other methods in terms of fluency, coherence, and non-redundancy. As with Yelp, it trails LexRank according to the opinion consensus criterion. Additionally, LexRank is slightly preferable overall. All pairwise differences between our model and comparison systems are statistically significant at  $p < 0.05$ .
Opinion consensus (OC) is a criterion that cap
tures the coverage of common opinions, and it seems to play a different role in the two datasets. On Yelp, LexRank has better coverage compared to our model, as indicated by the higher OC score, but is not preferred overall. In contrast, on Amazon, while the OC score is on the same par, LexRank is preferred overall. We suspect that presenting a breadth of exact details on Amazon is more important than on Yelp. Moreover, LexRank tends to produce summaries that are about 20 tokens longer than ours resulting in better coverage of input details.
Content Support The ROUGE metric relies on unweighted n-gram overlap and can be insensitive to hallucinating facts and entities (Falke et al., 2019). For example, referring to a burger joint as a veggie restaurant is highly problematic from a user perspective but yields only marginal differences in ROUGE. To investigate how well the content of the summaries is supported by the input reviews, we performed a second study. We used the same sets as in the human evaluation in Section 5.2, and split MeanSum and our system's summaries into sentences. Then, for each summary sentence, we assigned 3 AMT workers to assess how well the sentence is supported by the reviews. Workers were advised to read the reviews and rate sentences using one of the following three options. Full support: all the content is reflected in the reviews; Partial support: only some content is reflected in the reviews; No support: content is not reflected in the reviews.
The results in Table 7 indicate that our model is better at preserving information than MeanSum.
Yelp	Amazon
Copycat	MeanSum	Copycat	MeanSum
Full	44.50	28.41	38.23	24.41
Partial	32.48	30.66	33.95	31.23
No	23.01	40.92	27.83	44.36
Ablations To investigate the importance of the model's individual components, we performed ablations by removing the latent variables  $(z_{i}$  and  $c$  one at a time), and attention over the other reviews. The models were re-trained on the Amazon dataset. The results are shown in Table 8. They indicate that all components play a role, yet the most significant drop in ROUGE was achieved when the variable  $z$  was removed, and only  $c$  remained. Summaries obtained from the latter system were wordier and looked more similar to reviews. Dropping the attention  $(\mathrm{w / o}r_{-i})$  results in more generic summaries as the model cannot copy details from the input. Finally, the smallest quality drop in terms of ROUGE-L was observed when the variable  $c$  was removed.
In the introduction, we hypothesized that using the mean of latent variables would result in more "grounded" summaries reflecting the content of the input reviews, whereas sampling would yield texts with many novel and potentially irrelevant details. To empirically test this hypothesis, we sampled the latent variables during summary generation, as opposed to using mean values (see Section 3). We indeed observed that the summaries were wordier, less fluent, and less aligned to the input reviews, as is also reflected in the ROUGE scores (Table 8).
Copy Mechanism Finally, we analyzed which words are copied by the full model during summary generation. Generally, the model copies around 3-4 tokens per summary. We observed a tendency to copy product-type specific words (e.g., shoes) as well as brands and names.
Extractive weakly-supervised opinion summarization has been an active area of research. A recent example is Angelidis and Lapata (2018). First, they learn to assign sentiment polarity to review segments in a weakly-supervised fashion. Then, they induce aspect labels for segments relying on
Table 7: Content support on Yelp and Amazon datasets, percentages.
R1	R2	RL
w/o r_i	0.2866	0.0454	0.1863
w/o c	0.2767	0.0507	0.1919
w/o z	0.2926	0.0416	0.1739
Sampling	0.2563	0.0434	0.1716
Full	0.3197	0.0581	0.2016
Table 8: Ablations, ROUGE scores on Amazon.
a small sample of gold summaries. Finally, they use a heuristic to construct a summary of segments. Opinosis (Ganesan et al., 2010) does not use any supervision. The model relies on redundancies in opinionated text and PoS tags in order to generate short opinions. This approach is not well suited for the generation of coherent long summaries and although it can recombine fragments of input text, it cannot generate novel words and phrases. LexRank (Erkan and Radev, 2004) is an unsupervised extractive approach which builds a graph in order to determine the importance of sentences, and then selects the most representative ones as a summary. Isonuma et al. (2019) introduce an unsupervised approach for single review summarization, where they rely on latent discourse trees. Other earlier approaches (Gerani et al., 2014; Di Fabbrizio et al., 2014) relied on text planners and templates, while our approach does not require rules and can produce fluent and varied text. Finally, conceptually related methods were applied to unsupervised single sentence compression (West et al., 2019; Baziotis et al., 2019; Miao and Blunsom, 2016). The most related approach to ours is MeanSum (Chu and Liu, 2019) which treats a summary as a discrete latent state of an autoencoder. In contrast, we define a hierarchical model of a review collection and use continuous latent codes.
In this work, we presented an abstractive summarizer of opinions, which does not use any summaries in training and is trained end-to-end on a large collection of reviews. The model compares favorably to the competitors, especially to the only other unsupervised abstractive multi-review summarization system. Furthermore, human evaluation of the generated summaries (by considering their alignment with the reviews) shows that those created by our model better reflect the content of the input.
We would like to thank the anonymous reviewers for their valuable comments. Also, Stefanos Angelidis for help with the data as well as Jonathan Mallinson, Serhii Havrylov, and other members of Edinburgh NLP group for discussion. We gratefully acknowledge the support of the European Research Council (Titov: ERC StG BroadSem 678254; Lapata: ERC CoG TransModal 681760) and the Dutch National Science Foundation (NWO VIDI 639.022.518).
Stefanos Angelidis and Mirella Lapata. 2018. Summarizing opinions: Aspect extraction meets sentiment prediction and they are both weakly supervised. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3675-3686.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of International Conference on Learning Representations (ICLR).
Regina Barzilay, Kathleen R McKeown, and Michael Elhadad. 1999. Information fusion in the context of multi-document summarization. In Proceedings of the 37th annual meeting of the Association for Computational Linguistics, pages 550-557.
Christos Baziotis, Ion Androutsopoulos, Ioannis Konstas, and Alexandros Potamianos. 2019. Seq^3: Differentiable sequence-to-sequence-to-sequence autoencoder for unsupervised abstractive sentence compression. In Proceedings of the Association for Computational Linguistics, pages 673-681.
Julian Besag. 1975. Statistical analysis of non-lattice data. Journal of the Royal Statistical Society: Series D (The Statistician), 24(3):179-195.
John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th annual meeting of the association of computational linguistics, pages 440-447.
Samuel Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In Proceedings of the Twentieth Conference on Computational Natural Language Learning (CoNLL).
Giuseppe Carenini and Jackie Chi Kit Cheung. 2008. Extractive vs. nlg-based abstractive summarization
of evaluative text: The effect of corpus controversiality. In Proceedings of the Fifth International Natural Language Generation Conference, pages 33-41. Association for Computational Linguistics.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724-1734.
Eric Chu and Peter Liu. 2019. Meansum: a neural model for unsupervised multi-document abstractive summarization. In Proceedings of International Conference on Machine Learning (ICML), pages 1223-1232.
Hoa Trang Dang. 2005. Overview of duc 2005. In Proceedings of the document understanding conference, volume 2005, pages 1-12.
Giuseppe Di Fabbrizio, Amanda Stent, and Robert Gaizauskas. 2014. A hybrid approach to multi-document summarization of opinions in reviews. pages 54-63.
Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of artificial intelligence research, 22:457-479.
Tobias Falke, Leonardo FR Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2214-2220.
Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, and Lawrence Carin. 2019. Cyclic annealing schedule: A simple approach to mitigating kl vanishing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 240-250.
Kavita Ganesan, ChengXiang Zhai, and Jiawei Han. 2010. Opinosis: A graph based approach to abstractive summarization of highly redundant opinions. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 340-348.
Shima Gerani, Yashar Mehdad, Giuseppe Carenini, Raymond T Ng, and Bita Nejat. 2014. Abstractive summarization of product reviews using discourse structure. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1602-1613.
5159
Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256.
Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In Proceedings of the 25th international conference on world wide web, pages 507-517. International World Wide Web Conferences Steering Committee.
Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751.
Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168-177. ACM.
Masaru Isonuma, Toru Fujino, Junichiro Mori, Yutaka Matsuo, and Ichiro Sakata. 2017. Extractive summarization using multi-task learning with document classification. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2101-2110.
Masaru Isonuma, Junichiro Mori, and Ichiro Sakata. 2019. Unsupervised neural single-document summarization of reviews via learning latent discourse structure and its ranking. In Proceedings of ACL.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Diederik P Kingma and Max Welling. 2013. Autoencoding variational bayes. arXiv preprint arXiv:1312.6114.
Svetlana Kiritchenko and Saif M Mohammad. 2016. Capturing reliable fine-grained sentiment associations by crowdsourcing and best-worst scaling. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 811-817.
Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in neural information processing systems, pages 3294-3302.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the association for computational linguistics companion volume proceedings of the demo and poster sessions, pages 177-180.
Bing Liu. 2012. Sentiment analysis and opinion mining. Synthesis lectures on human language technologies, 5(1):1-167.
Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. In Proceedings of International Conference on Learning Representations (ICLR).
Jordan J Louviere, Terry N Flynn, and Anthony Alfred John Marley. 2015. Best-worst scaling: Theory, methods and applications. Cambridge University Press.
Jordan J Louviere and George G Woodworth. 1991. Best-worst scaling: A model for the largest difference judgments. University of Alberta: Working Paper.
Walaa Medhat, Ahmed Hassan, and Hoda Korashy. 2014. Sentiment analysis algorithms and applications: A survey. Ain Shams engineering journal, 5(4):1093-1113.
Qiaozhu Mei, Xu Ling, Matthew Wondra, Hang Su, and ChengXiang Zhai. 2007. Topic sentiment mixture: modeling facets and opinions in weblogs. In Proceedings of the 16th international conference on World Wide Web, pages 171-180. ACM.
Yishu Miao and Phil Blunsom. 2016. Language as a latent variable: Discrete generative models for sentence compression. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 319-328.
Frederic Morin and Yoshua Bengio. 2005. Hierarchical probabilistic neural network language model. *Aistats*, 5:246-252.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280-290.
Bryan Orme. 2009. Maxdiff analysis: Simple counting, individual-level logit, and hb. Sequim, WA: Sawtooth Software.
Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford InfoLab.
Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304.
Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 157-163.
Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379-389.
Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of Association for Computational Linguistics (ACL).
Ivan Titov and Ryan McDonald. 2008. Modeling online reviews with multi-grain topic models. In Proceedings of the 17th international conference on World Wide Web, pages 111-120. ACM.
Peter West, Ari Holtzman, Jan Buys, and Yejin Choi. 2019. Bottlesum: Unsupervised and self-supervised sentence summarization using the information bottleneck principle. arXiv preprint arXiv:1909.07405.
A.1 Derivation of the Lower Bound
To make the notation below less cluttered, we make a couple of simplifications:  $q_{\phi}(c|\cdot) = q_{\phi}(c|r_{1:N})$  and  $q_{\phi}(z|i) = q_{\phi}(z|r_i,c)$ .
$$
\begin{array}{l} \log \int \left[ p _ {\theta} (c) \prod_ {i = 1} ^ {N} p _ {\theta} (r _ {i} | c, r _ {- i}) \mathrm {d} c \right] \\ = \log \int \left[ p _ {\theta} (c) \prod_ {i = 1} ^ {N} \left[ \int p _ {\theta} \left(r _ {i}, z | c, r _ {\hat {z}}\right) \mathrm {d} z \right] \mathrm {d} c \right] \\ = \log \int \left[ p _ {\theta} (c) \frac {q _ {\phi} (c | \cdot)}{q _ {\phi} (c | \cdot)} \prod_ {i = 1} ^ {N} \left[ \int p _ {\theta} \left(r _ {i}, z \mid c, r _ {- i}\right) \right. \right. \\ \left. \frac {q _ {\phi} (z | i)}{q _ {\phi} (z | i)} \mathrm {d} z \right] \mathrm {d} c \bigg ] = \log \\ \underset {c \sim q _ {\phi} (c | \cdot)} {\mathbb {E}} \left[ \frac {p _ {\theta} (c)}{q _ {\phi} (c | \cdot)} \prod_ {i = 1} ^ {N} \underset {z \sim q _ {\phi} (z | i)} {\mathbb {E}} \left[ \frac {p _ {\theta} (r _ {i} , z | c , r _ {- i})}{q _ {\phi} (z | i)} \right] \right] \\ \geq \underset {c \sim q _ {\phi} (c | \cdot)} {\mathbb {E}} \left[ \sum_ {i = 1} ^ {N} \log \underset {z \sim q _ {\phi} (z | i)} {\mathbb {E}} \left[ \frac {p _ {\theta} (r _ {i} , z | c , r _ {- i})}{q _ {\phi} (z | i)} \right] \right] \\ - \mathbb {D} _ {\mathrm {K L}} \left[ q _ {\phi} (c | \cdot)\right) \| p _ {\theta} (c) ] \geq \\ \underset {c \sim q _ {\phi} (c | \cdot)} {\mathbb {E}} \left[ \sum_ {i = 1} ^ {N} \underset {z \sim q _ {\phi} (z | i)} {\mathbb {E}} \left[ \log \frac {p _ {\theta} \left(r _ {i} , z \mid c , r _ {- i}\right)}{q _ {\phi} (z | i)} \right] \right] - \\ \mathbb {D} _ {\mathrm {K L}} \left[ q _ {\phi} (c | \cdot)) | | p _ {\theta} (c) \right] = \\ \underset {c \sim q _ {\phi} (c | \cdot)} {\mathbb {E}} \left[ \sum_ {i = 1} ^ {N} \underset {z \sim q _ {\phi} (z | i)} {\mathbb {E}} \left[ \log p _ {\theta} \left(r _ {i} | z, r _ {- i}\right) \right] - \right. \\ \left. \sum_ {i = 1} ^ {N} \mathbb {D} _ {\mathrm {K L}} \left[ q _ {\phi} (z | i) | | p _ {\theta} (z | c) \right] \right] - \\ \mathbb {D} _ {\mathrm {K L}} \left[ q _ {\phi} (c | \cdot) \right\rVert p _ {\theta} (c) ] \tag {5} \\ \end{array}
$$
A.2 Dataset Pre-Processing
We selected only businesses and products with a minimum of 10 reviews, and thee minimum and maximum length of 20 and 70 words respectively, popular groups above the  $90^{th}$  percentile were removed. And each group was set to contain 8 reviews during training. From the Amazon dataset we selected 4 categories: Electronics; Clothing, Shoes and Jewelry, Home and Kitchen; Health and Personal Care.
For sequential encoding and decoding, we used GRUs (Cho et al., 2014) with 600-dimensional hidden states. The word embeddings dimension was
set to 200, and they were shared across the model (Press and Wolf, 2017). The vocabulary size was set to 50,000 most frequent words, and an extra 30,000 were allowed in the extended vocabulary, the words were lower-cased. We used the Moses' (Koehn et al., 2007) reversible tokenizer and true-caser. Xavier uniform initialization (Glorot and Bengio, 2010) of 2D weights was used, and 1D weights were initialized with the scaled normal noise  $(\sigma = 0.1)$ . We used the Adam optimizer (Kingma and Ba, 2014), and set the learning rate to 0.0008 and 0.0001 on Yelp and Amazon, respectively. For summary decoding, we used length-normalized beam search of size 5, and relied on latent code means. In order to overcome "posterior collapse" (Bowman et al., 2016) we applied cycling annealing (Fu et al., 2019) with  $r = 0.8$  for both the  $z$  and  $c$  related KL terms, with a new cycle over approximately every 2 epochs over the training set. The maximum annealing scalar was set to 1 for  $z$ -related KL term in on both datasets, and 0.3 and 0.65 for  $c$ -related KL-term on Yelp and Amazon, respectively. The reported ROUGE scores are based on F1.
The dimensions of the variables  $c$  and  $z$  were set to 600, and the  $c$  posterior's scoring neural network had a 300-dimensional hidden layer and the tanh non-linearity.
The decoder's attention mechanism used a single layer neural network with a 200-dimensional hidden layer, and the tanh non-linearity. The copy gate in the pointer-generator network was computed with a 100-dimensional single-hidden layer network, with the same non-linearity.
A.4 Human Evaluation Setup
To perform the human evaluation experiments described in Sections 5.2 and 5.2 we combined both tasks into single Human Intelligence Tasks (HITs). Namely, the workers needed to mark sentences as described in Section 5.2, and then proceed to the task in Section 5.2. We explicitly asked them to re-read the reviews before each task.
For worker requirements we set  $98\%$  approval rate,  $1000+$  HITS, Location: USA, UK, Canada, and the maximum score on a qualification test that we designed. The test was asking if the workers are native English speakers, and verifying that they correctly understand the instructions of both tasks by completing a mini version of the actual HIT.
A.5 Full Human Evaluation Instructions