SlowGuess/NAACL_2021_OCR huggingface dataset:
finally a ranking measure that evaluates the quality of the top ranked sentence (NDCG@1). The results are depicted in Table 6. The TF-IDF model has an acceptable performance but it suffers from out-of-vocabulary problem and ignores the sentence as a whole, for example, the model predicts a higher score than that of the annotators to the sentence "fantastic brilliant amazing superb good". In order to gain some understanding into what constitutes a helpful sentence, we checked the top positive and negative features of this model. We observed that the top positive words include sentiment words and product aspects. The results, however, indicate that these features are not sufficient to evaluate the helpfulness in a more fine-grained manner. The ST-RIDGE model significantly outperforms the TF-IDF model in all metrics. Finally, the BERT model is significantly better than the ST-RIDGE model in terms of MSE and Pearson correlation.
MSE	Pearson	NDCG@1
RANDOM	0.5 ± 0.026	0.018 ± 0.065	0.68 ± 0.044
TF-IDF	0.09 ± 0.006	0.63 ± 0.055	0.91 ± 0.022
ST-RIDGE	0.062 ± 0.0042	0.78 ± 0.037	0.94 ± 0.015
BERT	0.053 ± 0.0037	0.84 ± 0.022	0.95 ± 0.015
Table 6: Evaluation of Helpfulness Prediction (with confidence intervals).
Sentiment Analysis In this step, we employ the Amazon AWS comprehend sentiment analysis tool to assign each sentence a sentiment class and a score for each of the four classes: positive, negative, neutral and mixed. Sentences with a neutral or mixed classes are removed and all the rest are divided into a positive set and a negative set. The purpose of this step is twofold: first, the separation allows us to output a final sentence for both positive and negative sentiments. Second, we gain more confidence that semantically similar sentences (as measured in the downstream Similarity and Support component) have indeed the same meaning (and not the exact opposite).
Similarity and Support At this stage we aim to compute the support of each sentence, which we define as the size of the set of highly similar sentences. Formally, for a given sentence  $s_i$ , its support is  $|\{s_{j \neq i} | \text{sim}(s_i, s_j) > \sigma\}|$ , where  $\sigma$  is a predefined threshold.
8Top-10 positive features: great, sound, quality, good, excellent, price, easy, lens, recommend, perfect. Top-10 negative features: bought, review, know, don, got, amazon, gift, reviews, christmas, order.
To compute the similarity, we convert each sentence pair to the corresponding representations and compute the cosine similarity. In order to get the most accurate results, we compare several sentence representations on the semantic similarity task: the Sentence Transformers (Reimers and Gurevych, 2019), the Universal Sentence Encoder (USE) (Cer et al., 2018), FastText (Mikolov et al., 2018), and a bag-of-words representation weighted by the inverse document frequency. We find that the Sentence Transformers embeddings perform best.
To compare the methods, we sample 300,000 sentence pairs from the reviews of our 123 products, compute the similarity scores on this sample and select the top 500 pairs using each of the methods. We next consider the union of the above pairs to form a dataset of 2,035 pairs. We ask human annotators to determine if the sentences of each pair have a roughly similar meaning or not. We then calculate the precision at K (for K between 1 and 2,035) for each of the methods. As can be seen from Figure 2, Sentence-Transformers is superior to the other methods.
Finally, we derived a precision-oriented similarity score threshold  $(\sigma = 0.876)$  for Sentence Transformers that achieves a precision of  $0.9 \pm 0.286$  and a recall of  $0.46 \pm 0.022$  where the recall is estimated based on the set of 2,035 pairs.
Figure 2: Comparison of Similarity Measures.
Sentence Selection The Sentence Selection component is in charge of selecting a single sentence that is both helpful and well supported. We enforce a minimum support of 5, as we observed that such a limit increases the overall quality of the sentence and avoids surfacing esoteric opinions. After applying this threshold, we rank the remaining sentences according to the formula:  $\text{support} \times \text{helpful}^\alpha$ , where  $\alpha$  is a boosting parameter. To derive an appropriate value for  $\alpha$  we conducted another anno
tation task and obtained a value of  $\alpha = 38.8$  that gives a lot of emphasis to the helpfulness score. We describe this in detail in Section D in the Appendix.
The evaluation of our end-to-end model is challenging and does not have a natural scheme. Recall that we do not restrict our input to small random samples of the review set, as commonly done in review summarization, and was shown to produce biased results (Shapira and Levy, 2020). Instead, we allow for dozens or hundreds of reviews per product. Thus, we cannot expect annotators to carefully read the full input before choosing an RHS. Nonetheless, we show that our notion of helpfulness is indeed useful for surfacing important review content by comparing our models to previous summarization works in two different settings.
Single Review Summarization In this evaluation we only consider the helpfulness component, as means to create an extractive summary comprised of a single sentence.
Abstractive single review summarizers (Ma et al., 2018; Isonuma et al., 2019; Wang and Ren, 2018) are not suitable for comparison as these works are trained on header-like summaries of 4.36 words on average, much shorter than our extractive, one-sentence output. Instead, we consider the unsupervised single document summarization algorithm Textrank $^{9}$  (Mihalcea and Tarau, 2004). Textrank, which is extractive and can output any number of sentences, is a viable candidate for comparison as our goal is not to achieve SOTA results on this task, but rather to demonstrate that the helpfulness model can produce good extractive summaries without being trained on reference summaries.
We selected a sample of 300 reviews, in which the prediction of the two algorithms differed (the output was exactly the same on  $28\%$  of the reviews), and asked crowd workers to rate each of the selected sentences in a 5-level scale according to how helpful the selected sentence was for a purchase decision (our objective) and according to how well the selected sentence summarized the review (the traditional objective). Each sentence was annotated by 5 workers, where the sentences of the two algorithms appeared one next to the other but in random order. Table 7 summarizes the results,
<sup>9</sup>We used the implementation of https://pypi.org/project/sumy/
showing our method is superior in both aspects. $^{10}$
Helpfulness	Summarization
Mean	Std	Mean	Std
Helpful Sentence	3.41	1.11	3.34	1.05
Textrank	3.28	1.16	3.27	1.10
End-To-End Evaluation Our complete model resembles the task of Multi-Document Summarization (MDS) which ideally consumes the entire set of reviews related to a specific product and outputs a single summary or a single sentence in our case. In practice, MDS is applied to document sets of relatively small sizes, which significantly reduces the potential impact of our Similarity and Support sub-component. In order to put our evaluation in context of prior work, we evaluate our model with two minor modifications tailored for small review sets: we relax the similarity threshold to 0.75 and remove the minimal support constraint. We only consider the positive sentences in this evaluation, as the majority of the reviews are positive.
We use the dataset published in Bražinskas et al. (2020b) which covers 60 products from 4 different categories (Cloth, Electronics, Health Personal Care and Home Kitchen)[11] of which only 1 category is included in our own data. Each product has 8 reviews and 3 reference summaries written by humans. We evaluate our model in a straight forward manner by comparing the sentences selected by our model to sentence rankings provided by humans.
We ask expert annotators (one annotator per example) to read the reviews and rate each sentence from the reviews on a scale of 1 to 5. A score of 1 means that the sentence does not help to make a purchase decision or it does not reflect the overall theme of the reviews where a score of 5 means that the sentence is both helpful and aligns well with the common opinions expressed in the reviews. The mean score of the top sentence for each product is 4.31, which means that even for products with only 8 reviews it is common to find a sentence that is both helpful and supported by the reviews. We evaluate our model by averaging NDCG@K over all products for  $K \in \{1, 10\}$ . We compare the
performance of our model with two baselines: ranking the sentences in a random order and from the longest to the shortest. Our method outperforms the baselines by a large margin, see Table 8.
Table 7: Single Document Summarization - comparison to TextRank.
K=1	K=10
Our Model	0.87	0.94
From Longest to Shortest	0.60	0.68
Random	0.54	0.62
For the sake of completeness we also report the common MDS evaluation metric, ROUGE (Lin, 2004), which does not fully suit our setting, as it is based on n-gram comparisons between the output and golden summaries written by humans, which are typically much longer than a single sentence. In Table 9 we compare the ROUGE scores of 3 sentence selection variants: our model, a random sentence and an Oracle, i.e., the sentence that maximizes the ROUGE-L score. We also report the results of Copycat (Bražinskas et al., 2020b),[12] a state-of-the-art review MDS model. We note that Copycat is not truly comparable to our model due to the significantly different summary length requirement (in this dataset an average sentence contains 74 characters while an average reference summary contains 293 characters). Note, however, that in terms of precision, which is what we aim for with such an extreme "summary", the RHS is almost as good as the Oracle and much better than Copycat.
Table 8: Mean NDCG@K score.
Random	RHS	Oracle	Copycat
Rouge-1 f	0.127	0.166	0.250	0.297
Rouge-1 p	0.329	0.420	0.440	0.247
Rouge-1 r	0.084	0.109	0.185	0.386
Rouge-2 f	0.014	0.028	0.054	0.055
Rouge-2 p	0.045	0.084	0.106	0.045
Rouge-2 r	0.008	0.018	0.039	0.073
Rouge-L f	0.094	0.120	0.177	0.192
Rouge-L p	0.251	0.309	0.320	0.160
Rouge-L r	0.061	0.078	0.129	0.249
Table 9: ROUGE scores.
Examples of RHS We pick two examples from Bražinskas et al. (2020a), depicted in Table 10, and use our model to extract a single sentence for each. Each of the examples consists of 8 reviews and a reference summary written by a
12Results are based on our own computation using https: //pypi.org/project/py-rouge/
10The results are statistically significant using a 1-tail paired t-test, with a p-value of  $1.05E - 06$  for helpfulness and 0.005 for summarization. 14 of the products in this dataset are no longer available on amazon.com and we omitted them from the evaluation.
Reviews from Yelp!
RHS	They have ridiculous price also bad customer services.
Human	Bennett Medical has poor customer service. Phone calls can take a long time to get answered and leaving voice mails tend to be fruitless. The products are overpriced and take a long time to be refilled. Using this medical supply company can be a hassle.
Summarizer	This medical supply is the worst medical supply company in the valley. The customer service is horrible, the staff is rude, the wait times are long, and the service reps are not helpful at all. Do not recommend this company to anyone.
Reviews from Amazon.com
RHS	These thongs are fun, festive, flexible and surprisingly comfortable.
Human	These are a very comfortable and cute sandal. This thong sandal goes with a variety of outfits and the mushy sole allows for all day comfort. However, they do run a little small, sizing up provides a better fit. Overall, a reasonably priced shoe that will last for years to come.
Summarizer	These sandals are very cute and comfortable. They fit true to size and are very comfortable to wear. They look great with a variety of outfits and can be dressed up or down depending on the occasion.
Table 10: Helpful Sentences vs. Abstractive Summarization.
human.13
The first example deals with Yelp reviews and our model seems to do well even though it was only trained on product reviews from Amazon. While our extracted sentence is less elaborative compared to the human and abstractive summaries, it gives enough information to make a decision. Note also, that the abstractive summary does not refer to the high pricing. As for the second example, while not covering all aspects of the product, the helpful sentence is faithful to the reviews and aligns with the overall sentiment. The summarizer, on the other hand, contradicts the reviews with regarding to the sandals size.
Recall that these examples are constructed from 8 reviews only, while our model benefits considerably from large number of reviews, which is often the case for popular products. This is due to the greater sentence variety it can choose from and the fact that the support becomes more meaningful as more reviews are available. See Section E in the Appendix for additional examples and some statistics of our model outputs.
In this paper we address the challenge of summarizing product reviews with limited space, like when using a virtual assistant. We define a new notion that fits the needs of this setting, a representative
helpful sentence, and propose a new task accordingly: given a set of product reviews, extract a sentence that is both helpful for a purchase decision and well supported by the opinions expressed in the reviews.
As a first step, we collect and annotate a new dataset of review sentences with their helpfulness scores, and make this dataset available to facilitate further research. Next, we develop an end-to-end model for surfacing representative helpful sentences. Our model combines several necessary components which are optimized for our goal. In order to get a feeling for the performance of our model, we compare our results to summarization tasks that are similar in nature, and show that our model performs better in the aspects we target.
In this work, we make use of customer reviews published on Amazon.com. The reviews must comply with Amazon's Community Guidelines<sup>14</sup> which prohibit offensive, infringing, or illegal content. Amazon encourages anyone who suspects that content manipulation is taking place or that its Guidelines are being violated to notify Amazon. Amazon investigates concerns thoroughly and takes appropriate actions, including removal of reviews that violate these Guidelines, including reviews that contain hatred or intolerance for people on the basis of race, ethnicity, nationality, gender or gender identity, religion, sexual orientation, age, or disability. Among other things, Amazon has a broad license to use, reproduce, publish, and create derivative works from the customer reviews on Amazon.com. The authors of this paper are employees of Amazon and are authorized to use customer reviews in this work.
A small sample of annotated review sentences is released for research purposes according to the provided license.[15] Annotations were conducted by a service provider pursuant to a Service Agreement with Amazon. Under that Service Agreement, the service provider represents and warrants that it complies with all applicable laws, regulations, and ordinances when performing those services.
<sup>13</sup>We only show the summaries, the complete set of reviews are available in Brazinskas et al. (2020a).
14https://www.amazon.com/gp/help/ customer/display.html?nodeId= GLHXEX85MENUE4XF
15https://cdla.dev/sharing-1-0/
Hana Almagrabi, Areej Malibari, and John McNaught. 2018. Corpus analysis and annotation for helpful sentences in product. Computer and Information Science, 11(2).
Stefanos Angelidis and Mirella Lapata. 2018. Summarizing opinions: Aspect extraction meets sentiment prediction and they are both weakly supervised. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3675-3686.
Arthur Bražinskas, Mirella Lapata, and Ivan Titov. 2020a. Few-shot learning for opinion summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4119-4135.
Arthur Bražinskas, Mirella Lapata, and Ivan Titov. 2020b. Unsupervised opinion summarization as copycat-review generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5151-5169.
Giuseppe Carenini, Jackie Chi Kit Cheung, and Adam Pauls. 2006. Multi-document summarization of evaluative text. Comput. Intell., 29:545-576.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder. CoRR, abs/1803.11175.
Eric Chu and Peter Liu. 2019. Meansum: a neural model for unsupervised multi-document abstractive summarization. In International Conference on Machine Learning, pages 1223-1232.
Alexander Philip Dawid and Allan M Skene. 1979. Maximum likelihood estimation of observer error rates using the em algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), 28(1):20-28.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Gerardo Ocampo Diaz and Vincent Ng. 2018. Modeling and prediction of online product review helpfulness: a survey. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 698-708.
Shima Gerani, Yashar Mehdad, Giuseppe Carenini, Raymond T. Ng, and Bita Nejat. 2014. Abstractive summarization of product reviews using discourse structure. In EMNLP.
Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth
ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168-177.
Masaru Isonuma, Junichiro Mori, and Ichiro Sakata. 2019. Unsupervised neural single-document summarization of reviews via learning latent discourse structure and its ranking. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2142-2152.
Soo-Min Kim, Patrick Pantel, Tim Chklovski, and Marco Pennacchiotti. 2006. Automatically assessing review helpfulness. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, EMNLP '06, page 423-430, USA. Association for Computational Linguistics.
Kevin Lerman, Sasha Blair-Goldensohn, and Ryan T. McDonald. 2009. Sentiment summarization: Evaluating and learning user preferences. In EACL.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Jingjing Liu, Yunbo Cao, Chin-Yew Lin, Yalou Huang, and Ming Zhou. 2007. Low-quality product review detection in opinion summarization. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 334-342, Prague, Czech Republic. Association for Computational Linguistics.
Shuming Ma, Xu Sun, Junyang Lin, and Xuancheng Ren. 2018. A hierarchical end-to-end model for jointly improving text summarization and sentiment classification. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI'18, page 4251-4257. AAAI Press.
Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing, pages 404-411.
Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018).
Susan M. Mudambi and David Schuff. 2010. Research note: What makes a helpful online review? a study of customer reviews on amazon.com. MIS Quarterly, 34(1):185-200.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807.
Jahna Otterbacher. 2009. "helpfulness" in online communities: A measure of message quality. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '09, page 955-964, New York, NY, USA. Association for Computing Machinery.
Yue Pan and Jason Q. Zhang. 2011. Born unequal: A study of the helpfulness of user-generated product reviews. Journal of Retailing, 87(4):598 - 612.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Ori Shapira and Ran Levy. 2020. Massive multi-document summarization of product reviews with weak supervision.
Rion Snow, Brendan O'connor, Dan Jurafsky, and Andrew Y Ng. 2008. Cheap and fast—but is it good? evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 conference on empirical methods in natural language processing, pages 254-263.
Jiaxing Tan, Alexander Kotov, Rojiar Pir Mohammadiani, and Yumei Huo. 2017. Sentence retrieval with sentiment-specific topical anchoring for review summarization. Proceedings of the 2017 ACM on Conference on Information and Knowledge Management.
Oren Tsur and Ari Rappoport. 2009. Revrank: A fully unsupervised algorithm for selecting the most helpful book reviews. In ICWSM.
Hongli Wang and Jiangtao Ren. 2018. A self-attentive hierarchical model for jointly improving text summarization and sentiment classification. In *Asian Conference on Machine Learning*, pages 630–645.
Wenting Xiong and Diane Litman. 2014. Empirical analysis of exploiting review helpfulness for extractive summarization of online reviews. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1985-1995, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.
Yinfei Yang, Yaowei Yan, Minghui Qiu, and Forrest Bao. 2015. Semantic analysis and helpfulness prediction of text for online product reviews. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 38-44, Beijing, China. Association for Computational Linguistics.
Naitong Yu, Minlie Huang, Yuanyuan Shi, and Xiaoyan Zhu. 2016. Product review summarization by exploiting phrase properties. In Proceedings of
COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1113-1124. The COLING 2016 Organizing Committee.
Feng Zhu and Xiaoquan Zhang. 2010. Impact of online consumer reviews on sales: The moderating role of product and consumer characteristics.
Figure 3 displays a single annotation task as presented to the annotators.
Canon 430EX Speedlite Flash for Canon EOS SLR Cameras - Older Version
Figure 3: A single annotation as presented to the annotators.
Review Sentence this flash is a superb
How helpful is this sentence for a purchase decision? (required)
$\bigcirc$  Very helpful
Somewhat helpful
Not helpful
Helpfulness Distribution within the Train Set Figure 4 depicts the helpfulness distribution within the train set.
Figure 4: Helpfulness distribution among the train set. The green, dotted line represents the expected distribution if the annotators would have chosen an answer uniformly at random. The most frequent helpful score on the training set is 1.3.
B Annotation Analysis
Table 11 provides some statistics for the helpful and unhelpful subsets.
# Helpful Votes	# Reviews	# Sentences	Mean Sentences
Helpful Set	≥ 50	101	5032	49.82
Unhelpful Set	0	3803	22030	5.79
Table 11: Contrasting Sentence Helpfulness Score with Review Helpfulness votes — Helpful and unhelpful sets statistics.
C Analysis of Helpful Sentences