SlowGuess/NAACL_2021_OCR huggingface dataset:
Iftah Gamzu<sup>1</sup> Hila Gonen<sup>1</sup> Gilad Kutiel<sup>1</sup> Ran Levy<sup>1</sup> Eugene Agichtein<sup>2,3</sup>
{iftah,gonenhi,gkutiel,ranlevy,eugeneag}@amazon.com
$^{1}$ Amazon, Tel-Aviv, Israel
$^{2}$ Amazon, Seattle, WA, USA
<sup>3</sup>Emory University, GA, USA
In recent years online shopping has gained momentum and became an important venue for customers wishing to save time and simplify their shopping process. A key advantage of shopping online is the ability to read what other customers are saying about products of interest. In this work, we aim to maintain this advantage in situations where extreme brevity is needed, for example, when shopping by voice. We suggest a novel task of extracting a single representative helpful sentence from a set of reviews for a given product. The selected sentence should meet two conditions: first, it should be helpful for a purchase decision and second, the opinion it expresses should be supported by multiple reviewers. This task is closely related to the task of Multi Document Summarization in the product reviews domain but differs in its objective and its level of conciseness. We collect a dataset in English of sentence helpfulness scores via crowd-sourcing and demonstrate its reliability despite the inherent subjectivity involved. Next, we describe a complete model that extracts representative helpful sentences with positive and negative sentiment towards the product and demonstrate that it outperforms several baselines.
Customer reviews are known to be a valuable source of information for potential buyers. This is evident from the high engagement of customers with reviews, for example by up-voting a review for its helpfulness.<sup>1</sup> As online shopping platforms attract more traffic it is becoming increasingly difficult to consume the wealth of information customers share. For this reason, helpful reviews (defined as such by costumers) are made more visible than those that are less helpful.
The topic of review helpfulness has attracted a lot of academic interest in which reviews were always considered as a whole (see Diaz and Ng (2018) for a survey). However, in some scenarios, such as the limited real-estate in mobile screens, or in voice interactions with a virtual assistant, presenting a full review is impractical and the need to automatically extract helpful excerpts arises. While in the mobile scenario, a persistent customer may still be able to read the entire review, the voice scenario is inherently more challenging as it demands patience and focus from the customer, while the assistant reads the text out loud. As a result, the need for extreme brevity and the ability to understand what matters most to customers becomes crucial.
In addition to brevity and helpfulness, another desirable property from the extracted content is being faithful to the reviews as a whole. Indeed, a costumer looking for relevant and helpful reviews, often interacts with more than one review before making their decision, trying to pinpoint those helpful bits of information that are shared by multiple reviewers. This process is tedious because of the sheer amount of reviews and biased because of the order they appear in. A system that aims to replace this process while maintaining trust in the content it provides should be able to extract concise helpful texts that repeat across multiple reviews, indicating that they are faithful to the reviews' content (from here onward we shall refer to such texts as "faithful").
Our goal is to extract such sentences, i.e., sentences that are both helpful for a purchase decision and faithful. To this end, we first define two new notions: A Helpful Sentence is a sentence which is considered helpful by the average customer in their purchase decision process. A Representative Helpful Sentence (RHS) is a helpful sentence that is also highly supported, that is, the ideas it expresses appear in multiple reviews for the given product (not necessarily in the exact same wording).
<sup>1</sup>At the time of writing this paper, a review for the Echo Dot 3rd generation received more than 10K votes.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 678-691 June 6-11, 2021. ©2021 Association for Computational Linguistics
It is traditionally assumed that judging the importance of a text excerpt requires reading the entire text. We challenge this assumption, at least in the domain of product reviews, and collect a dataset of single review sentences with their helpfulness scores by averaging the scores assigned to them by multiple crowd workers. We show that despite the highly subjective nature of this task, and despite the fact that workers are exposed to sentences without their surrounding context, the resulting scores are reliable. Using the data we collected, from 6 different categories, ranging from Electronics to Books, we train and evaluate several supervised algorithms to predict helpfulness score, which achieve promising results. Finally, we present an initial implementation of a model that given a set of product reviews, extracts a single positive RHS (supports the purchase) and a single negative RHS (opposes the purchase).
In summary, the main contributions of this work are: (1) We propose a novel task that given a set of reviews for a product, outputs a single sentence that is both helpful for a purchase decision and supported by multiple reviewers; (2) We show that the helpfulness of a sentence can be reliably rated based solely on the sentence, allowing for an efficient dataset creation. These helpfulness scores can be leveraged for other tasks such as highlighting important parts of a review; (3) We publish a novel dataset of sentences taken from customer reviews along with their helpfulness score; (4) We develop an end-to-end model for our task that shows promising results and outperforms several baselines.
Review Helpfulness Modeling and Prediction
Customer reviews are a valuable source of information for customers researching a product before making a purchase (Zhu and Zhang, 2010). Diaz and Ng (2018) survey recent work on the tasks of modeling and predicting review helpfulness. While some researchers treat helpfulness votes as groundtruth, others have argued that these votes are not good indicators for actual review helpfulness (Liu et al., 2007; Tsur and Rappoport, 2009; Yang et al., 2015).
Some general observations have been made based on helpfulness votes, e.g., review length has
been shown to be strongly correlated to helpfulness (Kim et al., 2006; Liu et al., 2007; Otterbacher, 2009; Mudambi and Schuff, 2010; Pan and Zhang, 2011; Yang et al., 2015). Another widely-agreed indication for review helpfulness is the review star rating (Kim et al., 2006; Mudambi and Schuff, 2010; Pan and Zhang, 2011).
A related dataset was presented in Almagrabi et al. (2018). The main advantages of the dataset we create over this previously suggested one are: (1) Binary vs. continuous scores - We use continuous scores rather than binary scores. Our aim is to surface the most helpful sentences, which is not possible if many of the sentences are annotated as equally helpful; (2) Range of products/domains - The previous dataset includes only 5 products, all from the Electronics domain. Our dataset is significantly more diverse, providing annotations for 123 products from 6 different domains, allowing to evaluate a model's ability to generalize across domains.
Product Review Summarization The most common approach for product review summarization, which centers the summary around a set of extracted aspects and their respective sentiment, is termed aspect based summarization. One of the early abstractive works, by Hu and Liu (2004), was designed to output lists of aspects and sentiments. Other works target a traditional summarization output and at times somewhat simplify the task by assuming aspects or seed words are provided as input (Gerani et al., 2014; Angelidis and Lapata, 2018; Yu et al., 2016). Recently advances were made on unsupervised abstractive reviews summarization, by leveraging neural networks (Chu and Liu, 2019; Bražinskas et al., 2020b) followed by a few shot variant (Bražinskas et al., 2020a).
Extractive summarization include earlier works such as Carenini et al. (2006); Lerman et al. (2009) and Xiong and Litman (2014) who suggested to use review helpfulness votes as means to improve the content extraction process. More recently, Tan et al. (2017) suggested a novel generative topic aspect sentiment model.
3 Representative Helpful Sentences
Task Definition In this work, we focus on summarization of reviews in the setting of shopping over voice with the help of a virtual assistant. Our goal is to provide users with content that is both helpful and faithful in this challenging setting
2The dataset is available at https://registry. opendra-dataaws/.
where the information the user can absorb is extremely limited. First, we aim to maximize the informativeness, while maintaining brevity. To this end, we introduce a new notion of helpful sentences - sentences which the average costumer will consider as helpful for making a purchase decision. Next, to ensure faithfulness, we introduce the notion of support for a given sentence - the number of review sentences with a highly similar content. We seek to automatically identify a helpful sentence with a wide support, which we term representative helpful sentence (RHS). Note that Representative Helpful Sentences, being supported by many similar sentences, are by construction faithful to the review pool from which they are extracted. We restrict ourselves to single sentences that are extracted as-is from product reviews, as this serves as another mechanism to ensure faithfulness. We do not restrict the number of reviews in the input. Table 1 presents a few helpful sentences for example, as extracted by our model (see Section 5).
Product	Representative Helpful Sentence
Toy Pirate Ship	It was easy to put together, is the perfect height, and very durable.
Headphones	They fit well and produce good sound quality.
Speakers	Quality good, price okay, sound output great.
Our task resembles the well known (extractive) customer review summarization task (Hu and Liu, 2004) but differs in several important aspects. First, its output is very concise due to the extreme space constraint, resembling the extreme summarization task (Narayan et al., 2018), which however, deals with news articles and outputs an abstractive summary. In our application there is low tolerance for factually incorrect summaries, so we choose extraction over abstraction. Second, we do not restrict the system's output to aspect based opinions, as we find that sometimes factual content may also be quite helpful. Third, while traditional summarization systems favor information that appears frequently in the source documents, we target information that is both frequent and helpful.
Subjectivity As mentioned above, review helpfulness scores are derived from votes of actual customers. Deciding on whether or not to up-vote a review is a subjective decision as different customers may value different product qualities. However, the
underlying assumption of the voting mechanism is that reviews with many up-votes are indeed helpful for the average customer. Restricting the user to a single sentence makes matters even more challenging as it cannot possibly discuss all the product merits and shortcomings. To emphasize the subjectivity involved in assigning a helpfulness score for a standalone sentence, consider the examples in Table 2. The first example may be helpful for parents looking to buy a book for their children but entirely unhelpful for adults who wish to purchase the book for themselves. Similarly, the second one is more helpful to readers of extreme height (short or tall) than to those of medium height.
Table 1: Example Representative Helpful Sentences.
Product	Sentence
Harry Potter book	Finding 1 book that keeps your child intrigued and helps him or her develop a love for reading is amazing.
Jump rope	It's a pretty standard jump rope but it's really nice and you can adjust the length which is perfect because I'm really short.
Table 2: Review sentence examples.
Despite the evident subjectivity, we assume that there exists an "average helpfulness" score for every sentence, which can be estimated by averaging the ratings of multiple crowd workers. In the following section we establish this assumption by compiling a new dataset of sentences along with their helpfulness scores, and showing quantitatively that the annotations in our dataset are consistent and reliable.
4 Helpful Sentences Annotation
Our main contribution in this work lies in the notion of helpful sentences and the ability to identify such sentences without observing entire reviews. In what follows, we describe the process of compiling a dataset of sentences along with their helpfulness scores using crowdsourcing. Note that this dataset is intended solely for scoring helpfulness of sentences. Faithfulness is ensured by other means which are not reflected in the dataset, i.e. by requiring a RHS to have a wide support of similar sentences, as discussed in section 3 and implemented in our model, as described in Section 5.
4.1 Annotation Task
We consider a subset of 123 products arbitrarily selected from the Amazon.com website, so that
each has at least 100 customer reviews and they (approximately) equally represent 6 different categories (Toys, Books, Movies, Music, Camera and Electronics). We started with 45,091 reviews, split them into 210,121 sentences and randomly selected a train set with 20,000 sentences, and a test set with 2,000 sentences. We asked annotators to rate each sentence according to how helpful it is for reaching a purchase decision, using the Appen platform. Ratings were provided on a 3-level scale of Not Helpful (0), Somewhat Helpful (1), or Very Helpful (2). The final helpfulness score of a given sentence was set to the average rating. See Section A in the Appendix for more details on the annotation task guidelines.
Each example was rated by 10 different annotators in the training set and 30 different annotators in the test set. Initial experiments revealed that 10 annotations per sentence, while noisy, are still sufficient to train a model. We observed that the number of annotators used to calculate the test set affects the evaluation. This is due to the subjective nature of this task and the observed helpfulness score that becomes closer to its "real" score as the number of votes collected for each sentence increases. Table 3 demonstrates the effect the number of annotators used to rate each example in the test set has on the final evaluation. It shows that after fixing the model and predictions, the evaluation score (Pearson correlation in this case) increases as we average more votes. From our experience, there is no gain beyond 30 votes per sentence for this particular task.
# of votes	1	10	20	25	30
Pearson	0.523	0.776	0.822	0.831	0.838
We observe a skewed helpfulness distribution with a fairly high mode of 1.3 which shows that the raters did not provide random answers. Furthermore, under the assumption that most review authors aim for their reviews to be helpful, we should expect a distribution that is skewed towards higher scores. See Section A in the Appendix for a depiction of the helpfulness distribution within the train set.
Table 4 presents the most helpful sentence, a sentence that is somewhat helpful (with a median score) and the least helpful sentence from the test set for particular headphones as perceived by the annotators.
Table 3: For a fixed prediction the correlation between the prediction and the scores obtained by averaging individual scores increases as we consider more votes per sentence. The phenomenon is not unique to correlation.
Sentence	Helpfulness
Really great headphones, especially for $25, but honestly, they sound better than my gaming headset and my DJ headphones in many respects.	1.97
Call quality just can’t be beat for ear buds	1.47
Any thoughts from others?	0
Table 4: The most helpful, a somewhat helpful and the least helpful sentences for particular headphones.
4.2 Annotation Analysis
As mentioned earlier, rating sentence helpfulness is a highly subjective task, and some disagreement is expected. Nevertheless, we argue that the data we collected is reliable and demonstrate it through the three following experiments.
Inter-annotator Agreement We compute agreement in the spirit of the analysis performed in (Snow et al., 2008). For each annotator, we restrict the data to the set of rows that they completed and compute the Pearson correlation between their answers against the average of all other annotators. Finally, we take the average across all annotators after removing the worst  $10\%$  annotators according to the method of (Dawid and Skene, 1979). We get an average of  $0.44 \pm 0.01$  Pearson correlation on the train set (10 annotators per row) and  $0.57 \pm 0.02$  on the test set (30 annotators per row), which demonstrates good agreement given the subjective nature of this task. We also randomly split the annotators into two disjoint sets and calculated the correlation between the corresponding scores. There was a correlation of 0.49 for the train set and 0.81 for the test set.
Internal Consistency A necessary condition for ensuring reliability is that similar sentences get similar helpfulness scores. We verify that our crowdsourced test data meets this requirement by measuring the standard deviation of the helpfulness scores within groups of similar sentences. We use
4This scores are comparable, for example, with the scores reported in Snow et al. (2008) for the highly subjective Affective Text Analysis task.
3www.appen.com
the sentence-transformers embeddings of Reimers and Gurevych (2019) which were optimized for computing semantic similarity. For each sentence in the test set, we construct its semantic neighborhood by grouping together all sentences with high similarity. For each non-singleton group, we measure the standard deviation of the helpfulness score and compare it with the standard deviation of a similarly sized group of random sentences from the test set. We expect to get a tighter distribution of helpfulness scores within the similarity groups (compared to the random groups) if the data is internally consistent. Indeed, we found 217 groups with an average standard deviation of 0.16 while the average standard deviation of the corresponding random groups was 0.29.5
Sentence Helpfulness vs. Review Helpfulness
As the third and final reliability analysis, we compare the crowd helpfulness scores with review helpfulness votes taken from the Amazon.com website. We consider reviews for the 123 products selected earlier, and extract two subsets. The first (the helpful set) is the set of all reviews with at least 50 helpful votes. The second (the unhelpful set) is the set of all reviews with no helpful votes. See Section B in the Appendix for statistics on the two subsets. We randomly select 500 sentences from each set and collect crowd helpfulness ratings. For each set we calculate the mean helpfulness score and the ratio of sentences with helpfulness score greater than 1 and 1.5 respectively. Table 5 shows the results which demonstrate a higher mean helpfulness score in the helpful set.<sup>6</sup>
Mean Score	Ratio Score > 1	Ratio Score > 1.5
Helpful Set	1.21	70%	18%
Unhelpful Set	1.15	64%	14%
Table 5: Contrasting Sentence Helpfulness Score with Review Helpfulness votes — Results.
These results indicate that helpful reviews tend to include more helpful sentences on average. However, as can be expected, the differences are not dramatic. Looking at the average length of reviews
sheds some more light on the differences: a helpful review is almost 10 times longer than a non helpful review on average. This means that in order for a review to be helpful it must provide details, a requirement that a single sentence simply cannot meet. Therefore, we conjecture that a helpful sentence captures the most essential statements made in the review while a helpful review is one that includes details and justifies its rating.
4.3 Analysis of Helpful Sentences
A brief examination of the crowd-sourced data reveals two sentence characteristics that contribute to the helpfulness of a sentence: the length of the sentence and the sentiment, which is more strongly correlated with helpfulness.
Length The Pearson correlation between the length (in characters) and the helpfulness score on the test set is 0.37. This correlation is expected, since longer sentences can potentially convey more information and thus tend to be more helpful.
Sentiment We use Amazon AWS comprehend sentiment analysis tool to classify each sentence into one of four sentiment classes: positive, negative, neutral and mixed. We got a negative Pearson correlation of  $-0.53$  between the helpfulness scores of the sentences and the scores assigned to the neutral class. To better understand this relationship, we define a helpful sentence as one with score greater or equal to 1.5 and a sentence with sentiment as one that is not in the neutral class, and estimate two conditional probabilities:
$$
P (\text {H e l p f u l} \mid \text {S e n t i m e n t}) = 0. 1 5
$$
$$
P (\text {S e n t i m e n t} \mid \text {H e l p f u l}) = 0. 6 8
$$
This shows that having sentiment is an important condition for a sentence to be helpful, but it is not a sufficient condition. We indeed observed that sentences with sentiment that do not provide additional reasoning or details do not get high helpfulness scores. Some related examples from reviews can be found in Section C in the Appendix.
5 Surfacing Representative Helpful Sentences
We now turn to create an end-to-end model for surfacing representative helpful sentences (RHS): given a set of reviews for a certain product, we aim
<sup>7</sup>https://aws.amazon.com/comprehend/
5The differences were statistically significant with a pvalue of  $7E - 20$  using a paired two-tailed t-test. 6The difference is statistically significant with a p-value of approximately 0.0079 using a t-test with an equal variance assumption as well as t-test with different variance assumption, a.k.a Welch's t-test.
Figure 1: High-level overview of our model.
to output a single RHS with positive sentiment and a single RHS with negative sentiment. Figure 1 depicts the different sub-components of our model. Given a set of reviews, we preprocess the input and predict helpfulness scores for each of the sentences. Next, we analyze the sentiment of each sentence and separate into positive and negative sets. Following that, the support of each sentence is determined, and finally we select the RHS sentence based on its helpfulness score and its support. In what follows, we describe each of the components in details.
Preprocessing We remove HTML tags and split the cleaned reviews into sentences. The sentences are then filtered by removing sentences of extreme length (both short and long). See Section D in the Appendix for additional details.
Helpfulness Estimation This component assigns a helpfulness score for each sentence and removes all sentences with score below 1. This filtering serves two purposes: First, it ensures that we do not output any sentence in case there is no helpful sentence in the product reviews. Second, it reduces the runtime of the downstream Similarity and Support component which is quadratic in the number of sentences.
We experiment with three helpfulness models and find that a pre-trained BERT (Devlin et al., 2018) fine-tuned on our training data performs best. The two other models we compare are: (1) TF-IDF: a model that treats each sentence as a bag-of-words. We use TfidfVectorizer from the sklearn package to convert each sentence into a vector and then fit a Ridge regression model on top of it; (2) ST-RIDGE: a model that fits a Ridge regression on top of the Sentence-Transformers embedding (Reimers and Gurevych, 2019).
We use 3 measures for evaluation: Mean Squared Error (MSE), which is the traditional measure for regression, Pearson correlation between the predicted score and the ground-truth score, and