SlowGuess/NAACL_2021_OCR huggingface dataset:
Length Sometimes a sentence may be long but provide information that is not very helpful for customers. As an example for a long sentence with low helpfulness score, consider the sentence "As engineers, you must learn to handle the technical aspects, the social and political gamesmanship, the public reactions to our work and the daunting challenge of staying at pace with rapid developments in our fields" that was taken from a review about the book Mastery which deals with becoming a master in one's field. Indeed, this sentence is long and informative but it talks about engineers while the book itself is meant for everyone interested in mastery, not necessarily engineers.
Sentiment Consider the following sentence, demonstrating that sentiment is not always necessary for helpfulness: "It teaches parents to use twelve key strategies in everyday scenarios to help them in providing experiences to promote healthy connections in the brain." This sentence was deemed helpful by the annotators but does not express any sentiment, it merely states a fact about the product.
D Surfacing Representative Helpful Sentences
Preprocessing First, HTML markup is removed from each review using the BeautifulSoup<sup>16</sup> package and then the cleaned texts are split into sentences using the Spacy<sup>17</sup> package. Next, sentences with character length outside the range of [30, 200] are removed. We chose these thresholds based on manual inspection under the assumption that extremely short sentences will not be very helpful for customers while extremely long sentences will result in a frustrating customer experience (especially in the scenario of voice interactions). In the movies domain, for example, a typical short sentence would state that "This movie was really good."
or that "It's a must see film.", statements that do not contribute much on top of the star rating. In our dataset, long sentences are quite rare while short sentences, on the other hand, are more common and form  $10\%$  of the sentences.
Sentence Selection The Sentence Selection component is in charge of selecting a single sentence that is both helpful and well supported. We enforce a minimum support of 5, as we observed that such a limit increases the overall quality of the sentence and avoids surfacing esoteric opinions. After applying this threshold, we rank the remaining sentences according to the formula:
$$
\operatorname {s u p p o r t} \times \operatorname {h e l p f u l} ^ {\alpha} \tag {1}
$$
where  $\alpha$  is a boosting parameter. To derive an appropriate value for  $\alpha$  we conducted another annotation task in which annotators were asked again to score the helpfulness of the sentences presented to them. This time we consider all the sentences that are not dominated by any other sentence, i.e. we consider sentence  $s$  if and only if there is no sentence  $s'$  such that both helpful  $(s') > \text{helpful}(s)$  and support  $(s') > \text{support}(s)$ , in other words, we asked to annotate all the sentences from the Pareto front with respect to helpfulness and support. Each sentence was joined with a prefix that quantifies its support as in 20 customers agreed that: has very good pic quality and extremely easy to use. We optimized Formula 1 by minimizing the Kullback-Leibler divergence between the score distribution (softmax) from the annotators and the score distribution from the formula (softmax) and obtained  $\alpha = 38.8$ . While this value may seem enormous at a first glance we note that the helpfulness score obtained from our model for the sentences in the Pareto set tend to be very close to each other while their support may vary considerably. To put this number into proportion, consider two sentences with support and helpfulness (20, 1.5) and (10, 1.52) respectively, then the final score of the first sentence will only be slightly better than the final score of the second sentence (which is the expected behavior as its support is twice as large).
Interestingly, the experiment confirmed our hypothesis that customers perceive highly supported sentences as more helpful compared to the case when no support information is given.[18]
18We experimented with another prefix, that only states "one customer thought that".
$^{16}$ https://pypi.org/project/beautifulsoup4/  
 $^{17}$ https://spacy.io/
E Model Output Statistics
As for end-to-end results on our 123 products, our model found a positive helpful sentence for 114 products, and a negative helpful sentence for 16 products. The low coverage for negative helpful sentences might be explained by our choice to concentrate on popular products (having more than 100 reviews) which are probably of better quality than random products.
In Table 12 we present the selected sentence, its helpfulness score and its top supported sentences along with their similarity scores for 3 products.
It was easy to put together, is the perfect height, and very durable.	1.6
It was easy to put together and is sturdy.	0.94
Sturdy and easy to put together.	0.92
Also, it was very easy to put together.	0.92
It's sturdy and cleans easily.	0.91
Pretty sturdy, too, and easy to handle.	0.91
They fit well and produce good sound quality.	1.7
fits great and has great sound	0.93
The sound is pretty good as well.	0.93
Great Pair works great with excellent sound	0.93
They do have good sound, when I can keep them in.	0.91
These sound great and fit very well.	0.9
Quality good, price okay, sound output great.	1.7
For the price point, this set delivers good sound.	0.92
Very good sound for the cost!!	0.92
It works and the sound quality is good.	0.9
Good quality speakers that provide great sound.	0.9
For me ... the sound was very good.	0.88
Table 12: The selected sentence, its helpfulness score and its top supported sentences along with their similarity scores for 3 products.