SlowGuess/ACL_2020_OCR huggingface dataset:
Jordan J Louviere, Terry N Flynn, and Anthony Alfred John Marley. 2015. Best-worst scaling: Theory, methods and applications. Cambridge University Press.
Huaishao Luo, Tianrui Li, Bing Liu, and Junbo Zhang. 2019. DOER: Dual cross-shared RNN for aspect term-polarity co-extraction. In Proc. ACL '19, pages 591-601.
Zhengjie Miao, Yuliang Li, Xiaolan Wang, and Wang-Chiew Tan. 2020. Snippext: Semi-supervised opinion mining with augmented data. In Proc. WWW '20.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In Proc. ICLR '18.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get To The Point: Summarization with pointer-generator networks. In Proc. ACL '17, pages 1073-1083.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proc. NIPS '17, pages 5998-6008.
Wenya Wang, Sinno Jialin Pan, Daniel Dahlmeier, and Xiaokui Xiao. 2017. Coupled multi-layer attentions for co-extraction of aspect and opinion terms. In Proc. AAAI '17.
Xiaolan Wang, Yoshihiko Suhara, Natalie Nuno, Yu-liang Li, Jinfeng Li, Nofar Carmeli, Stefanos Angelidis, Eser Kandogan, and Wang-Chiew Tan. 2020. ExtremeReader: An interactive explorer for customizable and explainable review summarization. In Companion Proc. WWW '20, page 176-180.
5794
A Hyper-parameter Sensitivity Analysis
We present OPINIONDIGEST's hyper-parameters and their default settings in Table 6. Among these hyper-parameters, we found that the performance of OPINIONDIGEST is relatively sensitive to the following hyper-parameters: top- $k$  opinion  $(k)$ , merging threshold  $(\theta)$ , and maximum token length  $(L)$ .
To better understand OPINIONDIGEST's performance, we conducted additional sensitivity analysis of these three hyper-parameters. The results are shown in Figure 2.
Top- $k$  opinion vs Merging threshold: We tested different  $k = \{10, 11, \dots, 20, 30\}$  and  $\theta = \{0.6, 0.7, 0.8, 0.9\}$ . The mean (std) of R1, R2, and RL scores were 29.2 ( $\pm 0.3$ ), 5.6 ( $\pm 0.2$ ), and 18.5 ( $\pm 0.2$ ) respectively.
Top- $k$  opinion vs Maximum token length: We tested different  $k = \{10, 11, \dots, 20, 30\}$  and  $T = \{40, 50, \dots, 200\}$ . The mean (std) of R1, R2, and RL scores were 29.2 ( $\pm 0.4$ ), 5.6 ( $\pm 0.3$ ), and 18.5 ( $\pm 0.2$ ) respectively.
The results demonstrate that OPINIONDIGEST is robust to the choice of the hyper-parameters and constantly outperforms the best-performing baseline method.
B Human Evaluation Setup
We conducted user study via crowdsourcing using the FigureEight $^{6}$  platform. To ensure the quality of annotators, we used a dedicated expert-worker pool provided by FigureEight. We present the detailed setup of our user studies as follows.
Best-Worst Scaling Task: For each entity in the YELP and HOTEL datasets, we presented 8 input reviews and 3 automatically generated summaries to human annotators (Figure 3). The methods that generated those summaries were hidden from the annotators and the order of the summaries were shuffled for every entity. We further asked the annotators to select the best and worst summaries w.r.t. the following criteria:
- Informativeness: How much useful information about the business does the summary provide? You need to skim through the original reviews to answer this.
- Coherence: How coherent and easy to read is the summary?
Opinion Merging:
Word embedding	glove.6B.300d
Top-k opinion (k)	15
Merging threshold (θ)	0.8
Transformer model training:
SGD learning rate	0.1
Momentum (β)	0.1
Decay factor (γ)	0.1
Number of epochs	5
Training batch size	8
Decoding algorithm:
Beam size	5
Length penalty	0.6
n-gram blocking (n)	3
Maximum token length (L)	60
Table 6: List of OPINIONDIGEST hyper-parameters and the default settings.
- Non-redundancy: Is the summary successful at avoiding redundant and repeated opinions?
To evaluate the quality of the summaries for each criteria, we counted the number of best/worst votes for every system and computed the score as the Best-Worst Scaling (Louviere et al., 2015):
$$
s c o r e = \frac {\left| V o t e _ {b e s t} \right| - \left| V o t e _ {w o r s t} \right|}{\left| V o t e s _ {a l l} \right|}.
$$
The Best-Worst Scaling is known to be more robust for NLP annotation tasks and requires less annotations than rating-scale methods (Kiritchenko and Mohammad, 2016).
We collected responses from 3 human annotators for each question and computed the scores w.r.t. informativeness (I-score), coherence (C-score), and non-redundancy (R-score) accordingly.
Content Support Task: For the content support study, we presented the 8 input reviews to the annotators and an opinion summary produced from these reviews by one of the competing methods (ours or MeanSum). We asked the annotators to determine for every summary sentence, whether it is fully supported, partially supported, or not supported by the input reviews (Figure 4). We collected 3 responses per review sentence and calculated the ratio of responses for each category.
Aspect-Specific Summary Task: Finally, we studied the performance of OPINIONDIGEST in terms of its ability to generate controllable output. We presented the summaries to human judges and asked them to judge whether the summaries discussed the specific aspect exclusively, partially, or not at all (Figure 5). We again collected 3 responses
<sup>6</sup>https://www_figure-eight.com/
(d) ROUGE-1
(e) ROUGE-2
(f) ROUGE-L
Figure 2: Sensitivity analysis on hyper-parameters. Above row: Top- $k$  opinion  $(k)$  vs merging threshold  $(\theta)$ ; Bottom row: Top- $k$  opinion  $(k)$  vs max token size  $(L)$ .
per summary and calculated the percentage of responses.
5796
I recently stayed at the Club Quarters San Francisco. I cannot speak highly enough. The service by the Staff was over accommodating. The room was elegant and clean. The amenities at the hotel were great.
Staff professional and polite, Mattress/ pillows/ linens, good quality and comfortable. No moisturizer in bathroom.
Great little room, comfy bedding. I really liked the desk area and found it very useful during my stay! Plenty of outlets!
The location was good and close to the parking. Rooms were really small and didn't have the doors on the bedroom that I would have liked. We loved the bottled water dispenser (we have two small children). It was clean though and that is important. It had a cute little bar attached and the people were all really great and helpful.
It was very enjoyable, a welcoming environment with an attentive staff!
Review 6
Ideal location if you enjoy walking around like we do. Parking convenient just right across the way, and the most reasonable price in the city for 24hr parking. Ferry building, union square and Chinatown all within walking distance. Check in simple, elevators works only with key card as a guest, which makes the hotel safe and secure. On hand staff were friendly and has a great sense of humor on top of professionalism and customer service skills. Rooms were clean, beds comfortable, water pressure in the shower was more than adequate.
The room was a moderate size and clean. Nothing fancy, but it served its purpose for the few days it was there. It's just a few blocks down to Pier 1 on the waterfront and a relatively short trolley ride down to Fisherman's Wharf and the other tourist areas.
This Hotel Choice was great with regard to all important points. Very efficient. I am very familiar with San Francisco, so I knew what would be open and closed in the evening. Very close to all my Clients offices. Daytime is superb too for shopping and restaurants, and evening, just get low cost cabs (\(5 plus tip) to North Beach, walk to Chinatown, BART and Munl for other adventures, even direct to SFO ... BTW, a great shower too!
Summary A
Summary B
I really enjoyed staying here. The staff was professional and helpful. The location of this hotel is great, close to a lot of good restaurants and a short cab ride to fisherman's wharf. The room was comfortable and the bedding and linens were great.
Summary C
Rooms were clean, beds comfortable, water pressure in the shower was more than adequate. Great little room, comfy bedding. On hand staff were friendly and has a great sense of humor on top of professionalism and customer service skills.
Please read these opinion summaries carefully and select the best and worst one according to the following criteria:
Best:
Worst:
Coherence: 0
No Redundancy: ①
Figure 3: Screenshot of Best-Worst Scaling Task.
5797
Restaurant Reviews:
Wow! My order: Chicken Schwarma with a side of hummus and pita. Order of falafel. Cucumber drink. Side of garlic sauce. Side of cucumber sauce. Absolutely clean filling. Taste delicious! Will have you craving for more. I can't believe I hadn't heard of this restaurant sooner. After the fact I realize this place is all the rave!
I tried to order steak kebob but they made beef kebob. I asked for tzaziki on the side but they covered all the meat with tzaziki. Taste is more like middle eastern. Not Mediterranean. Price is good. Taste is okay.
Now this place is really good i always drive past it but today i decided to stop an check it out it is really good healthy an fresh
I was thinking this would be more of a sit down restaurant where you order from the table instead of a chipotleish style of Mediterranean food. Thought there would be more room inside for eating. The only thing good I had was the cucumber chiller which I would go back for. Not so much the food/service.
Parsley Modern Mediterranean is wonderful. Very responsive staff. Food is wonderful. I usually get the wraps (chicken or beef are my go-tos). Babaganoush and the warm pita bread is pretty amazing.
Very delicious food in love with cucumber drink, couldn't decide what I wanted and one specific Gentelman whipped up something very amazing for me! By the name of Jamil great service! Thanks you and will definitely be back!
This is Chipotle for Mediterranean food. And it. is. delicious. I've only been here once because the location is very inconvenient for me and I'm extremely lazy about driving more than 5 minutes to go anywhere, but if it were closer, I'd be here all the time. (It's probably better this way, I have very little self-control.) If you like spicy - get the hot sauce. Mix it with the white sauce, you won't be disappointed.
The food always taste fresh and leaves me very full without feeling tired. They have had a coupon for a very long time making this place an incredible value. This is my favorite Mediterranean place.
Opinion Summary Sentences:
I am a fan of the chicken shawarma.
I've also tried their chicken and it was delicious.
The chicken is so tender and juicy.
They also have a great selection of sauces and cheeses.
Is this sentence supported by the above reviews?
$\bigcirc$  Fully
$\odot$  Partially
Figure 4: Screenshot of Content Support Task.
Opinion summary about the recommendation of a restaurant:
I have been coming here for a few years now and have never had a bad experience. I highly recommend the fried rice and the fried rice. They are the best i've had in a long time. I will definitely be coming back to try other things on the menu. Is the place to go to for chinese
Does this opinion summary discuss the restaurant's recommendation?
Yes, exclusively
Yes, partially
Not at all
Figure 5: Screenshot of Aspect-Specific Summary Task.
5798