SlowGuess/NAACL_2022_OCR huggingface dataset:
Arthur Bražinskas<sup>1</sup> Ramesh Nallapati<sup>2</sup> Mohit Bansal<sup>2,3</sup> Markus Dreyer<sup>2</sup>
$^{1}$ ILCC University of Edinburgh  $^{2}$ Amazon
$^{3}$ UNC Chapel Hill
abrazinskas@ed.ac.uk,{rnallapa,mobansal, mddreyer}@amazon.com
Abstractive summarization models are typically pre-trained on large amounts of generic texts, then fine-tuned on tens or hundreds of thousands of annotated samples. However, in opinion summarization, large annotated datasets of reviews paired with reference summaries are not available and would be expensive to create. This calls for fine-tuning methods robust to overfitting on small datasets. In addition, generically pre-trained models are often not accustomed to the specifics of customer reviews and, after fine-tuning, yield summaries with disfluencies and semantic mistakes. To address these problems, we utilize an efficient few-shot method based on adapters which, as we show, can easily store in-domain knowledge. Instead of fine-tuning the entire model, we add adapters and pretrain them in a task-specific way on a large corpus of unannotated customer reviews, using held-out reviews as pseudo summaries. Then, fine-tune the adapters on the small available human-annotated dataset. We show that this self-supervised adapter pre-training improves summary quality over standard fine-tuning by 2.0 and 1.3 ROUGE-L points on the Amazon and Yelp datasets, respectively. Finally, for summary personalization, we condition on aspect keyword queries, automatically created from generic datasets. In the same vein, we pre-train the adapters in a query-based manner on customer reviews and then fine-tune them on annotated datasets. This results in better-organized summary content reflected in improved coherence and fewer redundancies.
Online reviews play an important role in purchasing decisions we make. They inform us about customer experiences – what aspects users like and dislike, and ultimately, whether a product or service is worth buying. Although significant progress has been made in supervised summarization in non-subjective single-document context, such as news
articles (Rush et al., 2015; Nallapati et al., 2016; See et al., 2017; Bražinskas et al., 2021b), modern deep learning methods rely on large amounts of annotated data, but these are not readily available in the review or opinion summarization domain and are expensive to produce. A key obstacle making such annotation expensive is that annotators need to consider multiple texts when writing a summary, which can be tens or even hundreds for realistic settings. Hence, most available datasets have human-written summaries for less than 100 products.
The lack of sufficiently large annotated datasets led to a variety of unsupervised abstractive models (e.g., COPYCAT (Bražinskas et al., 2020b), MEAN-SUM (Chu and Liu, 2019); DENOISESUM (Amplayo and Lapata, 2020)) that are trained on large collections of unannotated customer reviews. However, as the models are never exposed to actual summaries, they cannot capture their expected characteristics. This results in generated summaries mimicking the informal style of customer reviews and containing hallucinations and unimportant details. These limitations were addressed by few-shot methods learning from a handful of human-written summaries. (Bražinskas et al., 2020a; Oved and Levy, 2021). The first proposed few-shot learning model, FEWSUM (Bražinskas et al., 2020a), relies on various static features capturing differences between a customer review and a summary. While that model is more robust to overfitting, such features require manual, domain-specific engineering and can be sub-optimal for capturing correspondences between texts on the semantic level. We propose a simpler approach – ADASUM – which is based on adapters (Houlsby et al., 2019; Bapna et al., 2019). As we explain next, the adapters are pre-trained on customer reviews in a task-specific manner, and subsequently fine-tuned on gold summaries.
Adapters. We utilize a pre-trained model with powerful language understanding and generation
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1509 - 1523 July 10-15, 2022 ©2022 Association for Computational Linguistics
Figure 1: Illustration of the proposed approach. In Stage 1, all parameters of a large language model are pretrained on generic texts (we use BART). In Stage 2, we pre-train adapters (5% of the full model's parameters) on customer reviews using held-out reviews as summaries. In Stage 3, we fine-tune the adapters on a handful of reviews-summary pairs.
abilities in combination with a parameter-efficient fine-tuning method – adapters. As was shown in recent studies, this method is also robust to overfitting in low-resource settings (He et al., 2021). In this way, a large pre-trained model, BART (Lewis et al., 2020), in our case, remains frozen, and only small modules (0.6% - 5% of the model parameters) are optimized. This effectively retains acquired knowledge in the pre-trained language model (PLM) without specialized training objectives as in RECADEAM (Chen et al., 2020). However, available annotated data is not sufficient for learning in-domain specifics and results in summaries with subtle semantic mistakes. As explained next, we reduce these semantic mistakes by pre-training adapters on customer reviews.
Self-supervised Pre-training. Language models are pre-trained with generic objectives (e.g., single-document denoising) and rarely on in-domain data, such as customer reviews. Consequently, this makes them less attuned to in-domain specifics as these are hard to learn from a handful of summaries. This often results in subtle semantic mistakes. For instance, in Table 1, PASS (Oved and Levy, 2021) incorrectly concludes that thin material implies poor quality. To address this issue, we learn in-domain specifics from customer reviews. Concretely, we employ a self-supervised pre-training method: For any given product without a human-written summary, we predict one of the given reviews by conditioning on  $N$  other reviews with the highest lexical overlap in the leave-one-out fashion (Bražinskas et al., 2020b). As the standard training of PLMs is storage and memory inefficient (Mahabadi et al., 2021), we pre-train adapters only; see Stage 2 in Fig. 1. Afterwards, we fine-tune them on a small number of annotated reviews-summary pairs (< 100 pairs), see Stage 3 in
Fig. 1. All in all, our method combines the general text generation and understanding abilities of the PLM with in-domain knowledge directly related to the end task.
Content Planning. Well-organized content in summaries is easier to follow and thus improves user experience. However, the lack of annotated data makes it challenging to learn a desired content structure. For example, in Table 1, FEWSUM's summary does not end after a concluding phrase 'Other than that, it's a great top.' While the state-of-the-art model (PASS) addresses this issue by ranking multiple generated candidates with a specialized coherence model, we propose a simpler solution – ADAQSUM – that capitalizes on text planning (Hua and Wang, 2019; Moryossef et al., 2019). Specifically, we allow the model to plan ahead by providing an intermediate summary representation in the form of a query consisting of aspect keywords. As we show, this results in more coherent text patterns with fewer redundancies. Moreover, it can be useful for personalized summaries, better reflecting user interests.
Result Highlights. We evaluate the proposed models in terms of automatic metrics and human efforts. We find that pre-training and fine-tuning of adapters leads to more than 2.0 and 1.3 ROUGE-L points improvement over fine-tuning the entire model on Amazon and Yelp datasets, respectively. We also find that our pre-trained and fine-tuned query-based model improves ROUGE-L scores by more than 2.7 and 0.9 ROUGE-L points over PASS, on Amazon and Yelp datasets, respectively, and is more preferred by humans. We further demonstrate that the query-based model (ADAQSUM) substantially improves coherence and reduces redundancies in generated summaries.
FEWSUM	This tank top is well made, fits well, and is comfortable to wear. The only thing is that it runs a little small, so order a size up from what you normally wear. Other than that, it's a great top. It's well made and it looks like it will last a long time. Love it!
PASS	This is a basic tank. The photo shows it going well past the models hips. However, the material used to make it this long is thin and therefore not good quality. It is also thinner than other tanks on the market but is still comfortable to wear.
ADAQSUM	This is a basic tank top that fits well and is comfortable to wear. The color is great and the length is long enough to wear with leggings. The quality of the product is good.
REVIEW	... This is a basic tank ... || ... this tank fits like a normal tank top, not any longer ... I could wear it with leggings ... || ... It is THIN and runs SMALL ... It fits tight and is NOT long like in the picture ... || The tank fit very well and was comfortable to wear. I've bought much higher quality tanks ... || ... it is listed as a 'long' tank top and the photo even shows it going well past the models hips, however I'm short and the tank top is just a normal length. || ... They were a lot thinner than I like ... || Every women should own one in every color. Just feels quality I don't know how else to explain it ... || ... They are long enough that the color peeks out from under my tops. Looks cute.
Table 1: Generated summaries for an Amazon product by baseline models (FEWSUM and PASS) and our approach (ADAQSUM). Colored words indicate aspect keywords that were part of the query. The special marker 'll' separates truncated reviews.
- We propose a self-supervised pre-training method to learn in-domain knowledge by adapters that alleviate catastrophic forgetting;
- We propose, to the best of our knowledge, the first aspect-based abstractive opinion summarizer learned from a few annotated samples;
- We substantially increase summary coherence using the query-based approach;
- We show that self-supervised pre-training significantly improves performance on the query-based task;
- We demonstrate state-of-the-art results on two primary benchmarks in automatic and human evaluation.
Figure 2: Illustration of the query-based summarizer that inputs reviews and a text query consisting of aspects, such as 'volume,' 'price,' and 'bluetooth.' The query is automatically created from gold summaries in training and reviews in test time.
2.1 Opinion Summarization Tasks
In this work, we consider two tasks of customer review summarization. The first one is generic summarization (Chu and Liu, 2019; Bražinskas et al., 2020a), where the aim is to produce a summary that covers overall opinions in input reviews. Formally, given  $N$  input reviews  $r_{1:N}$ , the task is to predict word-by-word the summary  $s$ :
$$
\mathcal {L} (s, r _ {1: N}; \theta) = \sum_ {t = 1} ^ {T} \log p _ {\theta} (s ^ {t} | s ^ {1: t - 1}, r _ {1: N}).
$$
In the second task, query-based summarization, we assume that the user provides a query  $q$  consisting of aspect keywords, such as 'bluetooth,' 'resolution,' and 'battery life.' In turn, a summarizer should generate a summary reflecting customer opinions in  $r_{1:N}$  about these aspects. Formally, given a pair of input reviews and query  $(r_{1:N}, q)$ , the task is to predict word-by-word the summary  $s$ :
$$
\mathcal {L} (s, r _ {1: N}, q; \theta) = \sum_ {t = 1} ^ {T} \log p _ {\theta} (s ^ {t} | s ^ {1: t - 1}, r _ {1: N}, q).
$$
Unfortunately, abstractive opinion datasets with annotated aspect queries are unavailable in the domain. To mitigate this problem, we follow Ni et al. (2019) and create queries by extracting fine-grained aspect keywords from available generic summaries. Specifically, we utilize the model proposed by Zhang et al. (2014) to build a fine-grained aspect lexicon from review datasets. Further, we use simple rules to determine which aspects appear in summaries; see an annotated summary in Table 2. At test time, we follow the intuition that a summary should reflect common opinions and create a query from  $K$  most frequent aspect keywords in input reviews. The workflow is illustrated in Fig. 2.
<sup>1</sup>Our code and associated artifacts will be publicly available at https://github.com/amazon-research/adasum.
Our model is based on the Transformer (Vaswani et al., 2017) encoder-decoder architecture initialized with BART (Lewis et al., 2020). We adopt the same encoder as in (Raffel et al., 2020; Oved and Levy, 2021) where reviews are concatenated before encoding. This allows us to capture product-level features and leverage commonalities across reviews during encoding. For query-based summarization, we concatenate a query and reviews while indicating boundaries with special markers. In this way, the encoder can contextualize aspect keywords and focus on salient review fragments reflecting these aspects.
2.3 Adapters
In training, a large pre-trained model remains frozen and only tiny neural networks called adapters (Houlsby et al., 2019) are optimized. These modules are injected into the transformer layers (both encoder and decoder). Formally, given the input hidden vector  $h$ , the output vector  $\hat{h}$  is calculated as shown below:
$$
\hat {h} = f _ {2} (\tanh  f _ {1} (h)) + h.
$$
The functions  $f_{1}(\cdot)$  and  $f_{2}(\cdot)$  are the down- and up- projection layers. At each transformer layer, two adapters are inserted right after the self-attention and the feed-forward layers, respectively. These modules consist of substantially fewer parameters than the language model, usually around  $3\% - 5\%$ . Recent studies have shown that adapters are less prone to overfitting (He et al., 2021) and are more memory-efficient in training (Mahabadi et al., 2021). Finally, as the pre-trained model remains frozen, it retains all the prior knowledge for text understanding and generation. This effectively alleviates catastrophic forgetting (Goodfellow et al., 2013; Kemker et al., 2017) without modifying the training objective as in RECADEM (Chen et al., 2020; Yu et al., 2021). We refer to our approaches as ADASUM and ADAQSUM for generic and query-based summarization, respectively.
2.4 Self-supervised Pre-training
Language models, initially pre-trained on generic text corpora, are often not accustomed to in-domain specifics. Unsurprisingly perhaps, a wide range of
The cover offers durable protection for the MacBook, the retractable tilt stands offer protection for the wrists.
The keyboard cover can take some effort to fit properly, and adjustment to its feel may take time.
However, free and fast shipping make up for this one potential issue.
Table 2: Automatically annotated Amazon summary with fine-grained aspect keywords (underlined italic).
product-related specifics cannot be learned from a handful of annotated summaries during fine-tuning. Consequently, this can result in subtle semantic mistakes in generated summaries. We will discuss this problem and provide examples in Sec. 5.4. Furthermore, query-based summarization is even more challenging for learning than generic summarization. To be useful in practice, the summarizer should rely on the provided query after fine-tuning. However, a handful of annotated samples might be insufficient to learn this dynamic. We will analyze this problem in Sec. 5.1. To alleviate these two problems, we leverage unannotated customer reviews to construct synthetic datasets for pre-training.
Synthetic In-Domain Pre-Training Dataset. From a group of product reviews, we randomly sample one review as a pseudo summary  $s$  and select  $N$  reviews as input  $(r_{1:N})$ . We select  $N$  input reviews covering the content of the summary  $s$  – that have the highest ROUGE-1 F scores. Following the naming convention (Bražinskas et al., 2020b), we refer to this as leave-one-out pretraining (L1O). To closely resemble query-based summarization, we create aspect queries from pseudo summaries. Specifically, we leverage the aspect lexicon by matching summary keywords; in the same way as was explained in Sec. 2.1. In practice, we expect queries to have at least one aspect keyword. Therefore, we remove all pre-training pairs where the pseudo summary has no aspect keywords.
To create synthetic datasets, we used customer reviews from Amazon (He and McAuley, 2016) and
2We experimented with the independent review encoding as in Bražinskas et al. (2020b). However, the results were slightly worse.
3We also experimented with selecting pseudo summaries without personal pronouns - written in the formal style (Bražinskas et al., 2020a). However, we did not observe significant improvements.
Amazon Yelp
Split	Gold	Synthetic	Gold	Synthetic
Train	84	70,144 / 59,963	90	70,144 / 68,499
Valid	36	7,900 / 6,810	90	7,900 / 7,724
Test	60	-	120	-
Table 3: Source-target pair numbers for Amazon and Yelp, both gold and synthetic. Each pair has 8 source reviews. Generic and query-based pair statistics are separated by  $\prime /$ .
Yelp. Following Bražinskas et al. (2020a), we selected 4 categories: Electronics; Clothing, Shoes and Jewelry; Home and Kitchen; Health and Personal Care. We pre-processed the datasets by removing all reviews that are shorter than 20 words and longer than 120 words and evened the number of pairs in both datasets. Further, we used Amazon and Yelp gold summaries from (Bražinskas et al., 2020a) where each product/business has 3 references and is paired with 8 reviews. Gold and synthetic dataset statistics are presented in Table 3.
LEXRANK (Erkan and Radev, 2004) is an unsupervised extractive graph-based model that selects sentences based on graph centrality. Sentences represent nodes in a graph whose edges are weighted with tfidf.
MEANSUM (Chu and Liu, 2019) is an unsupervised abstractive summarization model which treats a summary as a structured latent state of an autoencoder trained to reconstruct reviews of a product.
COPYCAT (Bražinskas et al., 2020b) is the state-of-the-art unsupervised abstractive summarizer with hierarchical continuous latent representations to model products and individual reviews.
FEWSUM (Bražinskas et al., 2020a) is a few-shot framework where lexical features are used to differentiate between customer reviews and summaries. In the fine-tuning phase, features leading to generation of summaries are searched.
PASS (Oved and Levy, 2021) is based on a pretrained T5 model (Raffel et al., 2020) that is further fine-tuned on gold summaries. At inference, the model's input is perturbed to generate multiple candidates. These candidates are further ranked by a separate model based on coherence and fluency to select the best one.
We fine-tuned the full BART model (FULL) for
a fair comparison, with and without the leave-one-out pre-training. We also employed a number of simple summarization baselines. First, the CLUSTROID review was computed for each group of reviews as follows. We took each review from a group and computed ROUGE-L with respect to all other reviews. The review with the highest ROUGE score was selected as the clustroid review. Second, we sampled a RANDOM review from each group to be used as the summary. And lastly, we constructed the summary by selecting the leading sentences (LEAD) from each review of a group.
3.3 Experimental Details
We used a standard Transformer encoder-decoder (Vaswani et al., 2017), pre-initialized with BART large (Lewis et al., 2020), consisting of 400M parameters. We used two adapter sizes -  $0.6\%$  and  $5\%$  of the full model's parameters. All input reviews were concatenated, following Raffel et al. (2020); Oved and Levy (2021). For parameter optimization, we used ADAM (Kingma and Ba, 2014), and summary generation was performed via the beam search of size 5 and with 3-gram blocking (Paulus et al., 2017). We used ROUGE-L as the stopping criterion on the end task, and perplexity (PPL) for pre-training. The learning rate for most experiments was set to 5e-5. Aspect lexicons for query-based summarization contained 2809 and 4013 fine-grained aspects for Amazon and Yelp, respectively. In pre-training and fine-tuning, we shuffled aspects to break temporal dependencies. For fine-tuning on Yelp, we also found it useful to exclude summary aspect keywords that do not appear in input reviews. This approximately matched the number of aspect keywords for Amazon and Yelp. At test time, we selected up to 6 and 5 most frequent aspects for Amazon and Yelp, respectively. All computations were performed on an 8-GPU p3.8-xlarge Amazon instance.
4.1 Automatic Evaluation
Table 4 shows results on the Amazon and Yelp test sets for generic summarization. It shows ROUGE F1 scores (Lin, 2004) as a standard measure of informativeness<sup>6</sup> and perplexity (PPL) as a measure of confusion.
For consistency with previous works, we used the same Python package (https://github.com/google-research/google-research/tree/master/rouge)
4https://www.yelp.com/dataset
For the query-based setup, we removed all instances where targets had no aspects.
Amazon	Yelp
Params↓	PPL↓	R1↑	R2↑	RL↑	PPL↓	R1↑	R2↑	RL↑

CLUSTROID	-	-	27.16	3.61	16.77	-	28.90	4.90	18.00
LEAD	-	-	27.00	4.92	14.95	-	26.20	4.57	14.32
RANDOM	-	-	25.00	3.82	15.72	-	21.48	2.59	13.87
Unsupervised
LEXRANK (Erkan and Radev, 2004)	-	-	27.72	5.06	17.04	-	26.96	4.93	16.13
MEANSUM (Chu and Liu, 2019)	25M	-	26.63	4.89	17.11	-	27.50	3.54	16.09
COPYCAT (Bražinskas et al., 2020b)	25M	-	27.85	4.77	18.86	-	28.12	5.89	18.32
Few-shot
FEWSUM (Bražinskas et al., 2020a)	25M	-	33.56	7.16	21.49	-	37.29	9.92	22.76
PASS (Oved and Levy, 2021)	440M	-	37.43	8.02	23.34	-	36.91	8.12	23.09
FULL (100%)	400M	17.87	37.22	9.17	23.51	12.87	37.40	10.27	23.76
FULL (100%) + L10	400M	16.90	37.67	10.28	24.32	12.40	36.79	11.07	25.03
ADASUM (0.6%)	2.6M	13.45	38.49	9.84	24.37	11.94	37.55	10.11	24.08
ADASUM (0.6%) + L10	2.6M	12.06	38.94	10.63	24.95	11.23	37.78	11.31	24.04
ADASUM (5%)	21.3M	16.30	38.15	9.18	23.17	12.50	38.12	10.89	24.11
ADASUM (5%) + L10	21.3M	12.03	39.78	10.80	25.55	11.11	38.82	11.75	25.14
Table 4: Test set ROUGE F1 scores on gold Amazon and Yelp datasets for generic review summarization. L10 stands for leave-one-out pre-training. We also provide the total number of trainable parameters.
First of all, the results indicate the superiority of adapters over full fine-tuning and state-of-the-art few-shot models on both datasets. As was observed in He et al. (2021), adapters are less prone to overfitting, which is especially beneficial in few-shot settings. Second, we observe a significant improvement in ROUGE scores when pre-trained models are further trained using L1O. This signifies the importance of learning in-domain specifics before fine-tuning. We also observe that adapters are more effective on the Amazon dataset, which is more challenging as indicated by higher perplexity (PPL).<sup>7</sup> We hypothesize that the pretrained language model (BART) is more accustomed to restaurant- than product-related texts. Moreover, larger adapters  $(5\%)$  tend to overfit on the small number of annotated instances, and L1O pre-training helps substantially, as indicated both by ROUGE scores and PPL. We provide example generated summaries in the Appendix.
Coherence Improvement. As was observed in (Oved and Levy, 2021), opinion summarizers sometimes generate incoherent summaries. We hypothesized that a query should allow the model to plan ahead of time and thus generate more coherent and less redundant texts. To test the hypothesis, we compared  $5\%$  adapter-based models with and without the query; both were pre-trained via L10.
We performed human evaluation in terms of coherence and non-redundancy via Best-Worst Scaling (BWS) (Louviere and Woodworth, 1991; Louviere et al., 2015). BWS has been shown to produce more reliable results than ranking scales (Kiritchenko and Mohammad, 2016).