SlowGuess/ACL_2020_OCR huggingface dataset:
Arthur Bražinskas<sup>1</sup>, Mirella Lapata<sup>1</sup> and Ivan Titov<sup>1,2</sup>
$^{1}$  ILCC, University of Edinburgh
2 ILLC, University of Amsterdam
abrazinskas@ed.ac.uk, {mlap, ititov}@inf.ed.ac.uk
Opinion summarization is the task of automatically creating summaries that reflect subjective information expressed in multiple documents, such as product reviews. While the majority of previous work has focused on the extractive setting, i.e., selecting fragments from input reviews to produce a summary, we let the model generate novel sentences and hence produce abstractive summaries. Recent progress in summarization has seen the development of supervised models which rely on large quantities of document-summary pairs. Since such training data is expensive to acquire, we instead consider the unsupervised setting, in other words, we do not use any summaries in training. We define a generative model for a review collection which capitalizes on the intuition that when generating a new review given a set of other reviews of a product, we should be able to control the "amount of novelty" going into the new review or, equivalently, vary the extent to which it deviates from the input. At test time, when generating summaries, we force the novelty to be minimal, and produce a text reflecting consensus opinions. We capture this intuition by defining a hierarchical variational autoencoder model. Both individual reviews and the products they correspond to are associated with stochastic latent codes, and the review generator ("decoder") has direct access to the text of input reviews through the pointer-generator mechanism. Experiments on Amazon and Yelp datasets, show that setting at test time the review's latent code to its mean, allows the model to produce fluent and coherent summaries reflecting common opinions.
Summarization of user opinions expressed in online resources, such as blogs, reviews, social media, or internet forums, has drawn much attention due to its potential for various information access applications, such as creating digests, search, and report
Summary	This restaurant is a hidden gem in Toronto. The food is delicious, and the service is impeccable. Highly recommend for anyone who likes French bistro.
Reviews	We got the steak frites and the chicken frites both of which were very good ... Great service ... || I really love this place ... Côte de Boeuf ... A Jewel in the big city ... || French jewel of Spadina and Adelaide , Jules ... They are super accommodating ... moules and frites are delicious ... || Food came with tons of greens and fries along with my main course , thumbs uppp ... || Chef has a very cool and fun attitude ... || Great little French Bistro spot ... Go if you want French bistro food classics ... || Great place ... the steak frites and it was amazing ... Best Steak Frites ... in Downtown Toronto ... || Favourite french spot in the city ... crème brule for dessert
Table 1: A summary produced by our model; colors encode its alignment to the input reviews. The reviews are truncated, and delimited with the symbol  $\left| \right|$  .
generation (Hu and Liu, 2004; Angelidis and Lapata, 2018; Medhat et al., 2014). Although there has been significant progress recently in summarizing non-subjective context (Rush et al., 2015; Nallapati et al., 2016; Paulus et al., 2017; See et al., 2017; Liu et al., 2018), modern deep learning methods rely on large amounts of annotated data that are not readily available in the opinion-summarization domain and expensive to produce. Moreover, annotation efforts would have to be undertaken for multiple domains as online reviews are inherently multi-domain (Blitzer et al., 2007) and summarization systems highly domain-sensitive (Isonuma et al., 2017). Thus, perhaps unsurprisingly, there is a long history of applying unsupervised and weakly-supervised methods to opinion summarization (e.g., Mei et al. 2007; Titov and McDonald 2008; Angelidis and Lapata 2018), however, these
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5151-5169 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
approaches have primarily focused on extractive summarization, i.e., producing summaries by copying parts of the input reviews.
In this work, we instead consider abstractive summarization which involves generating new phrases, possibly rephrasing or using words that were not in the original text. Abstractive summaries are often preferable to extractive ones as they can synthesize content across documents avoiding redundancy (Barzilay et al., 1999; Carenini and Cheung, 2008; Di Fabbrizio et al., 2014). In addition, we focus on the unsupervised setting and do not use any summaries for training. Unlike aspect-based summarization (Liu, 2012), which rewards the diversity of opinions, we aim to generate summaries that represent consensus (i.e., dominant opinions in reviews). We argue that such summaries can be useful for quick decision making, and to get an overall feel for a product or business (see the example in Table 1).
More specifically, we assume we are provided with a large collection of reviews for various products and businesses and define a generative model of this collection. Intuitively, we want to design such a model that, when generating a review for a product relying on a set of other reviews, we can control the "amount of novelty" going into the new review or, equivalently, vary the extent to which it deviates from the input. At test time, we can force the novelty to be minimal, and generate summaries representing consensus opinions.
We capture this intuition by defining a hierarchical variational autoencoder (VAE) model. Both products and individual reviews are associated with latent representations. Product representations can store, for example, overall sentiment, common topics, and opinions expressed about the product. In contrast, latent representations of reviews depend on the product representations and capture the content of individual reviews. While at training time the latent representations are random variables, we fix them to their respective means at test time. As desired for summarization, these 'average' (or 'copycat') reviews differ in writing style from a typical review. For example, they do not contain irrelevant details that are common in customer reviews, such as mentioning the occasion or saying how many family members accompanied the reviewer. In order to encourage the summaries to include spe
cific details, the review generator ('decoder') has direct access to the text of input reviews through the pointer-generator mechanism (See et al., 2017). In the example in Table 1, the model included specific information about the restaurant type and its location in the generated summary. As we will see in ablation experiments, without this conditioning, model performance drops substantially, as the summaries become more generic.
We evaluate our approach on two datasets, Amazon product reviews and Yelp reviews of businesses. The only previous method dealing with unsupervised multi-document opinion summarization, as far as we are aware of, is MeanSum (Chu and Liu, 2019). Similarly to our work, they generate consensus summaries and consider the Yelp benchmark. Whereas we rely on continuous latent representations, they treat the summary itself as a discrete latent representation of a product. Although this captures the intuition that a summary should relay key information about a product, using discrete latent sequences makes optimization challenging; (Miao and Blunsom, 2016; Baziotis et al., 2019; Chu and Liu, 2019) all have to use an extra training loss term and biased gradient estimators.
- we introduce a simple end-to-end approach to unsupervised abstractive summarization;
- we demonstrate that the approach substantially outperforms the previous method, both when measured with automatic metrics and in human evaluation;
- we provide a dataset of abstractive summaries for Amazon products.2
2 Model and Estimation
As discussed above, we approach the summarization task from a generative modeling perspective. We start with a high level description of our model, then, in Sections 2.2 and 2.3, we describe how we estimate the model and provide extra technical details. In Section 3, we explain how we use the model to generate summaries.
2.1 Overview of the Generative Model
Our text collection consists of groups of reviews, with each group corresponding to a single product.
For simplicity, we refer to both products (e.g., iPhone X) and businesses (e.g., a specific Starbucks branch) as products.
$^{2}$ Data and code: https://github.com/ixlan/ Copycat-abstractive-opinion-summarizer.
(a) Conditional independence of the reviews given the group representation  $c$ .
Figure 1: Unfolded graphical representation of the model.
(b) The  $r_i$ 's decoder accesses other reviews of the group  $(r_1, \dots, r_{i-1}, r_{i+1}, \dots, r_N)$ .
Our latent summarization model (which we call COPYCAT) captures this hierarchical organization and can be regarded as an extension of the vanilla text-VAE model (Bowman et al., 2016). COPYCAT uses two sets of latent variables as shown in Figure 1a. Namely, we associate each review group (equivalently, each product) with a continuous variable  $c$ , which captures the group's 'latent semantics'. In addition, we associate each individual review  $(r_i)$  with a continuous variable  $z_i$ , encoding the semantics of that review. The information stored in  $z_i$  is used by the decoder  $p_{\theta}(r_i|z_i)$  to produce review text  $r_i$ . The marginal log-likelihood of one group of reviews  $r_{1:N} = (r_1,\dots,r_N)$  is given by
$$
\log p _ {\theta} (r _ {1: N}) =
$$
$$
\log \int \left[ p _ {\theta} (c) \prod_ {i = 1} ^ {N} \left[ \int p _ {\theta} (r _ {i} | z _ {i}) p _ {\theta} (z _ {i} | c) \mathrm {d} z _ {i} \right] \mathrm {d} c \right],
$$
where we marginalize over variables  $c$  and  $z_{1:N}$ .
When generating a new review  $r_i$ , given the set of previous reviews  $r_{1:i}$ , the information about these reviews has to be conveyed through the latent representations  $c$  and  $z_i$ . This bottleneck is undesirable, as it will make it hard for the model to pass fine-grain information. For example, at generation time, the model should be reusing named entities (e.g., product names or technical characteristics) from other reviews rather than 'hallucinating' or avoiding generating them at all, resulting in generic and non-informative text. We alleviate this issue by letting the decoder directly access other reviews. We can formulate this as an autoregressive model:
$$
p _ {\theta} \left(r _ {1: N} \mid c\right) = \prod_ {i = 1} ^ {N} p _ {\theta} \left(r _ {i} \mid r _ {1}, \dots , r _ {i - 1}, c\right). \tag {1}
$$
As we discuss in Section 2.3, the conditioning is instantiated using the pointer-generator mechanism (See et al., 2017) and, thus, will specifically help in generating rare words (e.g., named entities).
We want our summarizer to equally rely on every review, without imposing any order (e.g., temporal) on the generation process. Instead, as shown in Figure 1b, when generating  $r_i$ , we let the decoder access all other reviews within a group,  $r_{-i} = (r_1, \ldots, r_{i-1}, r_{i+1}, \ldots, r_N)$ . This is closely related to pseudolikelihood estimation (Besag, 1975) or Skip-Thought's objective (Kiros et al., 2015). The final objective that we maximize for each group of reviews  $r_{1:N}$ :
$$
\log \int p _ {\theta} (c) \prod_ {i = 1} ^ {N} \left[ \int p _ {\theta} \left(r _ {i} \mid z _ {i}, r _ {- i}\right) p _ {\theta} \left(z _ {i} \mid c\right) \mathrm {d} z _ {i} \right] \mathrm {d} c \tag {2}
$$
We will confirm in ablation experiments that both hierarchical modeling (i.e., using  $c$ ) and the direct conditioning on other reviews are beneficial.
2.2 Model Estimation
As standard with VAEs and variational inference in general (Kingma and Welling, 2013), instead of directly maximizing the intractable marginal likelihood in Equation 2, we maximize its lower bound:
$$
\mathcal {L} (\theta , \phi ; r _ {1: N}) =
$$
$$
\underset {c \sim q _ {\phi} (c | r _ {1: N})} {\mathbb {E}} \left[ \sum_ {i = 1} ^ {N} \underset {z _ {i} \sim q _ {\phi} (z _ {i} | r _ {i}, c)} {\mathbb {E}} [ \log p _ {\theta} (r _ {i} | z _ {i}, r _ {- i}) ] \right.
$$
$$
\left. - \sum_ {i = 1} ^ {N} \mathbb {D} _ {\mathrm {K L}} \left[ q _ {\phi} \left(z _ {i} \mid r _ {i}, c\right) | | p _ {\theta} \left(z _ {i} \mid c\right) \right] \right]
$$
$-\mathbb{D}_{\mathrm{KL}}[q_{\phi}(c|r_{1:N})||p_{\theta}(c)]$  See the derivations in Appendix A.1.
Figure 2: Production of latent code  $z_{N}$  for review  $r_{N}$ .
The lower bound includes two 'inference networks',  $q_{\phi}(c|r_{1:N})$  and  $q_{\phi}(z_i|r_i,c)$ , which are neural networks parameterized with  $\phi$  and will be discussed in detail in Section 2.3. They approximate the corresponding posterior distributions of the model. The first term is the reconstruction error: it encourages the quality reconstruction of the reviews. The other two terms are regularizers. They control the amount of information encoded in the latent representation by penalizing the deviation of the estimated posteriors from the corresponding priors, the deviation is measured in terms of the Kullback-Leibler (KL) divergence. The bound is maximized with respect to both the generative model's parameters  $\theta$  and inference networks' parameters  $\phi$ . Due to Gaussian assumptions, the Kullback-Leibler (KL) divergence terms are available in closed form, while we rely on the reparameterization trick (Kingma and Welling, 2013) to compute gradients of the reconstruction term.
The inference network predicting the posterior for a review-specific variable  $q_{\phi}(z_i|r_i,c)$  is needed only in training and is discarded afterwards. In contrast, we will exploit the inference network  $q_{\phi}(c|r_{1:N})$  when generating summaries, as discussed in Section 3.
2.3 Design of Model Components
2.3.1 Text Representations
A GRU encoder (Cho et al., 2014) embeds review words  $w$  to obtain hidden states  $h$ . Those representations are reused across the system, e.g., in the inference networks and the decoder.
The full architecture used to produce the latent codes  $c$  and  $z_{i}$  is shown in Figure 2. We make Gaussian assumptions for all distributions (i.e. posteriors and priors). As in Kingma and Welling (2013), we use separate linear projections (LPs) to compute the means and diagonal log-covariances.
2.3.2 Prior  $p(c)$  and posterior  $q_{\phi}(c|r_{1:N})$
We set the prior over group latent codes to the standard normal distribution,  $p(c) = \mathcal{N}(c;0,I)$ . In order to compute the approximate posterior  $q_{\phi}(c|r_{1:N})$ , we first predict the contribution ('importance') of each word in each review  $\alpha_{i}^{t}$  to the code of the group:
$$
\alpha_ {i} ^ {t} = \frac {\exp (f _ {\phi} ^ {\alpha} (m _ {i} ^ {t}))}{\sum_ {j = 1} ^ {N} \sum_ {k} ^ {T _ {j}} \exp (f _ {\phi} ^ {\alpha} (m _ {j} ^ {k}))},
$$
where  $T_{i}$  is the length of  $r_i$  and  $f_{\phi}^{\alpha}$  is a feed-forward neural network (FFNN) $^4$  which takes as input concatenated word embeddings and hidden states of the GRU encoder,  $m_{i}^{t} = [h_{i}^{t}\circ w_{i}^{t}]$ , and returns a scalar.
Next, we compute the intermediate representation with the weighted sum:  $\hat{h} = \sum_{i=1}^{N} \sum_{t}^{T_i} \alpha_i^t m_i^t$ . Finally, we compute the Gaussian's parameters using the affine projections:
$$
\mu_ {\phi} (r _ {1: N}) = L \hat {h} + b _ {L}
$$
$$
\log \sigma_ {\phi} (r _ {1: N}) = G \hat {h} + b _ {G}
$$
2.3.3 Prior  $p_{\theta}(z_i|c)$  and posterior  $q_{\phi}(z_i|r_i,c)$
To compute the prior on the review code  $z_{i}$ ,  $p_{\theta}(z_i|c) = \mathcal{N}(z_i;\mu_\theta (c),I\sigma_\theta (c))$ , we linearly project the product code  $c$ . Similarly, to compute the parameters of the approximate posterior  $q_{\phi}(z|r_i,c) = \mathcal{N}(z;\mu_{\phi}(r_i,c),I\sigma_{\phi}(r_i,c))$ , we concatenate the last encoder's state  $h_i^{T_i}$  of the review  $r_i$  and  $c$ , and perform affine transformations.
2.3.4 Decoder  $p_{\theta}(r_i|z_i,r_{-i})$
To compute the distribution  $p_{\theta}(r_i|z_i,r_{-i})$ , we use an auto-regressive GRU decoder with the attention mechanism (Bahdanau et al., 2015) and a pointer-generator network.
We compute the context vector  $c_i^t = \mathrm{att}(s_i^t, h_{-i})$  by attending to all the encoder's hidden states  $h_{-i}$  of the other reviews  $r_{-i}$  of the group, where the decoder's hidden state  $s_i^t$  is used as a query. The
<sup>4</sup>We use FFNNs with the tanh non-linearity in several model components. Whenever a FFNN is mentioned in the subsequent discussion, this architecture is assumed.
hidden state of the decoder is computed using the GRU cell as
$$
s _ {i} ^ {t} = \mathrm {G R U} _ {\theta} \left(s _ {i} ^ {t - 1}, \left[ w _ {i} ^ {t} \circ c _ {i} ^ {t - 1} \circ z _ {i} \right]\right). \tag {3}
$$
The cell inputs the previous hidden state  $s_i^{t-1}$ , as well as concatenated word embedding  $w_i^t$ , context vector  $c_i^{t-1}$ , and latent code  $z_i$ .
Finally, we compute the word distributions using the pointer-generator network  $g_{\theta}$ :
$$
p _ {\theta} \left(r _ {i} \mid z _ {i}, r _ {- i}\right) = \prod_ {t = 1} ^ {T} g _ {\theta} \left(r _ {i} ^ {t} \mid s _ {i} ^ {t}, c _ {i} ^ {t}, w _ {i} ^ {t}, r _ {- i}\right) \tag {4}
$$
The pointer-generator network computes two internal word distributions that are hierarchically aggregated into one distribution (Morin and Bengio, 2005). One distribution assigns probabilities to words being generated using a fixed vocabulary, and another one probabilities to be copied directly from the other reviews  $r_{-i}$ . In our case, the network helps to preserve details and, especially, to generate rare tokens.
3 Summary Generation
Given reviews  $r_{1:N}$ , we generate a summary that reflects common information using trained components of the model. Formally, we could sample a new review from
$$
\begin{array}{l} p _ {\theta} (r \mid r _ {1: N}) = \\ \underset {c \sim q _ {\phi} (c | r _ {1: N})} {\mathbb {E}} \left[ \underset {z \sim p _ {\theta} (z | c)} {\mathbb {E}} \left[ p _ {\theta} (r | z, r _ {1: N}) \right] \right]. \\ \end{array}
$$
As we argued in the introduction and will revisit in experiments, a summary or summarizing review, should be generated relying on the mean of the reviews' latent code. Consequently, instead of sampling  $z$  from  $p_{\theta}(z|c) = \mathcal{N}(z; \mu_{\theta}(c), I\sigma_{\theta}(c))$ , we set it to  $\mu_{\theta}(c)$ . We also found beneficial, in terms of evaluation metrics, not to sample  $c$  but instead to rely on the mean predicted by the inference network  $q_{\phi}(c|r_{1:N})$ .
Our experiments were conducted on business customer reviews from the Yelp Dataset Challenge and Amazon product reviews (He and McAuley, 2016). These were pre-processed similarly to Chu and Liu (2019), and the corresponding data statistics are
Dataset	Training	Validation
Yelp	38,776/1,012,280	4,311/113,373
Amazon	183,103/4,566,519	9,639/240,819
Table 2: Data statistics after pre-processing. The format in the cells is Businesses/Reviews and Products/Reviews for Yelp and Amazon, respectively.
shown in Table 2. Details of the pre-processing are available in Appendix A.2.
These datasets present different challenges to abstractive summarization systems. Yelp reviews contain much personal information and irrelevant details which one may find unnecessary in a summary. Our summarizer, therefore, needs to distill important information in reviews while abstracting away from details such as a listing of all items on the menu, or mentions of specific dates or occasions upon which customers visited a restaurant. On the contrary, in Amazon reviews, we observed that users tend to provide more objective information and specific details that are useful for decision making (e.g., the version of an electronic product, its battery life, its dimensions). In this case, it would be desirable for our summarizer to preserve this information in the output summary.
For evaluation, we used the same 100 human-created Yelp summaries released by Chu and Liu (2019). These were generated by Amazon Mechanical Turk (AMT) workers, who summarized 8 input reviews. We created a new test for Amazon reviews following a similar procedure (see Appendix A.6 for details). We sampled 60 products and 8 reviews for each product, and they were shown to AMT workers who were asked to write a summary. We collected three summaries per product, 28 products were used for development and 32 for testing.
4.2 Experimental Details
We used GRUs (Cho et al., 2014) for sequential encoding and decoding we used GRUs. We randomly initialized word embeddings that were shared across the model as a form of regularization (Press and Wolf, 2017). Further, optimization was performed using Adam (Kingma and Ba, 2014). In order to overcome the "posterior collapse" (Bowman et al., 2016), both for our model and the vanilla VAE baseline, we applied cyclical annealing (Fu et al., 2019). The reported ROUGE scores are based on F1 (see Appendix A.3 for details on hyperparameters).
R1	R2	RL
Copycat	0.2947	0.0526	0.1809
MeanSum	0.2846	0.0366	0.1557
LexRank	0.2501	0.0362	0.1467
Opinosis	0.2488	0.0278	0.1409
VAE	0.2542	0.0311	0.1504
Clustroid	0.2628	0.0348	0.1536
Lead	0.2634	0.0372	0.1386
Random	0.2304	0.0244	0.1344
Oracle	0.2907	0.0527	0.1863
Table 3: ROUGE scores on the Yelp test set.
R1	R2	RL
Copycat	0.3197	0.0581	0.2016
MeanSum	0.2920	0.0470	0.1815
LexRank	0.2874	0.0547	0.1675
Opinosis	0.2842	0.0457	0.1550
VAE	0.2287	0.0275	0.1446
Clustroid	0.2928	0.0441	0.1778
Lead	0.3032	0.0590	0.1578
Random	0.2766	0.0472	0.1695
Oracle	0.3398	0.0788	0.2160
Table 4: ROUGE scores on the Amazon test set.
Opinosis is a graph-based abstractive summarizer (Ganesan et al., 2010) designed to generate short opinions based on highly redundant texts. Although it is referred to as abstractive, it can only select words from the reviews.
LexRank is an unsupervised algorithm which selects sentences to appear in the summary based on graph centrality (sentences represent nodes in a graph whose edges have weights denoting similarity computed with tfidf). A node's centrality can be measured by running a ranking algorithm such as PageRank (Page et al., 1999).
MeanSum<sup>5</sup> is the unsupervised abstractive summarization model (Chu and Liu, 2019) discussed in the introduction.
We also trained a vanilla text VAE model (Bowman et al., 2016) with our GRU encoder and decoder. When generating a summary for  $r_1, \ldots, r_N$ , we averaged the means of  $q_{\phi}(z_i|r_i)$ .
Finally, we used a number of simple summarization baselines. We computed the clustroid review for each group as follows. We took each review from a group and computed ROUGE-L with respect to all other reviews. The review with the highest ROUGE score was selected as the clustroid review. Furthermore, we sampled a random review from each group as the summary, and constructed the summary by selecting the leading sentences from each review of a group.
Additionally, as an upper bound, we report the performance of an oracle review, i.e., the highest-scoring review in a group when computing ROUGE-L against reference summaries.
As can be seen in Tables 3 and 4, our model, Copycat, yields the highest scores on both Yelp and Amazon datasets.
We observe large gains over the vanilla VAE. We conjecture that the vanilla VAE struggles to properly represent the variety of categories under a single prior  $p(z)$ . For example, reviews about a sweater can result in a summary about socks (see example summaries in Appendix). This contrasts with our model which allows each group to have its own prior  $p_{\theta}(z|c)$  and access to other reviews during decoding. The gains are especially large on the Amazon dataset, which is very broad in terms of product categories.
Our model also substantially outperforms MeanSum. As we will confirm in human evaluation, MeanSum's summaries are relatively fluent at the sentence level but often contain hallucinations, i.e., information not present in the input reviews.