SlowGuess/ACL_2020_OCR huggingface dataset:
Reinald Kim Amplayo and Mirella Lapata  
Institute for Language, Cognition and Computation  
School of Informatics, University of Edinburgh  
reinald.kim@ed.ac.uk, mlap@inf.ed.ac.uk
The supervised training of high-capacity models on large datasets containing hundreds of thousands of document-summary pairs is critical to the recent success of deep learning techniques for abstractive summarization. Unfortunately, in most domains (other than news) such training data is not available and cannot be easily sourced. In this paper we enable the use of supervised learning for the setting where there are only documents available (e.g., product or business reviews) without ground truth summaries. We create a synthetic dataset from a corpus of user reviews by sampling a review, pretending it is a summary, and generating noisy versions thereof which we treat as pseudo-review input. We introduce several linguistically motivated noise generation functions and a summarization model which learns to denoise the input and generate the original review. At test time, the model accepts genuine reviews and generates a summary containing salient opinions, treating those that do not reach consensus as noise. Extensive automatic and human evaluation shows that our model brings substantial improvements over both abstractive and extractive baselines.
The proliferation of massive numbers of online product, service, and merchant reviews has provided strong impetus to develop systems that perform opinion mining automatically (Pang and Lee, 2008). The vast majority of previous work (Hu and Liu, 2006) breaks down the problem of opinion aggregation and summarization into three interrelated tasks involving aspect extraction (Mukherjee and Liu, 2012), sentiment identification (Pang et al., 2002; Pang and Lee, 2004), and summary creation based on extractive (Radev et al., 2000; Lu et al., 2009) or abstractive methods (Ganesan et al., 2010; Carenini et al., 2013; Gerani et al., 2014; Di Fabbrizio et al., 2014). Although po
tentially more challenging, abstractive approaches seem more appropriate for generating informative and concise summaries, e.g., by performing various rewrite operations (e.g., deletion of words or phrases and insertion of new ones) which go beyond simply copying and rearranging passages from the original opinions.
Abstractive summarization has enjoyed renewed interest in recent years thanks to the availability of large-scale datasets (Sandhaus, 2008; Hermann et al., 2015; Grusky et al., 2018; Liu et al., 2018; Fabbri et al., 2019) which have driven the development of neural architectures for summarizing single and multiple documents. Several approaches (See et al., 2017; Celikyilmaz et al., 2018; Paulus et al., 2018; Gehrmann et al., 2018; Liu et al., 2018; Perez-Beltrachini et al., 2019; Liu and Lapata, 2019; Wang and Ling, 2016) have shown promising results with sequence-to-sequence models that encode one or several source documents and then decode the learned representations into an abstractive summary.
The supervised training of high-capacity models on large datasets containing hundreds of thousands of document-summary pairs is critical to the recent success of deep learning techniques for abstractive summarization. Unfortunately, in most domains (other than news) such training data is not available and cannot be easily sourced. For instance, manually writing opinion summaries is practically impossible since an annotator must read all available reviews for a given product or service which can be prohibitively many. Moreover, different types of products impose different restrictions on the summaries which might vary in terms of length, or the types of aspects being mentioned, rendering the application of transfer learning techniques (Pan and Yang, 2010) problematic.
Motivated by these issues, Chu and Liu (2019) consider an unsupervised learning setting where
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1934-1945 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
There are only documents (product or business reviews) available without corresponding summaries. They propose an end-to-end neural model to perform abstractive summarization based on (a) an autoencoder that learns representations for each review and (b) a summarization module which takes the aggregate encoding of reviews as input and learns to generate a summary which is semantically similar to the source documents. Due to the absence of ground truth summaries, the model is not trained to reconstruct the aggregate encoding of reviews, but rather it only learns to reconstruct the encoding of individual reviews. As a result, it may not be able to generate meaningful text when the number of reviews is large. Furthermore, autoencoders are constrained to use simple decoders lacking attention (Bahdanau et al., 2014) and copy (Vinyals et al., 2015) mechanisms which have proven useful in the supervised setting leading to the generation of informative and detailed summaries. Problematically, a powerful decoder might be detrimental to the reconstruction objective, learning to express arbitrary distributions of the output sequence while ignoring the encoded input (Kingma and Welling, 2014; Bowman et al., 2016).
In this paper, we enable the use of supervised techniques for unsupervised summarization. Specifically, we automatically generate a synthetic training dataset from a corpus of product reviews, and use this dataset to train a more powerful neural model with supervised learning. The synthetic data is created by selecting a review from the corpus, pretending it is a summary, generating multiple noisy versions thereof and treating these as pseudo-reviews. The latter are obtained with two noise generation functions targeting textual units of different granularity: segment noising introduces noise at the word- and phrase-level, while document noising replaces a review with a semantically similar one. We use the synthetic data to train a neural model that learns to denoise the pseudo-reviews and generate the summary. This is motivated by how humans write opinion summaries, where denoising can be seen as removing diverging information. Our proposed model consists of a multi-source encoder and a decoder equipped with an attention mechanism. Additionally, we introduce three modules: (a) explicit denoising guides how the model removes noise from the input encodings, (b) partial copy enables to copy information from the source reviews only when necessary, and (c) a discriminator helps
the decoder generate topically consistent text.
We perform experiments on two review datasets representing different domains (movies vs businesses) and summarization requirements (short vs longer summaries). Results based on automatic and human evaluation show that our method outperforms previous unsupervised summarization models, including the state-of-the-art abstractive system of Chu and Liu (2019) and is on the same par with a state-of-the-art supervised model (Wang and Ling, 2016) trained on a small sample of (genuine) review-summary pairs.
Most previous work on unsupervised opinion summarization has focused on extractive approaches (Carenini et al., 2006; Ku et al., 2006; Paul et al., 2010; Angelidis and Lapata, 2018) where a clustering model groups opinions of the same aspect, and a sentence extraction model identifies text representative of each cluster. Ganesan et al. (2010) propose a graph-based abstractive framework for generating concise opinion summaries, while Di Fabbrizio et al. (2014) use an extractive system to first select salient sentences and then generate an abstractive summary based on hand-written templates (Carenini and Moore, 2006).
As mentioned earlier, we follow the setting of Chu and Liu (2019) in assuming that we have access to reviews but no gold-standard summaries. Their model learns to generate opinion summaries by reconstructing a canonical review of the average encoding of input reviews. Our proposed method is also abstractive and neural-based, but eschews the use of an autoencoder in favor of supervised sequence-to-sequence learning through the creation of a synthetic training dataset. Concurrently with our work, Bražinskas et al. (2019) use a hierarchical variational autoencoder to learn a latent code of the summary. While they also use randomly sampled reviews for supervised training, our dataset construction method is more principled making use of linguistically motivated noise functions.
Our work relates to denoising autoencoders (DAEs; Vincent et al., 2008), which have been effectively used as unsupervised methods for various NLP tasks. Earlier approaches have shown that DAEs can be used to learn high-level text representations for domain adaptation (Glorot et al., 2011) and multimodal representations of textual and visual input (Silberer and Lapata, 2014). Recent
work has applied DAEs to text generation tasks, specifically to data-to-text generation (Freitag and Roy, 2018) and extractive sentence compression (Fevry and Phang, 2018). Our model differs from these approaches in two respects. Firstly, while previous work has adopted trivial noising methods such as randomly adding or removing words (Fevry and Phang, 2018) and randomly corrupting encodings (Silberer and Lapata, 2014), our noise generators are more linguistically informed and suitable for the opinion summarization task. Secondly, while in Freitag and Roy (2018) the decoder is limited to vanilla RNNs, our noising method enables the use of more complex architectures, enhanced with attention and copy mechanisms, which are known to improve the performance of summarization systems (Rush et al., 2015; See et al., 2017).
3 Modeling Approach
Let  $\mathbf{X} = \{x_{1},\dots,x_{N}\}$  denote a set of reviews about a product (e.g., a movie or business). Our aim is to generate a summary  $y$  of the opinions expressed in  $\mathbf{X}$ . We further assume access to a corpus  $\mathbb{C} = \{\mathbf{X}_1,\dots,\mathbf{X}_M\}$  containing multiple reviews about  $M$  products without corresponding opinion summaries.
Our method consists of two parts. We first create a synthetic dataset  $\mathbb{D} = \{(\mathbf{X},y)\}$  consisting of summary-review pairs. Specifically, we sample review  $x_{i}$  from  $\mathbb{C}$ , pretend it is a summary, and generate multiple noisy versions thereof (i.e., pseudo-reviews). At training time, a denoising model learns to remove the noise from the reviews and generate the summary. At test time, the same denoising model is used to summarize actual reviews. We use denoising as an auxiliary task for opinion summarization to simulate the fact that summaries tend to omit opinions that do not represent consensus (i.e., noise in the pseudo-review), but include salient opinions found in most reviews (i.e., non-noisy parts of the pseudo-review).
3.1 Synthetic Dataset Creation via Noising
We sample a review as a candidate summary and generate noisy versions thereof, using two functions: (a) segment noisig add noise at the token and chunk level, and (b) document noisig add noise at the text level. The noise functions are illustrated in Figure 1.
Summary Sampling Summaries and reviews follow different writing conventions. For exam-
Figure 1: Synthetic dataset creation. Given a sampled candidate summary, we add noise using two methods: (a) segment noisng performs token- and chunk-level alterations, and (b) document noisng replaces the text with a semantically similar review.
ple, reviews are subjective, and often include first-person singular pronouns such as  $I$  and my and several unnecessary characters or symbols. They may also vary in length and detail. We discard reviews from corpus  $\mathbb{C}$  which display an excess of these characteristics based on a list of domain-specific constraints (detailed in Section 4). We sample a review  $y$  from the filtered corpus, which we use as the candidate summary.
Segment Noising Given candidate summary  $y = \{w_{1},\dots,w_{L}\}$ , we create a set of segment-level noisy versions  $\mathbf{X}^{(c)} = \{x_1^{(c)},\dots,x_N^{(c)}\}$ . Previous work has adopted noisng techniques based on random  $n$ -gram alterations (Fevry and Phang, 2018), however, we instead rely on two simple, linguistically informed noise functions. Firstly, we train a bidirectional language model (BiLM; Peters et al., 2018) on the review corpus  $\mathbb{C}$ . For each word in  $y$ , the BiLM predicts a softmax word distribution which can be used to replace words. Secondly, we utilize FLAIR (Akbik et al., 2019), an off-the-shelf state-of-the-art syntactic chunker that leverages contextual embeddings, to shallow parse each review  $r$  in corpus  $\mathbb{C}$ . This results in a list of chunks  $\mathbf{C}_r = \{c_1,\dots,c_K\}$  with corresponding syntactic labels  $\mathbf{G}_r = \{g_1,\dots,g_K\}$  for each review  $r$ , which we use for replacing and rearranging chunks.
Segment-level noise involves token- and chunk-
1https://github.com/zalandoresearch/flair
level alterations. Token-level alterations are performed by replacing tokens in  $y$  with probability  $p^{\mathcal{R}}$ . Specifically, we replace token  $w_{j}$  in  $y$ , by sampling token  $w_{j}^{\prime}$  from the BiLM predicted word distribution (see in Figure 1). We use nucleus sampling (Holtzman et al., 2019), which samples from a rescaled distribution of words with probability higher than a threshold  $p^{\mathcal{N}}$ , instead of the original distribution. This has been shown to yield better samples in comparison to top-  $k$  sampling, mitigating the problem of text degeneration (Holtzman et al., 2019).
Chunk-level alterations are performed by removing and inserting chunks in  $y$ , and rearranging them based on a sampled syntactic template. Specifically, we first shallow parse  $y$  using FLAIR, obtaining a list of chunks  $\mathbf{C}_y$ , each of which is removed with probability  $p^{\mathcal{R}}$ . We then randomly sample a review  $r$  from our corpus and use its sequence of chunk labels  $\mathbf{G}_r$  as a syntactic template, which we fill in with chunks in  $\mathbf{C}_y$  (sampled without replacement), if available, or with chunks in corpus  $\mathbb{C}$  otherwise. This results in a noisy version  $x^{(c)}$  (see Figure 1 for an example). Repeating the process  $N$  times produces the noisy set  $\mathbf{X}^{(c)}$ . We describe this process step-by-step in the Appendix.
Document Noising Given candidate summary  $y = \{w_{1},\dots ,w_{L}\}$ , we also create another set of document-level noisy versions  $\mathbf{X}^{(d)} = \{x_1^{(d)},\ldots ,x_N^{(d)}\}$ . Instead of manipulating parts of the summary, we altogether replace it with a similar review from the corpus and treat it as a noisy version. Specifically, we select  $N$  reviews that are most similar to  $y$  and discuss the same product. To measure similarity, we use IDF-weighted ROUGE-1 F1 (Lin, 2004), where we calculate the lexical overlap between the review and the candidate summary, weighted by token importance:
$$
\begin{array}{l} \operatorname {o v e r l a p} = \sum_ {w _ {j} \in x} \left(\operatorname {I D F} \left(w _ {j}\right) * 1 \left(w _ {j} \in y\right)\right) \\ \mathrm {P} = \text {o v e r l a p} / | x | \quad \mathrm {R} = \text {o v e r l a p} / | y | \\ \mathrm {F} _ {1} = (2 * \mathrm {P} * \mathrm {R}) / (\mathrm {P} + \mathrm {R}) \\ \end{array}
$$
where  $x$  is a review in the corpus,  $1(\cdot)$  is an indicator function, and  $\mathrm{P},\mathrm{R}$ , and  $\mathrm{F_1}$  are the ROUGE-1 precision, recall, and  $\mathrm{F_1}$ , respectively. The reviews with the highest  $\mathrm{F_1}$  are selected as noisy versions of  $y$ , resulting in the noisy set  $\mathbf{X}^{(d)}$  (see Figure 1).
We create a total of  $2*N$  noisy versions of  $y$ , i.e.,  $\mathbf{X} = \mathbf{X}^{(c)}\cup \mathbf{X}^{(d)}$  and obtain our synthetic train-
ing data  $\mathbb{D} = \{(\mathbf{X},y)\}$  by generating  $|\mathbb{D}|$  pseudo-review-summary pairs. Both noisng methods are necessary to achieve aspect diversity amongst input reviews. Segment noisng creates reviews which may mention aspects not found in the summary, while document noisng creates reviews with content similar to the summary. Relying on either noise function alone decreases performance (see the ablation studies in Section 5). We show examples of these noisy versions in the Appendix.
3.2 Summarization via Denoising
We summarize (aka denoise) the input  $\mathbf{X}$  with our model which we call DENOISESUM, illustrated in Figure 2. A multi-source encoder produces an encoding for each pseudo-review. The encodings are further corrected via an explicit denoising module, and then fused into an aggregate encoding for each type of noise. Finally, the fused encodings are passed to a decoder with a partial copy mechanism to generate the summary  $y$ .
Multi-Source Encoder For each pseudo-review  $x_{j} \in \mathbf{X}$  where  $x_{j} = \{w_{1},\dots,w_{L}\}$  and  $w_{k}$  is the  $k$ th token in  $x_{j}$ , we obtain contextualized token encodings  $\{h_k\}$  and an overall review encoding  $d_{j}$  with a BiLSTM encoder (Hochreiter and Schmidhuber, 1997):
$$
\begin{array}{l} \overrightarrow {h} _ {k} = \operatorname {L S T M} _ {f} \left(w _ {k}, \overrightarrow {h} _ {k - 1}\right) \\ \overleftarrow {h} _ {k} = \operatorname {L S T M} _ {b} \left(w _ {k}, \overleftarrow {h} _ {k + 1}\right) \\ h _ {k} = \left[ \overrightarrow {h} _ {k}; \overleftarrow {h} _ {k} \right] \\ d _ {j} = [ \overrightarrow {h} _ {L}; \overleftarrow {h} _ {1} ] \\ \end{array}
$$
where  $\vec{h}_k$  and  $\vec{h}_k$  are forward and backward hidden states of the BiLSTM at timestep  $k$ , and; denotes concatenation (see module (a) in Figure 2).
Explicit Denoising The model should be able to remove noise from the encodings before decoding the text. While previous methods (Vincent et al., 2008; Freitag and Roy, 2018) implicitly assign the denoising task to the encoder, we propose an explicit denoising component (see module (b) in Figure 2). Specifically, we create a correction vector  $c_{j}^{(c)}$  for each pseudo-review  $d_{j}^{(c)}$  which resulted from the application of segment noise.  $c_{j}^{(c)}$  represents the adjustment needed to denoise each dimension of  $d_{j}^{(c)}$  and is used to create  $\hat{d}_{j}^{(c)}$ , a denoised
Figure 2: Architecture of DENOISESUM: it consists of a multi-source encoder with explicit denoising, noise-specific fusion, a decoder with partial copy, and a review category classifier.
encoding of  $d_j^{(c)}$ :
$$
q = \sum_ {j = 1} ^ {N} d _ {j} ^ {(c)} / N
$$
$$
c _ {j} ^ {(c)} = \tanh (W _ {d} ^ {(c)} [ d _ {j} ^ {(c)}; q ] + b _ {d} ^ {(c)})
$$
$$
\hat {d} _ {j} ^ {(c)} = d _ {j} ^ {(c)} + c _ {j} ^ {(c)}
$$
where  $q$  represents a mean review encoding and functions as a query vector,  $W$  and  $b$  are learned parameters, and superscript  $(c)$  signifies segment noising. We can interpret the correction vector as removing or adding information to each dimension when its value is negative or positive, respectively. Analogously, we obtain  $\hat{d}_j^{(d)}$  for pseudo-reviews  $d_j^{(d)}$  which have been created with document noising.
Noise-Specific Fusion For each type of noise (segment and document), we create a noise-specific aggregate encoding by fusing the denoised encodings into one (see module (c) in Figure 2). Given  $\{\hat{d}_j^{(c)}\}$ , the set of denoised encodings corresponding to segment noisy inputs, we create aggregate encoding  $s_0^{(c)}$ :
$$
\alpha_ {j} ^ {(c)} = \operatorname {s o f t m a x} \left(W _ {f} ^ {(c)} \hat {d} _ {j} ^ {(c)} + b _ {f} ^ {(c)}\right)
$$
$$
s _ {0} ^ {(c)} = \sum_ {j} \hat {d} _ {j} ^ {(c)} * \alpha_ {j} ^ {(c)}
$$
where  $\alpha_{j}$  is a gate vector with the same dimensionality as the denoised encodings. Analogously,
we obtain  $s_0^{(d)}$  from the denoised encodings  $\{\hat{d}_j^{(d)}\}$  corresponding to document noisy inputs.
Decoder with Partial Copy Our decoder generates a summary given encodings  $s_0^{(c)}$  and  $s_0^{(d)}$  as input. An advantage of our method is its ability to incorporate techniques used in supervised models, such as attention (Bahdanau et al., 2014) and copy (Vinyals et al., 2015). Pseudo-reviews created using segment noisings include various chunk permutations, which could result in ungrammatical and incoherent text. Using a copy mechanism on these texts may hurt the fluency of the output. We therefore allow copy on document noisy inputs only (see module (d) in Figure 2).
We use two LSTM decoders for the aggregate encodings, one equipped with attention and copy mechanisms, and one without copy mechanism. We then combine the results of these decoders using a learned gate. Specifically, token  $w_{t}$  at timestep  $t$  is predicted as:
$$
\begin{array}{l} s _ {t} ^ {(c)}, p ^ {(c)} (w _ {t}) = \operatorname {L S T M} _ {\text {a t t}} \left(w _ {t - 1}, s _ {t - 1} ^ {(c)}\right) \\ s _ {t} ^ {(d)}, p ^ {(d)} (w _ {t}) = \mathrm {L S T M} _ {\mathrm {a t t + c o p y}} (w _ {t - 1}, s _ {t - 1} ^ {(d)}) \\ \lambda_ {t} = \sigma (W _ {p} [ w _ {t - 1}; s _ {t} ^ {(c)}; s _ {t} ^ {(d)} ] + b _ {p}) \\ p (w _ {t}) = \lambda_ {t} * p ^ {(c)} (w _ {t}) + (1 - \lambda_ {t}) * p ^ {(d)} (w _ {t}) \\ \end{array}
$$
where  $s_t$  and  $p(w_t)$  are the hidden state and predicted token distribution at timestep  $t$ , and  $\sigma(\cdot)$  is the sigmoid function.
We use a maximum likelihood loss to optimize the generation probability distribution based on summary  $y = \{w_{1},\dots,w_{L}\}$  from our synthetic dataset:
$$
\mathcal {L} _ {g e n} = - \sum_ {w _ {t} \in y} \log p (w _ {t})
$$
The decoder depends on  $\mathcal{L}_{gen}$  to generate meaningful, denoised outputs. As this is a rather indirect way to optimize our denoising module, we additionally use a discriminative loss providing direct supervision. The discriminator operates at the output of the fusion module and predicts the category distribution  $p(z)$  of the output summary  $y$  (see module (e) in Figure 2). The type of categories varies across domains. For movies, categories can be information about their genre (e.g., drama, comedy), while for businesses their specific type (e.g., restaurant, beauty parlor). This information is often included in reviews but we assume otherwise and use an LDA topic model (Blei et al., 2003) to infer  $p(z)$  (we present experiments with human labeled and automatically induced categories in Section 5). An MLP classifier takes as input aggregate encodings  $s^{(c)}$  and  $s^{(d)}$  and infers  $q(z)$ . The discriminator is trained by calculating the KL divergence between predicted and actual category distributions  $q(z)$  and  $p(z)$ :
$$
q (z) = \operatorname {M L P} _ {d} \left(s ^ {(c)}, s ^ {(d)}\right)
$$
$$
\mathcal {L} _ {d i s c} = D _ {K L} (p (z) \parallel q (z))
$$
The final objective is the sum of both loss functions:
$$
\mathcal {L} = \mathcal {L} _ {g e n} + \mathcal {L} _ {d i s c}
$$
At test time, we are given genuine reviews  $\mathbf{X}$  as input instead of the synthetic ones. We generate a summary by treating  $\mathbf{X}$  as  $\mathbf{X}^{(c)}$  and  $\mathbf{X}^{(d)}$ , i.e., the outcome of segment and document noising.
Dataset We performed experiments on two datasets which represent different domains and summary types. The Rotten Tomatoes dataset $^2$  (Wang and Ling, 2016) contains a large set of reviews for various movies written by critics. Each set of reviews has a gold-standard consensus summary written by an editor. We follow the partition
Rotten Tomatoes	Train*	Dev	Test
#movies	25k	536	737
reviews/movie	40.0	98.0	100.3
tokens/review	28.4	23.5	23.6
tokens/summary	22.7	23.6	23.8
corpus size	245,848
Yelp	Train*	Dev	Test
#businesses	100k	100	100
#reviews/business	8.0	8.0	8.0
tokenizer/review	72.3	70.3	67.8
tokenizer/summary	64.8	70.9	67.3
corpus size	2,320,800
Table 1: Dataset statistics; Train* column refers to the synthetic data we created through noising (Section 3.1).