SlowGuess/ACL_2020_OCR huggingface dataset:
Raphael Schumann<sup>1</sup>, Lili Mou<sup>2</sup>, Yao Lu<sup>3</sup>, Olga Vechtomova<sup>3</sup>, Katja Markert<sup>1</sup>
$^{1}$ Institute of Computational Linguistics, Heidelberg University, Germany {rschuman, markert}@cl.uni-heidelberg.de
$^{2}$ University of Alberta, Canada; Alberta Machine Intelligence Institute (Amii)
doublepower.mou@gmail.com
<sup>3</sup>University of Waterloo, Canada
{yao.lu, ovechtom}@uwaterloo.ca
Automatic sentence summarization produces a shorter version of a sentence, while preserving its most important information. A good summary is characterized by language fluency and high information overlap with the source sentence. We model these two aspects in an unsupervised objective function, consisting of language modeling and semantic similarity metrics. We search for a high-scoring summary by discrete optimization. Our proposed method achieves a new state-of-the-art for unsupervised sentence summarization according to ROUGE scores. Additionally, we demonstrate that the commonly reported ROUGE F1 metric is sensitive to summary length. Since this is unwillingly exploited in recent work, we emphasize that future evaluation should explicitly group summarization systems by output length brackets. $^{1}$
Sentence summarization transforms a long source sentence into a short summary, while preserving key information (Rush et al., 2015). Sentence summarization has wide applications, for example, news headline generation and text simplification.
State-of-the-art sentence summarization systems are based on sequence-to-sequence neural networks (Rush et al., 2015; Nallapati et al., 2016; Wang et al., 2019), which require massive parallel data for training. Therefore, unsupervised sentence summarization has recently attracted increasing interest. Cycle-consistency approaches treat the summary as a discrete latent variable and use it to reconstruct the source sentence (Wang and Lee, 2018; Baziotis et al., 2019). Such latent-space generation fails to explicitly model the resemblance between the source sentence and the target summary.
Figure 1: Summarizing a sentence  $\mathbf{x}$  by hill climbing. Each row is a Boolean vector  $\mathbf{a}_t$  at a search step  $t$ . A black cell indicates a word is selected, and vice versa. Randomly swapping two values in the Boolean vector yields a new summary that is scored by an objective function that measures language fluency and semantic similarity. If the new summary increases the objective, this summary is accepted as the current best solution. Rejected solutions are not depicted.
Zhou and Rush (2019) propose a left-to-right beam search approach based on a heuristically defined scoring function. However, beam search is biased towards the first few words of the source.
In this paper, we propose a hill-climbing approach to unsupervised sentence summarization, directly extracting words from the source sentence. This is motivated by the observation that human-written reference summaries exhibit high word overlap with the source sentence, even preserving word order to a large extent. To perform word extraction for summarization, we define a scoring function — similar to Miao et al. (2019) and Zhou and Rush (2019) — that evaluates the quality of a candidate summary by language fluency, semantic similarity to the source, and a hard constraint on output length. We search towards our scoring function by first choice hill-climbing (FCHC), shown in Figure 1. We start from a random subset of words of the required output length. For each search step, a new candidate is sampled by randomly swapping
1Our code and system outputs are available at: https://github.com/raphael-sch/HC_Sentence_Summarization
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5032-5042 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
a selected word and a non-selected word. We accept the new candidate if its score is higher than the current one. In contrast to beam search (Zhou and Rush, 2019), our summary is not generated sequentially from the beginning of a sentence, and therefore not biased towards the first few words.
Due to the nature of the search action, our approach is able to explicitly control the length of a summary as a hard constraint. In all previous work, the summary length is weakly controlled by length embeddings or a soft length penalty (Zhou and Rush, 2019; Wang and Lee, 2018; Fevry and Phang, 2018; Baziotis et al., 2019). Thus, the generated summaries by different systems vary considerably in average length, for example, ranging from 9 to 15 on a headline corpus (Section 4.1). Previous work uses ROUGE F1 to compare summaries that might differ in length. We show that ROUGE F1 is unfortunately sensitive to summary output length, in general favoring models that produce longer summaries. Therefore, we argue that controlling the output length should be an integral part of the summarization task and that a fair system comparison can only be conducted between summaries in the same length bracket.
Our model establishes a new state-of-the-art for unsupervised sentence summarization across all commonly-used length brackets and different ROUGE metrics on the Gigaword dataset for headline generation (Rush et al., 2015) and on DUC2004 (Over and Yen, 2004).
- We propose a novel method for unsupervised sentence summarization by hill climbing with word-level extraction.
- We outperform current unsupervised sentence summarization systems, including more complex sentence reconstruction models.
- We show that ROUGE F1 is sensitive to summary length and thus emphasize the importance of explicitly controlling summary length for a fair comparison among different summarization systems.
Text Summarization. The task can be categorized by source text types, such as multi-document summarization (Erkan and Radev, 2004; Radev et al., 2000; Haghighi and Vanderwende, 2009) and single-document summarization (Mihalcea and Tarau, 2004; Zhou and Hovy, 2004; Zheng
and Lapata, 2019). Traditional approaches are mostly extractive, i.e., they extract entire sentences from a document. Recently, sequence-to-sequence (Seq2Seq) models have been used for abstractive summaries, where the system is able to synthesize new sentences (Nallapati et al., 2016, 2017; Gehrmann et al., 2018; Lewis et al., 2019; Fabbri et al., 2019). The copy mechanism (Gu et al., 2016) in a Seq2Seq model can be viewed as word-level extraction in abstractive summarization (See et al., 2017; Paulus et al., 2018). Both state-of-the-art extractive and abstractive approaches are usually supervised.
Sentence summarization yields a short summary for a long sentence. Hori and Furui (2004) and Clarke and Lapata (2006) extract single words from the source sentence based on language model fluency and linguistic constraints. They search via dynamic programming with a trigram language model, which restricts the model capacity. The Hedge Trimmer method (Dorr et al., 2003) also uses hand-crafted linguistic rules to remove constituents from a parse tree until a certain length is reached.
Rush et al. (2015) propose a supervised abstractive sentence summarization system with an attention mechanism (Bahdanau et al., 2015), and they also introduce a dataset for headline generation derived from Gigaword. Subsequent models for this dataset were also supervised and mostly based on Seq2seq architectures (Nallapati et al., 2016; Chopra et al., 2016; Wang et al., 2019).
Recently, unsupervised approaches for sentence summarization have attracted increasing attention. Fevry and Phang (2018) learn a denoising autoencoder and control the summary length by a length embedding. Wang and Lee (2018) and Baziotis et al. (2019) use cycle-consistency (He et al., 2016) to learn the reconstruction of the source sentence and return the intermediate discrete representation as a summary. Zhou and Rush (2019) use beam search to optimize a scoring function, which considers language fluency and contextual matching.
Our work can be categorized under unsupervised sentence summarization. We accomplish this by word-level extraction from the source sentence.
Constrained Sentence Generation. Neural sentence generation is usually accomplished in an autoregressive way, for example, by recurrent neu
$^{2}$ https://catalog.ldc.upenn.edu/LDC2003T05
ral networks generating words left-to-right. This is often enhanced by beam search (Sutskever et al., 2014), which keeps a beam of candidates in a partially greedy fashion. A few studies allow hard constraints on this decoding procedure. Hokamp and Liu (2017) use grid-beam search to impose lexical constraints during decoding. Anderson et al. (2017) propose constrained beam search to predict fixed image tags in an image transcription task. Miao et al. (2019) propose a Metropolis-Hastings sampler for sentence generation, where hard constraints can be incorporated into the target distribution. This is further extended to simulated annealing (Liu et al., 2020), or applied to the text simplification task (Kumar et al., 2020). Different from the above concurrent work, this paper applies the stochastic search framework to text summarization, and design our specific search space and search actions for word extraction.
In previous work on text summarization, length embeddings (Kikuchi et al., 2016; Fan et al., 2018) have been used to indicate the desired summary length. However, these are not hard constraints, because the model may learn to ignore such information.
3 Proposed Model
Given a source sentence  $\mathbf{x} = (x_{1}, x_{2}, \ldots, x_{n})$  as input, our goal is to generate a shorter sentence  $\mathbf{y} = (y_{1}, y_{2}, \ldots, y_{m})$  as a summary of  $\mathbf{x}$ . We perform word-level extraction, in addition keeping the original word order intact. Thus,  $\mathbf{y}$  is a subsequence of  $\mathbf{x}$ . Our word-level extraction optimizes a manually defined objective function  $f(\mathbf{y}; \mathbf{x}, s)$ , where the summary length  $s$  is predefined ( $s < n$ ) and not subject to optimization. In the remainder of this section, we will describe the objective function, search space, and the search algorithm in detail.
3.1 Search Objective
We define an objective function  $f(\mathbf{y}; \mathbf{x}, s)$ , which our algorithm maximizes. It evaluates the fitness of a candidate sentence  $\mathbf{y}$  as the summary of an input  $\mathbf{x}$ , involving three aspects, namely, language fluency  $f_{\overleftrightarrow{\mathrm{LM}}}(\mathbf{y})$ , semantic similarity  $f_{\mathrm{SIM}}(\mathbf{y}; \mathbf{x})$ , and a length constraint  $f_{\mathrm{LEN}}(\mathbf{y}, s)$ . This is given by
$$
f (\mathbf {y}; \mathbf {x}, s) = f _ {\overline {{\mathrm {L M}}}} (\mathbf {y}) \cdot f _ {\mathrm {S I M}} (\mathbf {y}; \mathbf {x}) ^ {\gamma} \cdot f _ {\mathrm {L E N}} (\mathbf {y}; s), \tag {1}
$$
where the relative weight  $\gamma$  balances  $f_{\mathrm{LM}}^{\leftrightarrow}(\mathbf{y})$  and  $f_{\mathrm{SIM}}(\mathbf{y};\mathbf{x})$ . We treat the summary length as a hard
constraint, and therefore we do not need a weighting hyperparameter for  $f_{\mathrm{LEN}}$ .
Language Fluency. The language fluency scorer quantifies how grammatical and idiomatic a candidate summary  $\mathbf{y}$  is. Our model generates a candidate summary in a non-autoregressive fashion, in contrast to the beam search in Zhou and Rush (2019). Thus, we are able to simultaneously consider forward and backward language models, using the geometric average of their perplexities. Using both forward and backward language models is less biased towards sentence beginnings or endings.
$$
\overleftrightarrow {\mathrm {P P L}} (\mathbf {y}) = \sqrt [ 2 | \mathbf {y} | ]{\prod_ {i} ^ {| \mathbf {y} |} \frac {1}{p _ {\overleftrightarrow {\mathrm {L M}}} (y _ {i} | \mathbf {y} <   i)} \prod_ {i} ^ {| \mathbf {y} |} \frac {1}{p _ {\overleftrightarrow {\mathrm {L M}}} (y _ {i} | \mathbf {y} > i)}}.
$$
Our fluency scorer is the inverse perplexity.
$$
f _ {\overrightarrow {\mathrm {L M}}} (\mathbf {y}) = \overleftrightarrow {\overrightarrow {\mathrm {P P L}}} (\mathbf {y}) ^ {- 1}. \tag {2}
$$
Depending on applications, the language models could be pretrained on a target corpus. In this case, the fluency scorer also measures whether the summary style is consistent with the target language. This could be important in certain applications, e.g., headline generation, where the summary language differs from the input in style.
Semantic Similarity. A semantic similarity scorer ensures that the summary keeps the key information of the input sentence. We adopt the cosine similarity between sentence embeddings as
$$
f _ {\mathrm {S I M}} (\mathbf {y}; \mathbf {x}) = \cos (e (\mathbf {x}), e (\mathbf {y})), \tag {3}
$$
where  $e$  is a sentence embedding method. In our work, we use unigram word embeddings learned by the sent2vec model (Pagliardini et al., 2018). Then,  $e(\mathbf{x})$  is computed as the average of these unigram embeddings, weighted by the inverse-document frequency ( $idf$ ) of the words.
We use sent2vec because it is trained in an unsupervised way on individual sentences. By contrast, other unsupervised methods like SiameseCBOW (Kenter et al., 2016) or BERT (Devlin et al., 2019) use adjacent sentences as part of the training signal.
Length Constraint. Our discrete searching approach is able to impose the output length as a hard constraint, allowing the model to generate summaries of any given length. Suppose the desired output length is  $s$ , then our length scorer is
<sup>3</sup>We use the terminology unsupervised summarization, following Zhou and Rush (2019). While we train the language models on the desired target language, we do not need parallel source-target pairs, i.e., sentences together with their groundtruth summaries.
$$
f _ {\text {L E N}} (\mathbf {y}; s) = \left\{ \begin{array}{l l} 1, & \text {i f} | \mathbf {y} | = s, \\ - \infty , & \text {o t h e r w i s e .} \end{array} \right. \tag {4}
$$
In other words, a candidate summary  $\mathbf{y}$  is infeasible if it does not satisfy the length constraint. In practice, we implement this hard constraint by searching among feasible solutions only.
3.2 Search Space
Most sentence generation models choose a word from the vocabulary at each time step, such as autoregressive generation that predicts the next word (Sutskever et al., 2014; Rush et al., 2015), and edit-based generation with deletion or insertion operations (Miao et al., 2019; Dong et al., 2019). In these cases, the search space is  $|\mathcal{V}|^s$ , given a vocabulary  $\mathcal{V}$  and a summary length  $s$ .
However, reference summaries are highly extractive. In the headline generation dataset (Rush et al., 2015), for example,  $45\%$  of the words in the reference summary also appear in the source sentence. This yields a ceiling of 45 ROUGE-1 F1 points for a purely extractive method, which is higher than the current state-of-the-art supervised abstractive result of 39 points (Wang et al., 2019). We are thus motivated to propose our word-extraction approach that extracts a subsequence of the input as the summary. Additionally, we arrange the words in the same order as the input, motivated by the monotonicity assumption in summarization (Yu et al., 2016; Raffel et al., 2017).
Formally, we define the search space as  $\mathbf{a} = (a_{1},\ldots ,a_{n})\in \{0,1\}^{n}$ , where  $n$  is the length of the input sentence  $\mathbf{x}$ . The vector  $\mathbf{a}$  is a Boolean filter over the source words  $\mathbf{x}$ . The summary sequence can then be represented by  $\mathbf{y} = \mathbf{x}_a$ , i.e., we sequentially extract words from the source sequence  $\mathbf{x}$  by the Boolean vector  $\mathbf{a}$ . If  $a_{i} = 1$ , then  $x_{i}$  is extracted for the summary, and vice versa.
Further, we only consider the search space of all feasible solutions  $\{a:f(\mathbf{x}_a;\mathbf{x},s) > - \infty \}$ . That is to say, the candidate summary has to satisfy the length constraint in Section 3.1. Equivalently, the output length can be expressed by a constraint on the search space such that  $\sum_{i}a_{i} = s$ .
The above restrictions reduce the search space to  $\binom{n}{s}$  solutions. In a realistic setting, our search
space is much smaller than that of generating words from the entire vocabulary.
3.3 Search Algorithm
We optimize our objective function  $f(\mathbf{y}; \mathbf{x}, s)$  by first-choice hill climbing (FCHC, Russell and Norvig, 2016). This is a stochastic optimization algorithm that proposes a candidate solution by local change at every search step. The candidate is accepted if it is better than the current solution. Otherwise, the algorithm keeps the current solution. FCHC maximizes the objective function in a greedy fashion and yields a (possibly local) optimum.
Algorithm 1 shows the optimization procedure of our FCHC. For each search step, a new candidate is sampled from the neighbor function  $q(a'|a)$ . This is accomplished by randomly swapping two actions  $a_i$  and  $a_j$  for  $a_i \neq a_j$ , i.e., replacing a word in the summary with a word from the source sentence that is not in the current summary. The order of selected words is kept as in the source sentence. If the candidate solution achieves a higher score, then it is accepted. Otherwise, the candidate is rejected and the algorithm proceeds with the current solution. Our search terminates if it exceeds a predefined budget. The last solution is returned as the summary, as it is also the best-scored candidate due to our greedy algorithm.
One main potential drawback of hill climbing algorithms is that they may get stuck in a local optimum. To alleviate this problem, we restart the algorithm with multiple random initial word selections  $\mathbf{a}_0$  and return the overall best solution. We set the number of restarts as  $\beta_R \cdot ns^2$  and number of search steps as  $\beta_T \cdot ns^2$ , where  $\beta_R$  and  $\beta_T$  are controlling hyperparameters. We design the formula to encourage more search for longer input sentences, but only with a tractable growth: linear for input length and quadratic for summary length. As the summary length is usually much smaller
<sup>4</sup>We assume an extracted summary has the same length as the reference, and  $45\%$  words of the reference are in the original sentence. This gives us a ceiling of  $45\%$  precision and recall.
than the input length, quadratic search is possible. Increasing the number of restarts (and search steps) monotonically improves the scoring function, and thus in practice can be set according to the available search budget.
Other discrete optimization algorithms can be explored for sentence generation, such as simulated annealing (Liu et al., 2020) and genetic algorithms. Our analysis on short sentences (where exhaustive search is tractable) showed that hill climbing with restarts achieves ROUGE scores similar to exhaustive search (Section 5.4).
4 Evaluation Framework
In this section, we will describe the datasets, evaluation metrics, and a widely used baseline (called Lead). Additionally, we report the observation that the commonly used evaluation metric, ROUGE F1, is sensitive to summary length, preferring longer summaries. Thus, we propose to group models with similar output length during evaluation for fair comparison.
We evaluate our models on the dataset provided for DUC2004 Task 1 (Over and Yen, 2004) and a headline generation corpus $^5$  (Rush et al., 2015), both widely adopted in the summarization literature.
The DUC2004 dataset is designed and used for testing only. It consists of 500 news articles, each paired with four human written summaries. We follow Rush et al. (2015) and adopt DUC2004 for sentence summarization by using only the first sentence of an article as input. The reference summaries are around 10 words long on average.
The headline generation dataset (Rush et al., 2015) is derived from the Gigaword news corpus. Each headline/title is viewed as the reference summary of the first sentence of an article. The dataset contains 3.8M training instances and 1951 test instances. The average headline contains  $\sim 8$  words; the average source sentence contains  $\sim 30$  words. We use 500 held-out validation instances for hyperparameter tuning. Note that the training set is only used to train a language model and sent2vec embeddings. The summarization process itself is not trained in our approach.
4.2 Lead Baselines
Lead baselines are a strong competitor that extracts the first few characters or words of the input sentence. The DUC2004 shared task includes a Lead baseline, which extracts the first 75 characters as the summary. We call it Lead-C-75. For the Gigaword dataset, the reference has 8 words on average, and it is common to compare with a Lead variant that chooses the first 8 words. We call this baseline Lead-N-n when we choose  $n$  words. For fair comparison with previous work (Baziotis et al., 2019; Fevry and Phang, 2018) in Section 5.2, we further introduce a new variant that returns the first  $p$  percent of source words as the summary. We denote this baseline by Lead-P-p.
4.3 ROUGE Scores
Summarization systems are commonly evaluated by ROUGE scores (Lin, 2004). The ROUGE-1 (or ROUGE-2) score computes the unigram (or bigram) overlap of a generated summary and the reference. ROUGE-L calculates the longest common subsequence. Depending on the dataset, either ROUGE Recall or ROUGE F1 variant is adopted. Since the ROUGE Recall metric is not normalized with regard to length, DUC2004 standard evaluation truncates the summary at 75 characters. This procedure was also adopted by Rush et al. (2015) for the headline generation task, but later Chopra et al. (2016) proposed to report the "more balanced" ROUGE F1 metric for the Gigaword headline generation dataset and abandoned truncation. We follow previous work and use ROUGE F1 for headline generation and truncated ROUGE Recall for DUC2004.
4.4 Summary Length
As mentioned, ROUGE F1 was introduced to the evaluation of sentence summarization to better compare models with different output lengths (Chopra et al., 2016; Nallapati et al., 2016). To investigate the effect of summary length on ROUGE F1, we calculate ROUGE F1 scores for the Lead-N-n and Lead-P-p baselines with different length parameters. Figure 2 shows that ROUGE F1 peaks at  $n \approx 18$  or  $p \approx 50$ . The difference between the maximum performance at  $n \approx 18$  and the widely adopted baseline (Lead-N-8) is large: 4.2 ROUGE-1 F1 points. A similar effect is observed by Sun et al. (2019) for document summarization. This shows that ROUGE F1 is still sensitive to summary length, and this effect should be
<sup>5</sup>https://github.com/harvardnlp/NAMAS
Figure 2: ROUGE F1 scores on the test set of headline generation for Lead-N and Lead-P baselines with different number  $n$  and percentage  $p$  of leading words.
considered during evaluation. We propose to report the average output length of a model and only compare models in the same length bracket.
We conduct experiments with two settings, dependent on how the scorers  $f_{\overrightarrow{\mathrm{LM}}}$  and  $f_{\mathrm{SIM}}$  are trained. In the first setting, we train the language model and sent2vec embeddings on the source (article) side of the Gigaword headline generation dataset. This complies with Fevry and Phang (2018) and Baziotis et al. (2019). In the second setting, we train the language model and sent2vec embeddings on the target (title) side like Zhou and Rush (2019). In both settings, we do not need parallel source-target pairs.
For output length, our headline generation experiment sets the desired target length as 8 words, 10 words, and  $50\%$  of the input, as these mirror either the average reference summary length or the average output lengths of our competitors (Wang and Lee, 2018; Zhou and Rush, 2019; Fevry and Phang, 2018; Baziotis et al., 2019). For DUC2004, the desired summary length is set to 13 words, because the standard evaluation script truncates after the first 75 characters (roughly 13 words) in the summary.
Our forward and backward language models use long short term memory units (Hochreiter and Schmidhuber, 1997) and are optimized for 50
epochs by stochastic gradient descent. Embeddings and hidden sizes are set to 1024 dimensions.