SlowGuess/NAACL_2021_OCR huggingface dataset:
Kaiqiang Song, $^{1}$  Bingqing Wang, $^{2}$  Zhe Feng, $^{2}$  Fei Liu $^{1}$
<sup>1</sup>University of Central Florida, Orlando, FL
$^{2}$ Robert Bosch LLC, Sunnyvale, CA
kqsong@knights.ucf.edu, {bingqing.wang,zhe.feng2}@us.bosch.com feiliu@cs.ucf.edu
We propose a new approach to generate multiple variants of the target summary with diverse content and varying lengths, then score and select admissible ones according to users' needs. Abstractive summarizers trained on single reference summaries may struggle to produce outputs that achieve multiple desirable properties, i.e., capturing the most important information, being faithful to the original, grammatical and fluent. In this paper, we propose a two-staged strategy to generate a diverse set of candidate summaries from the source text in stage one, then score and select admissible ones in stage two. Importantly, our generator gives a precise control over the length of the summary, which is especially well-suited when space is limited. Our selectors are designed to predict the optimal summary length and put special emphasis on faithfulness to the original text. Both stages can be effectively trained, optimized and evaluated. Our experiments on benchmark summarization datasets suggest that this paradigm can achieve state-of-the-art performance.
The learning objective of a modern abstractive summarizer is to produce system outputs that resemble reference summaries on a word-to-word basis. It does not promote outputs that possess multiple desirable properties, i.e., capturing the most important information, being faithful to the original text, grammatical and fluent, though some of these properties are exhibited by system abstracts as a natural outcome of a learned summarizer (See et al., 2017; Takase et al., 2016; Tan et al., 2017; Chen and Bansal, 2018; Celikyilmaz et al., 2018; Gehrmann et al., 2018; Liu and Lapata, 2019; Lebanoff et al., 2019b; Fabbri et al., 2019; Bražinskas et al., 2020). Without direct optimization of desired properties, system abstracts often change the meaning of the original document or fail to convey the main concepts (Kryscinski et al., 2020).
Police arrested five anti-nuclear protesters Thursday after they sought to disrupt loading of a French Antarctic research and supply vessel, a spokesman for the protesters said.
Police arrest anti-nuclear protesters
$\checkmark$  Protesters target French research ship
X French police arrest five anti-nuclear protesters
Police arrest five anti-nuclear protesters in Antarctica
Police arrest five anti-nuclear protesters at French Antarctic
Table 1: Example of alternative summaries generated from the source text. Admissible summaries are marked by  $\checkmark$ . System summaries that fail to preserve the meaning of the source input are marked by  $\times$ .
In this paper, we propose a new approach to overgenerate and select admissible summaries, which allows a summarizer to juggle multiple objectives and strike a good balance between them (Belz and Reiter, 2006). Our approach consists of two stages. Given a source text, a generator explores the space of all possible lengths to produce multiple variants of the target summary that contain diverse content. We then devise selectors to validate the quality of alternative summaries to predict whether they are admissible. Our selection mechanism can be customized to suit particular needs without changing the generation space. Both stages can be effectively trained, optimized and evaluated.
Crucially, we take a confidence-driven approach to summary generation rather than using a left-to-right order. Beginning writers and language learners do not write in a strict sequential manner. In a similar vein, our generator produces a summary by "filling-in-the-blanks" with appropriate words. The most confident words are generated first, less vital ones later. With confidence-driven generation, our summarizer learns to dynamically add or remove content, and even paraphrase to produce a summary of a given length. In Table 2, we show an example illustrating the difference between our method and left-to-right generation. Our method dramatically enhances the capability of the generator, making it possible to explore summaries of varying lengths.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1392-1404 June 6-11, 2021. ©2021 Association for Computational Linguistics
Source Text: A court here Thursday sentenced a 24-year-old man to 10 years in jail after he admitted pummelling his baby son to death to silence him while watching television.
Left to Right Generation (1 Summary)
Man who
Man who killed [...]
Man who killed baby to hear television better gets 10
Man who killed baby to hear television better gets 10 years	Confidence Driven Generation (4 Summaries)
Man
Man who kill the baby
Man who kill the baby to hear television
Man who kill the baby to hear television better
gets 10 years
gets 10 years
gets 10 years
gets 10 years
gets 10 years
Table 2: An example of the difference between left-to-right and confidence-driven summary generation. (LEFT) A single summary is produced in a left-to-right order. (RIGHT) Four summaries are generated in a confidence-driven mode. The most confident words are generated first, less vital ones later. Our generator learns to dynamically add or remove content given a target length to produce summaries of varying lengths—short, medium and long. The output is a diverse set of alternative summaries.
Identifying admissible summaries with desired properties is critical for a summarizer. Summaries of very short lengths may fail to capture the main concepts, and this kind of incomplete or partial information can lead to false assumptions about the original content. Moreover, summaries of moderate lengths may still contain hallucinated content that is nonexistent in the source text (Maynez et al., 2020). We present two summary selectors to combat these issues. Our first selector aims to predict what summary length is most suitable for a source text, whereas a second selector puts special emphasis on the overall quality of the system summary, in particular its faithfulness to the original text (Falke et al., 2019; Durmus et al., 2020).
A novel dataset has been introduced in this work where we associate a source text with multiple summaries, and admissible ones are manually labelled by human annotators. Not only can the dataset be used to judge the effectiveness of summary selectors, but it provides a new testbed for future summarizers to compare their outputs against multiple reference summaries, which is key to improve the reliability of evaluation results (Louis and Nenkova, 2013). We have focused on generating abstractive summaries from single source sentences, but the insights gained from this study could inform the design of summarizers of all forms. Our method also has a great potential to incorporate human-in-the-loop to teach the model to select the best summary. The main contributions of this paper are:
- We propose a new approach to generate multiple variants of the target summary that have varying lengths, then score and select the best summaries according to our needs.
- Our generator controls over the length of the summary, which is especially well-suited when space is limited. Our selectors are designed to predict the optimal summary length and put special em
phasis on faithfulness to the original text.
- Our experiments on benchmark summarization datasets suggest that this paradigm can surpass results of previous studies or rival state-of-the-art. We conclude with a discussion of our key findings, which has implications for the development of robust abstractive summarizers.<sup>1</sup>
It is important for neural abstractive summarizers to produce summaries that are faithful to the original texts (Cao et al., 2017; Kryscinski et al., 2019; Lebanoff et al., 2019a; Wang et al., 2020; Dong et al., 2020; Zhang et al., 2020b). However, it remains questionable as to whether a summarizer must acquire that ability by learning from human reference summaries, or possibly through external resources such as textual entailment predictions (Falke et al., 2019). In this paper, we present a two-stage strategy to over-generate, then score system summaries externally for faithfulness and overall quality.
Previous work has sought to control various aspects of the generated summary, including the style, length and amount of reused text (Kikuchi et al., 2016; Hu et al., 2017; Fan et al., 2018; Keskar et al., 2019; Makino et al., 2019; Song et al., 2020). In contrast, our generator focuses on producing multiple variants of the target summary that have diverse content and varying lengths. It offers precise control over the length of the summary, which has an important implication for fair comparison between different summarization systems (Napoles et al., 2011; Shapira et al., 2018).
Our methodology allows for greater flexibility in designing summary selectors. The selectors may allow multiple admissible summaries to be identi
1Our code and annotated data are made available on Github at https://github.com/ucfnlpvarying-length-summ
Figure 1: An illustration of the generation process. A sequence of placeholders ("[MASK]") are placed following the source text. Our model simultaneously predicts the most probable tokens for all positions, rather than predicting only the most probable next token in an autoregressive setting. We obtain the token that has the highest probability, and use it to replace the [MASK] token of that position. Next, the model makes new predictions for all remaining positions, conditioned on the source text and all summary tokens seen thus far. Our generator produces a summary having the exact given length and with a proper endpoint.
fied for any source input according to users' needs. On the contrary, post-editing of system summaries through a set of basic operations such as insertion and deletion (Gu et al., 2019; Malmi et al., 2019; Dong et al., 2019b; Correia and Martins, 2019) may have intrinsic limitations by learning from single reference summaries to produce single outputs. In this paper, we provide a new dataset where each source text is associated with multiple admissible summaries to encourage diverse outputs.
Our generator is inspired by unsupervised pretraining of deep neural models (Peters et al., 2018; Radford et al., 2019; Devlin et al., 2019; Yan et al., 2020; Zhang et al., 2020a; Lewis et al., 2020) and non-autoregressive machine translation (Gu et al., 2018; Ghazvininejad et al., 2019). Distinct from these is our confidence-driven generation that goes beyond left-to-right order. It uses a denoising objective during training and is conveniently transformed into a semi-autoregressive generator at test time. We introduce a customized beam search algorithm to promote the generation of diverse outputs. In the following section, we describe in detail our two-step strategy.
3 A Confidence-Driven Generator
We seek to produce a highly diverse set of alternative summaries from any source input, but standard neural language generators with beam search only produce high-likelihood sequences rather than diverse ones (Ippolito et al., 2019). To address this limitation, we devise a new generator that is capable of producing summaries of varying lengths. A long summary can cover more important information of the source text, whereas a short summary is easy-to-read. Moreover, it produces a summary having the exact given length and with a proper
endpoint. This is achieved by shifting away from left-to-right generation but building a summary using a confidence-driven approach.
Our generator is illustrated in Figure 1. To generate a summary of  $L$  tokens, we place a number of [MASK] tokens following the source text, which serve as "placeholders" for summary tokens. Importantly, our generator simultaneously predicts the most probable tokens for all positions, as opposed to predicting only the most probable next token in an autoregressive setting. We obtain the token that has the highest probability across all positions, and use it to replace the [MASK] token of that position. Next, the model continues to make predictions for all remaining positions, conditioned on the source text and the summary tokens seen thus far of varying positions.
Let  $\mathbf{x} = \{x_{i}\}_{i = 1}^{N}$  be the source and  $\mathbf{y} = \{y_j\}_{j = 1}^M$  the summary sequence. Our confidence-driven generation process defines a new order of summary tokens,  $\mathbf{o} = \{o_j\}_{j = 1}^M$ ,  $o_j\in [M]$ , according to which  $P_{\theta}(\mathbf{y}|\mathbf{x})$  is factorized into a product of conditional probabilities  $P_{\theta}(y_{o_j}|y_{\mathbf{o}_{< j}}$ ,  $\mathbf{x})$  (Eq. (1)), where  $\theta$  are model parameters to be optimized during training. Our learning objective is to minimize the negative data log-likelihood (Eq. (2)) to predict missing tokens  $y_{o_j}^*$  conditioned on the source text  $\mathbf{x}$  and the summary tokens seen thus far  $y_{\mathbf{o}_{< j}}$ .
$$
P _ {\theta} (\mathbf {y} | \mathbf {x}; \mathbf {o}) = \prod_ {j = 1} ^ {M} P _ {\theta} \left(y _ {o _ {j}} \mid y _ {\mathbf {o} <   j}, \mathbf {x}\right) \tag {1}
$$
$$
\mathcal {L} (\theta) = - \sum_ {j = 1} ^ {M} \log P _ {\theta} \left(y _ {o _ {j}} ^ {*} \mid y _ {\mathbf {o} _ {<   j}}, \mathbf {x}\right) \tag {2}
$$
Our generator is trained with a denoising objective. It consists of a decoder-only architecture with 12 Transformer blocks (Dong et al., 2019a). Given
Table 3: The target summary length L is adjusted to produce alternative summaries that have diverse content. Our generator can dynamically add or remove content, and paraphrase to produce a summary of a given length. The numbers indicate the order in which the summary tokens are generated. "BoJ" stands for "Bank of Japan". It maps to two tokens according to Byte Pair Encoding (BPE). Each summary has an ending period, so the last word also maps to two tokens.
a source text and a summary, we replace a portion of their tokens by the [MASK] token, and the model is trained to reconstruct the original data from the corrupted text. It differs from autoregressive models in that the context of each position can consist of tokens from both left and right—a source word can attend to other source words and a summary word can attend to source words and summary words seen thus far of varying positions—hence capturing a bidirectional context. The training procedure is thus analogous to that of permutation-based language modeling (Yang et al., 2019).
Our training schedule begins with masking out  $10\%$  of source tokens and linearly decreases it to  $0\%$  throughout training steps. Masking out a portion of source tokens helps the model learn contextualized representations given bidirectional context. On the target side, the schedule begins with masking out  $90\%$  of summary tokens and linearly decreases it to  $60\%$ . It allows the model to learn to predict missing summary tokens and copy source tokens to the summary. When a token is chosen, it is replaced with the [MASK] token  $80\%$  of the time, a random token of the vocabulary  $10\%$  of the time, and remains unchanged otherwise.
In Table 3, we present example summaries produced by our new confidence-driven generator for a source input. The summaries have varying lengths and levels of details. Our generator learns to add or remove content, and even paraphrase to produce a summary of a given length. We adjust the target summary length  $(L)$  to produce diverse summaries. Moreover, there exists more than one admissible summaries that capture the important information of the source text, while being grammatical and faithful to the original. It is important to note that, to decode the best summary of length  $L$ , our generator requires a position-aware beam search algorithm to explore the space of candidate summaries, which is described next.
3.1 Position-Aware Beam Search
A position-aware beam of size  $K$  not only contains the  $K$ -best candidate summaries having the highest log-likelihood at any time step, but it also records the positions of summary tokens seen thus far for each candidate summary. The tokens of candidate summaries can be decoded in any order and occur in different positions, marking an important distinction between position-aware and traditional beam search (Meister et al., 2020). The method is realized by associating each candidate summary with a binary matrix  $\mathcal{M} \in \{0,1\}_{L \times |\mathcal{V}|}$ , which records what positions have been filled by which summary tokens and what positions remain available.
Concretely, we use  $S^{\prime}$  to denote a candidate sum
- German art experts have authenticated a painting believed to be the last portrait ever made of the composer Wolfgang Amadeus Mozart, the body which runs Berlin's museums said on Thursday.
German experts identify last known portrait of Mozart
German experts identify last known portrait of Mount Mayon's
US Secretary of State Condoleezza Rice suggested Tuesday that International Atomic Energy Agency chief Mohamed ElBaradei should not interfere in diplomatic issues after he warned against the hasty use of force in the Iranian nuclear dispute.
$\nu$  Rice suggests IAEA chief should stay clear of diplomacy
X Rice suggests IAEA chief shouldn't stay clear of diplomacy
Incomplete Summary
- Total Hong Kong dollar deposits grew 2.2 percent in March, compared to 2.1 percent in February, according to the Hong Kong Monetary Authority.
HK Bank Deposits Increase in March
X Increase in March
Israel is on course to complete the main tranche of its controversial West Bank security barrier in 2004 and wrap up the project in the following year, the defence ministry said Wednesday
$\nu$  Israel surges ahead with West Bank barrier construction
X Soul-searching in Israel over shooting of West Bank barrier protestor
Swap Segments
The Security Council on Thursday voted unanimously to extend the mandate of the UN mission in Georgia for four months ahead of next week's international talks on the fallout of the recent Caucasus conflict.
$\nu$  Security Council extends mandate of UN mission in Georgia
UN mission in Georgia Security Council extends mandate of
Table 4: Corruption types. A positive instance for the selector consists of a ground-truth summary (marked by  $\checkmark$ ) and its source text. A negative instance consists of a corrupted summary ( $\times$ ) and its source text. Entity Replacement: replacing a named entity of the ground-truth summary with a random entity. Negation: negating a ground-truth summary sentence. Incomplete Summary: replacing the ground-truth summary with one of its sentence constituents to produce a corrupted summary that contains 5 words or less. Search and Replace: swapping the ground-truth summary with a similar summary in the training set that have 4 or more common bigrams. Swap Segments: splitting the ground-truth into two parts of similar length, the parts are swapped to produce an ungrammatical summary.
mary,  $score'$  is its data log-likelihood and  $\mathcal{M}'$  is a binary mask (Line 9). Our generator predicts the token probabilities  $\mathcal{P}_{L \times |\mathcal{V}|}$  for all positions, conditioned on the source text and the summary tokens seen thus far. The binary mask  $\mathcal{M}'$  indicates positions that remain available (Line 11-12). We obtain the top- $K$  tokens that have the highest probability scores across all positions, record their summary hypotheses and likelihood scores. These positions are then marked as taken (Line 14-18).
The decoding process continues until all of the  $L$  positions are filled by summary tokens. This makes our method different from traditional beam search, the latter terminates when an end-of-sequence symbol [SEP] is generated for the summary. Particularly, our method is advantageous as it exerts precise control over the summary length. The model learns to decide what content to be included in the summary given the limited space available, yielding summaries with varying levels of details.
4 The Selectors
We present two selectors to respectively assess the overall quality of the summary and predict the optimal summary length. Our selectors assume the role of a responsible agent that, when provided with a source text and multiple alternative summaries, can effectively recognize the admissible ones. It has the potential to incorporate human-in-the-loop in future to teach the model to select best summaries.
4.1 Best Overall Quality
Our goal is to build a selector to discern the difference between high and low-quality summaries. In an ideal scenario, we have human annotators to vet each source text/summary pair, the annotated data are used to train the selector. The process, however, is both expensive and time-consuming. Inspired by Krysciński et al. (2020), we automatically construct a large number of minimally different pairs, where a positive instance comprises of the source text and its ground-truth summary, and a negative instance includes the source text and a corrupted summary. We experiment with various means to generate corrupted summaries from a ground-truth summary. The corruptions should resemble common mistakes made by neural abstractive summarizers, including generating factually incorrect details, failing to convey the main points of the source text, and being ungrammatical. The corruption types experimented in this paper are illustrated in Table 4.
Distinguishing our work from that of Krysciński et al. (2020) are (i) Search and Replace, we swap the ground-truth summary with a similar summary in the training set that have  $\geq 4$  common bigrams to form a negative instance. (ii) Swap Segments splits a ground-truth summary into two parts of similar lengths, then swaps them to produce an ungrammatical summary. (iii) Incomplete Summary replaces a ground-truth summary by one of its sentence constituents, yielding a corrupted summary
that fails to convey the main ideas. These corruptions are designed to emulate system summaries that are too short to capture the main concepts, or contain hallucinated content that is not found in the source text.
We next build a binary classifier to predict if a summary is admissible given the source text. To distill information from the source text and the summary, we encode them into hidden vectors using RoBERTa (Liu et al., 2019). These are denoted by  $\mathbf{h}_{\mathbf{x}}$  and  $\mathbf{h}_{\mathbf{y}}$ , respectively. We create a vector for the pair,  $\mathbf{h} = \mathbf{h}_{\mathbf{x}} \oplus \mathbf{h}_{\mathbf{y}} \oplus |\mathbf{h}_{\mathbf{x}} - \mathbf{h}_{\mathbf{y}}| \oplus (\mathbf{h}_{\mathbf{x}} * \mathbf{h}_{\mathbf{y}})$ , consisting of a concatenation of the two hidden vectors, their absolute difference  $|\mathbf{h}_{\mathbf{x}} - \mathbf{h}_{\mathbf{y}}|$  and their element-wise product  $(\mathbf{h}_{\mathbf{x}} * \mathbf{h}_{\mathbf{y}})$ .  $\oplus$  is a concatenation of vectors. The output vector  $\mathbf{h}$  is expected to capture the gist of the source text and the summary, and a similar approach is being used for natural language inference (Chen et al., 2018). The vector  $\mathbf{h}$  is fed to a feed-forward layer to predict whether the summary is admissible given the source text. We have chosen to design the selector as a classifier rather than a ranking model because there can exist multiple, equally valid summaries for any source input. The classifier allows us to identify admissible summaries that are not only true-to-original but has the best overall quality.
4.2 Best Summary Length