SlowGuess/NAACL_2022_OCR huggingface dataset:
Wang Xu and Tiejun Zhao*  
Harbin Institute of Technology, Harbin, China  
xuwang@hit-mtlab.net, tjzhao@hit.edu.cn
Abstractive summarization can generate high quality results with the development of the neural network. However, generating factual consistency summaries is a challenging task for abstractive summarization. Recent studies extract the additional information with off-the-shelf tools from the source document as a clue to guide the summary generation, which shows effectiveness to improve the faithfulness. Unlike these work, we present a novel framework based on conditional variational autoencoders, which induces the guidance information and generates the summary equipped with the guidance synchronously. Experiments on XSUM and CNNDM dataset show that our approach can generate relevant and fluent summaries which is more faithful than the existing state-of-the-art approaches, according to multiple factual consistency metrics.
Document summarization aims to produce the shorter version of a document while preserving salient information, which helps people out of the information explosion (Mihalcea and Tarau, 2004; Daumé III and Marcu, 2006; Allahyari et al., 2017). Compared with extractive summarization that retrieves essential sentences from the source document, abstractive summarization has no constraint on the words and phrases, which has attracted more attention. With the development of neural network and the large pre-trained language models, systems can generate summarizes with a high level fluency and coherence (Devlin et al., 2019; Dong et al., 2019; Lewis et al., 2020; Zhang et al., 2020a).
Generating faithful summaries is a challenging task for abstractive summarization (Kryscinski et al., 2020; Maynez et al., 2020; Gabriel
Figure 1: Our framework trains guidance induction and summary generation jointly. It avoids the domain mismatch of the external tools and the guidance extraction is refined during training.
et al., 2021; Zhou et al., 2021). Previous studies have shown that the generated summaries distort or fabricate the facts of the source document, which also refers to the hallucination phenomenon (Huang et al., 2021). It statistics that most models produce  $80\%$  summaries with factual errors in XSUM dataset (Narayan et al., 2018) which limits the usage of summarization system (Pagnoni et al., 2021).
Recent studies provide different guidance information as input to enhance the factual consistency of the summary (Cao et al., 2018; Zhu et al., 2021). Generally, these models act a separate two-stage processing, the guidance extracting by off-the-shelf tools and summary generation conditioned on source document and guidance. Typically, Dou et al. (2021) propose an extensible guided summarization framework GSum, which has achieved impressive results. It uses an oracle to select guidance during training and extracts the keywords by out-of-box tools (Li et al., 2018) at test time. Then two transformers (Vaswani et al., 2017) are used to encode the source document and guidance.
However, the performances of separate two-stage processing models are limited by the external tools which may suffer from domain mismatch. In fact, the experiments of GSum have shown that the performance would have a significant gain when
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2340 - 2350 July 10-15, 2022 ©2022 Association for Computational Linguistics
the model uses an oracle to select guidance in testing, rather than the external tools. Moreover, the inaccuracy of the guidance extraction leads to the unfaithfulness of the summary.
In this paper, we present a novel framework which trains Guidance Induction and Summary Generation (GISG) jointly via conditional variational autoencoder. Specifically, we use phrases as the information granularity of our guidance and we induce the keyphrases of the source document, which appear in the summary semantically. First, we extract all phrases from the source document by part-of-speech tagger as candidates and we use latent variables to indicate the keyphrases. Then we learn to induce the latent variables and generate the summary jointly. Our approach avoids the domain mismatch of the external tools while the guidance extraction is refined during training. Then the faithful summaries are generated conditioned on the accurate guidance information.
Experiments on XSUM (Narayan et al., 2018) and CNNDM (Hermann et al., 2015) datasets show that our approach can generate relevant and fluent summaries which is more faithful to the source document than existing state-of-the-art approaches, according to multiple factual consistency metrics.
2.1 Abstractive Summarization
Abstractive Summarization is prone to generate factual inconsistency text with the source document (Durmus et al., 2020; Gabriel et al., 2021). Recent studies divide factual inconsistency error into two categories, intrinsic error and extrinsic error separately (Zhou et al., 2021). The intrinsic error refers to the error which is contradicted to the source document. And the extrinsic error refers to the error which is neither supported nor contradicted by the source document. Recent efforts for improving factual consistency are mainly categorized into factual guidance methods, contrastive learning methods and post-edit-based methods.
Factual guidance methods provide the models with additional information for the encoder, including the relation triples, keywords and important sentences, which guide summarization systems to pay attention to the facts and to reduce consistent error (Cao et al., 2018; Xu et al., 2021b,a; Dou et al., 2021). Zhu et al. (2021) explore using knowledge graphs to model the facts to the source
document. Dou et al. (2021) design a unified framework to introduce different information by an additional transformer encoder.
Contrastive learning methods encourage models to distinguish between positive and negative examples (Nan et al., 2021; Cao and Wang, 2021; Liu et al., 2021; Xu et al., 2022). Nan et al. (2021) generate multiple summaries candidates by sampling from the pre-trained models and selecting positive and negative examples according to the question answer based metric. Cao and Wang (2021) construct positive and negative examples by the heuristic rules, for example, replacing the entity in the references or paraphrasing the references.
Post-edit based method aims to apply a correction over the generated results to obtain more factual-consistent summarization (Dong et al., 2020; Cao et al., 2020; Chen et al., 2021a). Dong et al. (2020) leverages the question answering models to correct the factual error iteratively via span selection over the generated summaries. Cao et al. (2020) propose a corrector model to identify and correct factual errors in generated summaries. The model is trained on the synthesis data which is transformed from the reference summaries.
2.2 Conditional Variational Autoencoder
The variational auto-encoder (VAE) is a directed graphical model with certain types of latent variables, such as Gaussian latent variables (Kingma and Welling, 2014; Sohn et al., 2015; Rezende et al., 2014). A generative process of the VAE contains two stages; a set of latent variables are generated from the prior distribution and the data is generated by the generative distribution conditioned on latent variables.
Conditional VAE (CVAE) (Sohn et al., 2015; Zhao et al., 2017; Chen et al., 2021b) is a recent modification of VAE to generate diverse example conditioned on additional constrained information. Instead of providing additional information in the output, CVAE models introduce latent variables to represent the information. Inspired by CVAE, we view the keyphrases as the conditional attributes and adapt CVAE to train keyphrases induction and faithful summarization generation jointly.
Given the source input document  $X = \{X_{1},X_{2}\dots X_{N}\}$ , of length  $N$ . The task of abstractive summarization is to generate a short
Figure 2: General framework of our model. There are mainly three parts, keyphrases prediction network, induction network and condition generation network.
version of the source document, i.e.  $Y = \{Y_{1},Y_{2}\dots ,Y_{M}\}$ , where  $M$  is the length of summary. Each token  $X_{n},Y_{m}$  takes one value from a vocabulary  $\mathcal{V}$ .
Abstractive summarization is generally formulated as  $P(Y|X) = \sum_{t=1}^{M} P(Y_t \mid Y_{<t}, X)$ , which is a typical sequence to sequence generation problem. We use BART (Lewis et al., 2020) which is based on Transformer-based encoder and decoder architectures (Vaswani et al., 2017) as our backbone. Transformer layers use multi-heads self-attention to capture the dependency between the input (Vaswani et al., 2017). Concretely, the input  $X$  is converted into a vector sequence  $\mathbf{X} = \{\mathbf{x}_1, \dots, \mathbf{x}_N\}$  by the encoder, where  $\mathbf{x}_n \in \mathbb{R}^h$  and  $h$  is the size of hidden representation. In decoding step  $t$ , the decoder generates the  $t$ th word representation  $\mathbf{y}_t$  by attending to the input contextual representation  $\mathbf{X}$  and the prefix words  $\{Y_1, \dots, Y_{t-1}\}$  through the encoder-decoder attention. The probability of predicting the next token  $Y_t$  from the vocabulary  $\mathcal{V}$  is
$$
P \left(Y _ {t} \mid Y _ {<   t}, X\right) = \operatorname {s o f t m a x} \left(\boldsymbol {E} \boldsymbol {y} _ {t}\right) \tag {1}
$$
where  $\pmb{E} \in \mathbb{R}^{|\mathcal{V}| \times h}$  is the embedding matrix of the vocabulary.
4.1 Summarization with Conditional Variational Autoencoders
Previous work uses external tools to extract the guidance (e.g. keyphrases, important sentences
or relation triplets) and generate the summaries conditioned on the source document and the guidance. Our idea is to induce the guidance and generate the summary jointly. Phrases are the meaning semantic information unit of the document, which is important to express the facts of the document. Compared with a single word or a sentence of the document, a phrase contains more abundant and accurate information and is refined without lots of useless information. We will use the phrases as the information granularity of our guidance and our framework can easily be generalized to the sentence or the relation triplets.
We extract all phrases from the source document as the candidates, since the keyphrases are the subset of the phrases of the document. Then we assume a latent variable  $Z$  to indicate the keyphrases set.
Based on CVAE, we introduce an induction network  $Q(Z|X,Y)$  to approximate the true posterior distribution  $P(Z|X,Y)$ . Sohn et al. (2015) have shown that the variational lower bound can be written as:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {C V A E}} = \operatorname {K L} (Q (Z | X, Y) | | P (Z | X)) \\ \begin{array}{l} \mathcal {L} _ {\mathrm {C V A E}} = \operatorname {K L} (\mathcal {Q} (Z | X, Y) | | T (Z | X)) \\ - \mathbb {E} _ {Q} (\log P (Y | X, Z)) \geq - \log P (Y | X) \end{array} \tag {2} \\ \end{array}
$$
Thus, we jointly learn the keyphrases prediction  $P(Z|X)$  and summary generation  $P(Y|X,Z)$ . Intuitively, the term  $\mathbb{E}_Q(\log P(Y|X,Z))$  ensures the model generates the summary conditioned on  $X$  and  $Z$ , while the KL diversity term tries to guide the keyphrases prediction  $P(Z|X)$  approximate the induction  $Q(Z|X,Y)$ .
When the model is evaluated, a latent variable  $Z$  is first predicted from  $P(Z|X)$ . Then the decoder  $P(Y|X,Z)$  generates the summaries conditioned on  $X$  and  $Z$ .
We will describe our approach in detail in the following sections. The overview of our framework is in Figure 2. First, we describe candidate phrases extraction in Section 4.2. In Section 4.3, we present the prediction network and keyphrases induction network. Section 4.4 further presents the conditional summary generation network.
4.2 Candidate Phrases Extraction
We extract the phrases from the source document including the noun and verb phrases. Following Wu et al. (2021)'s work, we use the rule-based matchers to extract noun and verb phrases by the part-of-speech $^{1}$  (POS). Concretely, we use SpaCy (Montani et al., 2020) to obtain the POS tag of each word. The noun phrases are extracted by the built-in function of Spacy. And a phrase will be treated as the verb phrase if any of the cases are satisfied. 1). [AUX] VERB. The words with the verb POS tag are extracted besides the auxiliary verb. 2). VERB [RP]. A verb phrase may be followed by the particle including prepositions or adverbs (e.g., walk down). 3). AUX not VERB [RP]. "not" is considered to handle negation (e.g., would not find). And we filter out the phrases that contain less than three words. The extracted context phrases of the source document are treated as the phrases candidates.
Although, we also use the external tools POS tagger to extract the phrases of the source document, we do not directly use the phrases to guide the summary generation. Generally, the keyphrases are only a small subset of the candidates extracted by the POS tagger. We believe that our approach is robust even with an inaccurate POS tagger.
4.3 Keyphrases Prediction and Induction
We use the output of the encoder to obtain the phrase representation by averaging the representation of the corresponding words. Specially, suppose a phrase is  $X_{s:t}$ . The representation of the phrase is  $\pmb{q} = \frac{1}{t - s}\sum_{k = s}^{t}\pmb{x}_k$ . Thus, we get the representation of the candidates  $Q = \{q_1,q_2,\dots ,q_I\}$ , where  $Q\in \mathbb{R}^{I\times h}$  and  $I$  is the number of the phrases candidates.
Generally, every phrase candidate is assigned a latent variable to indicate whether the phrase is the keyphrase and the selection of each phrase is a binary classification problem. However, we find that the models tend to select redundancy phrases or even all the candidates. We argue that it is because the candidates contain similar phrases and the binary classification would lead to repetition without being constrained with the number of the phrases.
To solve the problem, we use the latent variable to select the keyphrases from the candidates. Formally, we assume the maximum of keyphrases in a document is  $B$ . We define  $Z = \{z_{1}, z_{2}, \dots, z_{B}\}$  as a latent indicator variable, where  $Z \in \mathbb{R}^{B \times I}$  and  $z_{i}$  is one-hot vector.  $z_{i}^{j} = 1$  means the phrase  $j$  is the  $i$ th keyphrase. The model can select less than  $B$  keyphrases by having the repetition latent value in  $Z$ .
Then we have the prediction network and keyphrases induction network as follows:
$$
Q \left(z _ {i} \mid X, Y\right) = \operatorname {s o f t m a x} \left(\operatorname {M L P} _ {1} ^ {i} (\boldsymbol {Q}) \boldsymbol {y} _ {\text {d o c}} ^ {T}\right) \tag {3}
$$
$$
P \left(z _ {i} | X\right) = \operatorname {s o f t m a x} \left(\mathrm {M L P} _ {2} ^ {i} (\boldsymbol {Q})\right)
$$
where  $\pmb{y}_{doc}$  is the representation of the summary.  $\pmb{y}_{doc}$  is obtained by averaging  $\{\pmb{y}_1\cdot \cdot \cdot \pmb{y}_M\}$
4.4 KeyPhrases Guide Summary Generation
We calculate the distribution of the word by attending to the source contextual representation and the keyphrases representation for the generation network  $P(Y|X,Z)$ . Similar to Aralikatte et al. (2021)'s work, we introduce a bias in Eqn. (1) to help the model focus on the keyphrases.
Formally, the generation probability of  $Y_{t}$  is formulated as:
$$
P \left(Y _ {t} \mid Y _ {<   t}, X, Z\right) = \operatorname {s o f t m a x} \left(\boldsymbol {y} _ {t} \boldsymbol {E} + \boldsymbol {f} _ {t} \boldsymbol {E}\right) \tag {4}
$$
where  $\pmb{f}_t = \mathbf{S}\mathbf{A}\mathbf{M}\mathbf{L}\mathbf{P}_3(\pmb{Q})$  and  $\pmb{S} \in \mathbb{R}^{1 \times I}$  is the selection vector.  $\pmb{S}_i = 1$  means the  $i$ th candidate is selected as the keyphrases.  $\pmb{A}$  is the attention score over the selected keyphrases and  $\pmb{A} = \mathrm{softmax}(\mathrm{MLP}_3(\pmb{Q})\pmb{y}_t^T)$ .
Basically,  $S$  is obtained from the  $Q(Z|X,Y)$  during training. As Eqn. (2) indicated, we need to calculate the expectation of  $P(Y|X,Z)$  over the distribution  $Q(Z|X,Y)$ . We use the Gumbel-Softmax trick (Jang et al., 2017) to sample from  $Q(Z|X,Y)$  and obtain low-variance gradients. Concretely, the sample probability  $\hat{Q}_i$  is
'https://spacy.io/usage/ linguistic-features
Dataset	Pairs	Tokens
Train	Valid	Test	Doc	Sum
XSUM	203028	11273	11332	430.2	23.3
CNNDM	287084	13367	11489	766.1	58.2
Table 1: Statistics of the dataset with respect to corpus size of training, validation and test set, average document (source) and summary (reference) length (in terms of tokens).
as following:
$$
\hat {Q} _ {i} = \operatorname {s o f t m a x} \left(\frac {\operatorname {M L P} _ {1} ^ {i} (\boldsymbol {Q}) \boldsymbol {y} _ {\text {d o c}} ^ {T} + \epsilon}{\tau}\right) \tag {5}
$$
where  $\epsilon$  is Gumbel noise and  $\tau$  is temperature. Thus  $S = \sum_{i=1}^{B} \hat{Q}_i$ . To avoid repetition selection among latent variables, we normalize the  $\hat{S} = \frac{S}{\max(S)}$ . In this way, the model is encouraged to extract different keyphrases, otherwise only one keyphrase is selected.
During testing, we obtain  $S$  from  $P(z_{i}|X)$ .  $S = \sum_{i=1}^{B} \text{one\_hot}(\arg \max(P(z_{i}|X)))$ . The upper value of  $S$  is clipped into 1 to avoid repetition selection.
The vanishing latent variable problem (Bowman et al., 2016; Lucas et al., 2019) exists when training with VAE. There are multiple techniques to address the problem (Zhao et al., 2017; Zhu et al., 2020). Following Zhao et al. (2017)'s work, we introduce an auxiliary loss encouraging the keyphrases to predict the words of the summary. The auxiliary loss would guide the selected phrase representation to better represent the content of the summary. Then additional loss is following:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {w}} = \frac {1}{| \mathcal {V} |} \sum_ {i = 1} ^ {| \mathcal {V} |} \left\{\left| v _ {i} \in \hat {Y} \right| \log \left(\sigma \frac {e ^ {\boldsymbol {f} _ {\text {k e y}} \boldsymbol {E} _ {i}}}{\sum e ^ {\boldsymbol {f} _ {\text {k e y}} \boldsymbol {E} _ {i}}}\right) \right. \tag {6} \\ + | v _ {i} \notin \hat {Y} | \log (1 - \sigma \frac {e ^ {\pmb {f} _ {\mathrm {k e y}} \pmb {E} _ {i}}}{\sum e ^ {\pmb {f} _ {\mathrm {k e y}} \pmb {E} _ {i}}}) \} \\ \end{array}
$$
where  $f_{\mathrm{key}} = \mathsf{SMLP}_3(\mathbf{Q})$ .  $\hat{Y}$  is the target summary and  $\sigma$  is Sigmoid function. Then our final loss function is:
$$
\mathcal {L} = \mathcal {L} _ {\mathrm {C V A E}} + \lambda \mathcal {L} _ {\mathrm {w}} \tag {7}
$$
Datasets. We evaluate our models on extreme document summarization (XSUM) (Narayan et al., 2018) and CNN/Daily Mail (CNNDM) (Hermann
et al., 2015). Both of the datasets are extracted from the news and the detailed statistics of the datasets are listed in Table 1. In XSUM dataset, the documents are summarized into single-sentence summaries. These summaries demonstrate a high level of abstraction which requires document-level inference, abstraction, and paraphrasing. CNNDM is a high quality summarization dataset consisting of news articles and human annotation summaries.
Implementation Details. We introduce our framework into BART (Lewis et al., 2020) which is a strong abstractive summarization model pretrained with a denoising autoencoding objective. We use the  $\mathrm{FairSeq}^2$  as the implementation of our baseline and model. We inherit their provided hyper-parameters of XSUM and CNNDM. Concretely, the total number of the updates is  $1.5\mathrm{w}$  in XSUM and  $2\mathrm{w}$  in CNNDM. The maximum number of tokens in a batch is 4096 with gradient accumulation steps of 4. We use Adam optimizer and the learning rate is set to  $3\mathrm{e - }5$ . The  $\epsilon$  is  $1\mathrm{e - }8$  and  $\beta$  is (0.9, 0.999). The maximum of the keyphrases  $B$  is set to 8. And the temperature  $\tau$  is set to 0.1 for Gumbel-Softmax during training. We use mixed-precision to speed up model training and the warm-up is set to 500 steps. All the experiments are done on 2 and 4 NVIDIA 3090 in XSUM and CNNDM. For the beam search, the minimum summary length is 11 and 56 for XSUM and CNNDM, respectively. The number of beams is 4 for XSUM and 6 for CNNDM. And the ROUGE-L score on the validation set is used to pick the best model.
Evaluation Metrics. ROUGE $^3$  (Lin and Hovy, 2003) considers lexical overlap against the reference summaries, which is widely used to evaluate the informativeness and fluency of the summary. We report on ROUGE-1, ROUGE-2 and ROUGE-L to measure summary qualities.
We also use BERTScore $^4$  (Zhang et al., 2020b) to evaluate the semantic similarity between a hypothesis and the reference summary by contextual representation.
However ROUGE and BERTScore perform poorly in capturing factual consistency with the source document. Recent studies have developed different categories to evaluate the faithfulness of a generated summary given its
2https://github.com/pytorch/fairseq
<sup>3</sup>https://github.com/pltrdy/files2rouge
<sup>4</sup>https://github.com/Tiiger/bert_score
Models	Lexical Overlap	Semantic BERTScore	Relation Triplet	QA-based
R1	R2	RL	Close	Open	QAGS	QuesEval
XSUM
EXTORA* (Narayan et al., 2018)	29.82	8.83	22.68	85.74	18.57	72.46	69.20	45.84
FASum* (Zhu et al., 2021)	30.28	10.03	23.76	88.03	1.63	0.36	11.13	31.18
GSum* (Dou et al., 2021)	44.93	21.19	35.96	90.41	1.92	1.75	13.58	36.90
BART* (Lewis et al., 2020)	45.49	21.82	36.69	90.83	1.89	2.02	13.76	36.91
BART†	44.65	21.28	36.09	90.62	2.03	2.11	13.76	37.07
GISG (ours)	45.54	21.99	36.82	92.11	2.14	2.18	14.31	37.09
CNNDM
MATCH* (Zhong et al., 2020)	44.47	20.92	40.05	87.32	50.92	89.21	77.70	59.93
FASum* (Zhu et al., 2021)	40.53	17.84	37.40	87.86	38.56	67.82	71.33	57.58
GSum* (Dou et al., 2021)	45.89	22.27	42.68	88.64	41.23	70.69	71.11	57.54
BART* (Lewis et al., 2020)	44.25	21.11	41.16	88.33	41.40	70.79	71.89	57.65
BART†	44.11	21.16	40.55	88.13	39.16	69.69	71.25	57.80
GISG (ours)	44.50	21.45	41.05	88.56	42.07	70.83	72.34	58.52
Table 2: Main results. MATCHSum (denoted by MATCH) and EXT-ORACLE (denoted by EXTORA) are extractive summarization models. The results with * are computed based on the output files in the EXPLAINABoard. The results with † are our reimplement of the baseline models. Bold indicates the best performance in the abstractive summarization models groups.
source document (Zhang et al., 2020b; Dong et al., 2020; Liu et al., 2021). Our approach is evaluated with four factual consistency metrics, including relation triplets based and question answering (QA) based metrics.
Relation triplets based metrics evaluate factual consistency by comparing structured data of factual information extracted from the summary and the source document. Close Scheme Fact Triple (Dong et al., 2020) extract (Subject, Relation, Object) by named entity recognition and relation extraction models and then calculate the precision between the triples extracted from the summary and source document. Open Scheme Fact Triple (Dong et al., 2020) extract the text spans to indicate the relation and calculate the precision similar to close scheme fact triple.
Question answering based metrics use the pretrained QA model to evaluate the faithfulness. QAGS (Wang et al., 2020) extracts text spans from the predicted summary and generates questions conditioned on the predicted summary by a trained question generation model. A pretrained QA model answers the questions from the document. Then the matching score is calculated between the answer from the document and the summary. QuestEval (Rebuffel et al., 2021) not only generate (question, answer) pairs from the summary, but also from the source document, which considers to measure the recall performance.
We use factsumm<sup>5</sup> and OpenIE<sup>6</sup> to calculate close scheme fact triple and open scheme fact triple. And we use the repository to calculate QAGS<sup>5</sup> and QuesEval<sup>7</sup>. We only calculate factual consistency metrics of 1k (document, reference, summary) for the computing efficiency.