SlowGuess/NAACL_2022_OCR huggingface dataset:
Xin Sheng $^{1}$ , Linli Xu $^{2,3*}$ , Yinlong Xu $^{1}$ , Deqiang Jiang $^{4}$ , Bo Ren $^{4}$
<sup>1</sup> School of Computer Science and Technology, University of Science and Technology of China.
<sup>2</sup> Anhui Province Key Laboratory of Big Data Analysis and Application,
School of Computer Science and Technology, University of Science and Technology of China.
3 IFLYTEK Co., Ltd. 4 Tencent Youtu Lab.
xins@mail.ustc.edu.cn,{linlixu,ylxxu}  $@$  ustc.edu.cn,
{dqiangjiang,timren}@tencent.com
We propose a novel siamese generative adversarial net for abstractive text summarization (SSPGAN), which can preserve the main semantics of the source text. Different from previous generative adversarial net based methods, SSPGAN is equipped with a siamese semantic-preserving discriminator, which can not only be trained to discriminate the machine-generated summaries from the human-summarized ones, but also ensure the semantic consistency between the source text and target summary. As a consequence of the min-max game between the generator and the siamese semantic-preserving discriminator, the generator can generate a summary that conveys the key content of the source text more accurately. Extensive experiments on several text summarization benchmarks in different languages demonstrate the effectiveness of the proposed method.
Abstractive text summarization endeavors to produce a concise and fluent summary for a given text, while maintaining the key content and overall meaning. Previous attempts tackle this problem with either rule-based or statistical-based methods. Recently, with the successes obtained on the machine translation task (Sutskever et al., 2014; Sheng et al., 2020), the neural network based sequence-to-sequence framework is also applied to the abstractive text summarization task. Specifically, the sequence-to-sequence architecture consists of an encoder responsible for transforming the source sequence  $\mathbf{x} = \{x_{1}, x_{2}, \ldots, x_{Tx}\}$  into an intermediate representation, and a decoder to generate a target sequence  $\mathbf{y} = \{y_{1}, y_{2}, \ldots, y_{Ty}\}$  using the previously generated intermediate representation. Furthermore, to dynamically generate a context vector for a target word being generated, the attention mechanism (Bahdanau et al., 2014; Luong
Source:成都市软件和信息技术服务业近些年来一直保持快速增长势头，稳居中西部城市之首，已成为我国西部“硅谷”。《2013年度成都市软件和信息技术服务产业发展报告》日前发布……详情请见：@成都日报@成都发布
The software and information technology service industry in Chengdu has maintained the momentum of rapid development in recent years, ranking first among the cities in the central and western regions, and has become the "Silicon Valley" in the west of our country.
"The 2013 Chengdu Software and Information
Technology Service Industry Development Report" was released a few days ago... For details, please see:
@Chengdu Daily@Chengdu post
Reference: 成都倾力打造西部“硅谷”
Chengdu strives to build the Western "Silicon Valley"
Generated: 成都发布软件和信息技术服务产业发展报告
Chengdu releases software and information technology service industry development report
Figure 1: The case of lacking saliency in abstractive text summarization. Bold text represents the key content, while the underlined parts represent the unimportant content.
et al., 2015) is proposed to strengthen the sequence-to-sequence models, which enables the model to focus on the relevant parts of the source-side sequence. Based on the encoder-decoder framework, many variants of model structures, such as convolutional neural network (CNN) and recurrent neural network (RNN) are proposed (Bahdanau et al., 2014; Gehring et al., 2017). With the emergence of Transformer (Vaswani et al., 2017), which is based entirely on the attention mechanism, state-of-the-art performance is achieved on many sequence-to-sequence tasks. Nevertheless, for the task of abstractive text summarization, one of the dominant challenges is to maintain saliency, which requires the generated summary to convey the important information accurately. As shown in Figure 1, the key content of the source text "Chengdu become the 'Silicon Valley' in the west of our country" is
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2121 - 2132
accurately summarized in the reference, while the generated summary expresses the unimportant content "Chengdu releases software and information technology service industry development report".
Intuitively, the lack of saliency in summarization is usually caused by attending to wrong parts of the source text, inspiring many attention optimization methods for more accurate attention mechanism. Among them, (Lin et al., 2018) proposes a global encoding framework, which controls the attention information flow from the encoder to the decoder based on the global information of the source context. (Gui et al., 2019) proposes an effective method to regularize the attention weights from both global and local aspects. (Duan et al., 2019) introduces a novel attention mechanism, where the attention weights on relevant parts of the source side are encouraged while the attention weights on less relevant or irrelevant parts are discouraged with a softmax and a softmax function respectively. However, for these methods, the underlying nature of saliency, which is actually the sentence-level semantic consistency between the source text and the generated summary, is generally overlooked.
To explicitly maintain the semantic consistency, we propose a novel Siamese Semantic-Preserving Generative Adversarial Net (SSPGAN) for abstractive text summarization. In SSPGAN, different from conventional adversarial training (Goodfellow et al., 2014) which mainly focuses on how to generate more realistic data, a novel training paradigm is introduced to generate a summary that is more semantically consistent with the source text. Specifically, the proposed model consists of two adversarial modules which play a min-max game:
- A conventional neural encoder-decoder based generator, which aims to generate the summary sequence based on the input text.
- A siamese semantic-preserving discriminator. Different from the conventional discriminator in a generative adversarial net (GAN), in addition to distinguishing the real summary from the generated summary, it is also required to capture the semantic consistency between the source text and the target summary. And we adopt a pseudo siamese net to achieve that. Specifically, we aim to maximize the semantic similarity for a real sentence pair (text, real summary), while minimizing it for a generated sentence pair (text, generated summary).
During the training process, in terms of the authenticity and semantic consistency with the input source text, the generator aims to fool the discriminator into believing that its output is a human-generated summary, and the discriminator makes efforts not to be fooled by improving its ability to distinguish the machine-generated summary from the human-generated one. This kind of adversarial training achieves a win-win situation when the generator and the discriminator reach a Nash Equilibrium (Zhao et al., 2016; Arora et al., 2017; Guimaraes et al., 2017).
Different from conventional GANs, which assume the existence of a generator in a continuous space, in our proposed framework, the text summarization model is in fact not a typical generative model, but instead a probabilistic transformation that maps a source text to a target summary, both in a discrete space. To this end, we turn to a policy gradient method named REINFORCE (Williams, 1992), which can guarantee that both the two sub models are effectively optimized in an adversarial manner. In addition to the conventional reward, which is the estimated probability of the generated summary being discriminated as the real one, we also adopt the semantic similarity between the source text and the generated summary as a supplementary reward signal. Besides, we employ Transformer (Vaswani et al., 2017) as the basis of our discriminator to capture both the global and local features of the sentence.
- We propose a siamese net based discriminator to ensure the semantic consistency between the generated summary and the source text.
- A generative adversarial net based entirely on Transformer is proposed. As far as we know, this work is the first attempt to apply such framework into the text summarization task.
- Experimental results on both English and Chinese text summarization datasets show that the proposed model outperforms conventional GAN-based methods. And we also demonstrate that the proposed method can maintain semantic consistency from multiple perspectives.
Automatic text summarization can be broadly divided into extractive and abstractive summarization.
The extractive methods simply extract important parts of the source text and reorganize them in a certain order (Jing and McKeown, 2000; Knight and Marcu, 2000; Neto et al., 2002). In comparison, abstractive text summarization is closer in principle to the process of manual summarization, which extracts the essential information of the source text and describes it in a shorter version as the abstractive summary. In this paper, we focus on abstractive text summarization.
Previous works on abstractive text summarization are mainly designed with statistical methods and rule-based methods (Banko et al., 2000; Dorr et al., 2003; Zajic et al., 2004; Cohn and Lapata, 2008). Recently, the sequence-to-sequence neural framework becomes predominant on the task of abstractive text summarization (Chopra et al., 2016; Nallapati et al., 2016; Li et al., 2017b). Later on, with the advent of Transformer (Vaswani et al., 2017), more and more works choose it as the base model in their frameworks.
For the abstractive text summarization task, out-of-vocabulary (OOV), repetitions and lack of saliency are three dominant challenges. To tackle the problem of OOV, some works introduce the pointer network and copy mechanism (Nallapati et al., 2016; See et al., 2017; Gu et al., 2016; Paulus et al., 2017). On the issue of repetitions, (See et al., 2017) adopts a coverage mechanism, which is inspired by the coverage vector from neural machine translation (Tu et al., 2016). Regarding saliency, some works (Duan et al., 2019; Gui et al., 2019) focus on how to optimize the attention mechanism, while (Zhu et al., 2021) tries to enhance the factual consistency with a fact corrector. Meanwhile, (Narayan et al., 2021) adopts the content planning to improve the performance of abstractive summarization model. However, the essence of saliency, which is the sentence-level semantic consistency between the source text and the generated summary, is intuitive yet usually overlooked.
The proposed training principle is based on adversarial learning (Goodfellow et al., 2014). In conventional adversarial training, a generator and a discriminator compete with each other, forcing the generator to produce high quality samples that can fool the discriminator. Adversarial training typically excels in image generation (Goodfellow et al., 2014), with less applications in natural language processing tasks (Yu et al., 2017; Li et al., 2017a), mainly due to the difficulty of propagating
the signals from the discriminator to the generator through the discretely generated tokens. (Yu et al., 2017) addresses this issue with a reinforcement learning approach for sequence generation. Thus, the adversarial training paradigm can improve the model on the sentence-level instead of the vanilla token-level (e.g., maximum likelihood estimation).
To address the semantic inconsistency problem mentioned above, we introduce the paradigm of siamese net into GANs. Siamese net is a class of neural network architectures that contain more than one identical or different sub networks, which depends on whether the inputs are similar or not. Siamese net is generally used to measure the similarity between the inputs by comparing their corresponding output feature vectors, and can be broadly divided into two types: true siamese net and pseudo siamese net. The true siamese net contains identical sub networks which share the same architecture and network parameters, while the pseudo siamese net contains sub networks which have different parameters and even different architectures. Among the existing works, (Kenter et al., 2016) is the first to adopt siamese net into unsupervised sentence embedding learning. (Mueller and Thyagarajan, 2016) proposes MaLSTM to learn sentence similarities with Manhattan distance. (Neculoiu et al., 2016) considers similarity matching of a sentence pair as a binary classification task and replaces the Manhattan distance with cosine similarity. Recently, (Reimers and Gurevych, 2019) introduces the principle of siamese net to fine-tune BERT (Devlin et al., 2019) for better sentence embedding.
Different from previous GAN-based abstractive text summarization model in the work of (Liu et al., 2018), by incorporating siamese net into GANs, the generator can generate summaries which are more semantically consistent with the source texts. As far as we know, this work is the first attempt to apply siamese net to the GAN-based sequence-to-sequence generation task.
3 Siamese Semantic-Preserving Generative Adversarial Net
In this section, we introduce the architecture of the proposed Siamese Semantic-Preserving Generative Adversarial Net (SSPGAN) in detail. The model consists of two main components. The first component is a standard Transformer-based summary generator  $G$  (Figure 2). During adversarial training, the generator  $G$  is treated as an agent tak
Figure 2: The summary generator, taking a conventional Transformer based encoder-decoder architecture (Vaswani et al., 2017), where the predicted word from the previous step serves as the input of the current step during inference. We omit some layers for brevity.
ing sequential actions (i.e., generating words) and trained using policy gradient given the reward of each generated word. The second component is a siamese network based discriminator  $D$ , which is also implemented based on the Transformer. On the one hand, the discriminator  $D$  is required to distinguish the generated summary from the real one. On the other hand, it aims to capture the semantic similarity between the source text and the target summary. Specifically, it is expected to maximize the semantic similarity for the real sentence pair (text, real summary), while minimizing the semantic similarity of the generated pair (text, generated summary). From these two perspectives, we compute a composite reward for each generated summary. Both the generator  $G$  and the discriminator  $D$  are iteratively trained. Figure 3 shows the overview of the adversarial training framework. In the following, we describe the generator  $G$  and the siamese semantic-preserving discriminator  $D$  in detail.
3.1 Generator
At time step  $t$ , the generator  $G$  takes an action (i.e., a word  $y_{t}$ ) according to a stochastic policy  $\pi_{\theta}(y_t|\mathbf{x},\mathbf{y}_{t - 1})$ , where  $\mathbf{x}$  is the input source text,  $\mathbf{y}_{t - 1} = [y_1,\dots ,y_{t - 1}]$  is the previously generated partial summary, and  $\theta$  is the parameter of the policy. We utilize the conventional Transformer based encoder-decoder framework (Vaswani et al., 2017) as the model of the policy. By sequentially gener
ating each word  $y_{t}$  using the policy  $\pi_{\theta}(.)$  until the end, a complete sentence  $\mathbf{y}$  is generated. In conventional sequence-to-sequence learning, the model is trained to minimize the cross-entropy loss:
$$
\mathcal {J} (\theta) = - \sum_ {n = 1} ^ {N} \sum_ {t = 1} ^ {T _ {n}} \log \pi_ {\theta} \left(\hat {y} _ {t} ^ {n} \mid \mathbf {x} ^ {n}, \hat {\mathbf {y}} _ {t - 1} ^ {n}\right) \tag {1}
$$
where  $N$  is the number of text-summary pairs,  $T_{n}$  is the length of the ground-truth summary  $\hat{\mathbf{y}}^n$ , Loss is the cross-entropy loss,  $\hat{\mathbf{y}}_{t - 1}^n$  and  $\hat{y}_t^n$  are the ground-truth partial summary and word, respectively. Nevertheless, in adversarial training, these are no explicit supervised information for computing the cross entropy loss. Hence, we adopt our discriminator  $D$  to assess the quality of the generated complete summary  $\mathbf{y}^n$ . Specifically, the discriminator  $D$  is responsible for calculating a reward using the generated summary  $\mathbf{y}^n$  and the source text  $\mathbf{x}^n$  (See Section 3.3 for details).
3.2 Siamese Semantic-Preserving Discriminator
Our discriminator  $D$  aims to not only distinguish the real summary from the generated one, but also capture the semantic similarity between the source text and the target summary. Here, the discriminator  $D$  is implemented based on the Transformer, as Transformer is capable of capturing both local and global sentence features. In the meantime, to capture the semantic similarity, the whole framework of the discriminator  $D$  is designed based on the siamese net (right panel in Figure 3).
Given the source text  $\mathbf{x} = \{x_{1}, x_{2}, \ldots, x_{Tx}\}$  and the target summary  $\mathbf{y} = \{y_{1}, y_{2}, \ldots, y_{Ty}\}$  (here  $\mathbf{y}$  represents both the real and generated summary for simplicity), where  $x_{t}$  and  $y_{t}$  are the  $t$ -th words in the corresponding sequences. For the source text sequence  $\mathbf{x}$ , we take it as input of the Transformer encoder. After the processing of Transformer blocks, a hidden state sequence  $h_{\mathbf{x}}$  will be produced:
$$
h _ {\mathbf {x}} = \left\{h _ {x _ {1}}, h _ {x _ {2}}, \dots , h _ {x _ {T _ {x}}} \right\} \tag {2}
$$
where  $h_{x_t}$  is the hidden state corresponding to  $x_t$  in the input sequence. Thus,  $h_{x_t}$  contains not only the positional information, but also the global and local correlation information. To get the final feature representation  $f_{\mathbf{x}}$  for the input sequence, a mean-pooling operation is leveraged over the output hidden state sequence  $h_{\mathbf{x}}$ . Since there exists difference between the textual structures of the source text
Figure 3: Overview of the model. Left panel: our generator  $G$  produces a summary conditioned on the source text. At each time step, the expected reward of a newly generated word ("earthquake" in the presented example) is computed from the siamese semantic-preserving discriminator  $D$  using Monte Carlo rollout. We use policy gradient to update the generator  $G$  toward generating summaries with higher rewards. Right panel: the discriminator  $D$  observes the generated summary and aims at distinguishing it from the real one. Besides, the discriminator  $D$  is responsible for capturing the semantic consistency between the source text and the target summary. During adversarial training, both the generator  $G$  (left) and the discriminator  $D$  (right) are iteratively updated to improve.
and the target summary, for the target summary, we adopt the same encoder framework as the one for the source text, but share no parameters (i.e., pseudo siamese net). And the corresponding final feature representation  $f_{\mathrm{y}}$  is also obtained using the mean-pooling operation. Finally, given both the source text and the target summary, the probability that the target summary is classified as real can be calculated as:
$$
p = \sigma \left(V \left[ f _ {\mathbf {x}}, f _ {\mathbf {y}} \right]\right) \tag {3}
$$
where  $V$  is the weight matrix to transform the concatenation of  $f_{\mathbf{x}}$  and  $f_{\mathbf{y}}$  into a 2-dimensional embedding and  $\sigma$  is the logistic function. Finally, the training objective for discriminating the real summary from the generated one can be formulated as a supervised classification objective:
$$
\mathcal {L} _ {\text {r e a l}} (\phi) = - \sum_ {n = 1} ^ {N} \log p \left(l ^ {n} \mid \mathbf {x} ^ {n}, \mathbf {y} ^ {n}; \phi\right) \tag {4}
$$
where  $N$  is the number of text-summary pairs,  $\phi$  is the model parameters of the discriminator  $D$ , and  $l^n$  is the corresponding label (i.e., 0 for the generated summary and 1 for the real summary).
To capture the semantic similarity between the source text and the target summary, we further utilize the final features of the pseudo siamese net.
Specifically, we aim to maximize the semantic similarity between the source text and the real summary, while minimizing it for the pair of the source text and the generated summary. To this end, we adopt the cosine function to evaluate the similarity of the sentence pair:
$$
\mathcal {S} _ {\text {c o s}} = \frac {\langle f _ {\mathbf {x}} , f _ {\mathbf {y}} \rangle}{| | f _ {\mathbf {y}} | | | | f _ {\mathbf {y}} | |} \tag {5}
$$
and the value of  $S_{cos}$  ranges from  $-1$  to 1. Next, we can obtain the contrastive loss  $\mathcal{L}_{sim}$  for siamese semantic similarity learning:
$$
\begin{array}{l} \mathcal {L} _ {s i m} = \frac {1}{N} \sum_ {n = 1} ^ {N} l ^ {n} \mathcal {L} _ {+} \left(f _ {\mathbf {x}} ^ {n}, f _ {\mathbf {y}} ^ {n}\right) \tag {6} \\ + (1 - l ^ {n}) \mathcal {L} _ {-} \left(f _ {\mathbf {x}} ^ {n}, f _ {\mathbf {y}} ^ {n}\right) \\ \end{array}
$$
where  $N$  is the number of text-summary pairs,  $l^n$  is the corresponding summary label (i.e., 1 for the real sentence pair and 0 for the generated sentence pair),  $\mathcal{L}_{+}$  and  $\mathcal{L}_{-}$  are the corresponding loss functions for the real and generated sentence pair, respectively. The two sub loss functions are given by:
$$
\begin{array}{l} \mathcal {L} _ {+} \left(f _ {\mathbf {x}} ^ {n}, f _ {\mathbf {y}} ^ {n}\right) = \left(1 - \mathcal {S} _ {\cos}\right) ^ {2} \\ \mathcal {L} _ {-} \left(f _ {\mathbf {x}} ^ {n}, f _ {\mathbf {y}} ^ {n}\right) = \left\{ \begin{array}{l l} \mathcal {S} _ {c o s} ^ {2} & \text {i f} \mathcal {S} _ {c o s} > 0 \\ 0 & \text {o t h e r w i s e} \end{array} \right. \tag {7} \\ \end{array}
$$
Thus, we can obtain the final objective of the siamese semantic-preserving discriminator  $D$ :
$$
\mathcal {L} _ {d} = \eta \mathcal {L} _ {\text {r e a l}} + (1 - \eta) \mathcal {L} _ {\text {s i m}} \tag {8}
$$
where  $\eta$  is a hyper-parameter to balance the two sub training objectives.
3.3 Policy Gradient Training
Following (Yu et al., 2017), during adversarial training, the goal of the generator  $G$  is defined as to generate a summary sequence from the start state to maximize its expected overall reward. Formally, the objective function is calculated as:
$$
\mathcal {J} _ {a d v} (\theta) = \sum_ {\mathbf {y} _ {1: T}} G _ {\theta} \left(\mathbf {y} _ {1: T} | \mathbf {x}\right) \cdot R _ {D _ {\phi}} ^ {G _ {\theta}} \left(\mathbf {y} _ {1: T - 1}, \mathbf {x}, y _ {T}\right) \tag {9}
$$
where  $\theta$  denotes the parameters of  $G$ ,  $\mathbf{y}_{1:T} = \{y_1,\dots ,y_T\}$  indicates the generated target summary,  $\mathbf{x}$  is the source text. Here we denote  $T_{y}$  as  $T$  for simplicity.  $R_{D_{\phi}}^{G_{\theta}}$  is the action-value function of the generated summary given the source text  $\mathbf{x}$  (i.e., the expected accumulative reward starting from the state  $(\mathbf{y}_{1:T - 1},\mathbf{x})$ , taking action  $y_{T}$ , and adopting the policy  $G_{\theta}$ ). To estimate the action-value function, we combine the probability of being classified as real by the discriminator  $D$  with the cosine similarity as the total reward:
$$
\begin{array}{l} R _ {D _ {\phi}} ^ {G _ {\theta}} (\mathbf {y} _ {1: T - 1}, \mathbf {x}, y _ {T}) = \lambda \cdot s _ {r e a l i t y} \\ + (1 - \lambda) \cdot s _ {\text {s i m i l a r i t y}} \\ s _ {r e a l i t y} = D _ {\phi} (\mathbf {x}, \mathbf {y} _ {1: T}) - b (\mathbf {x}, \mathbf {y} _ {1: T}) \\ \end{array}
$$
$$
s _ {\text {s i m i l a r i t y}} = \mathcal {S} _ {\text {c o s}} (\mathbf {x}, \mathbf {y} _ {1: T}) \tag {10}
$$