SlowGuess/ACL_2020_OCR huggingface dataset:
Wang Chen<sup>1</sup>, Hou Pong Chan<sup>1</sup>, Piji Li<sup>2</sup>, Irwin King<sup>1</sup>
<sup>1</sup>The Chinese University of Hong Kong, Shatin, N.T., Hong Kong
1{wchen, hpchan, king}@cse.cuhk.edu.hk
$^{2}$ pijili@tencent.com
Keyphrase generation (KG) aims to summarize the main ideas of a document into a set of keyphrases. A new setting is recently introduced into this problem, in which, given a document, the model needs to predict a set of keyphrases and simultaneously determine the appropriate number of keyphrases to produce. Previous work in this setting employs a sequential decoding process to generate keyphrases. However, such a decoding method ignores the intrinsic hierarchical compositionality existing in the keyphrase set of a document. Moreover, previous work tends to generate duplicated keyphrases, which wastes time and computing resources. To overcome these limitations, we propose an exclusive hierarchical decoding framework that includes a hierarchical decoding process and either a soft or a hard exclusion mechanism. The hierarchical decoding process is to explicitly model the hierarchical compositionality of a keyphrase set. Both the soft and the hard exclusion mechanisms keep track of previously-predicted keyphrases within a window size to enhance the diversity of the generated keyphrases. Extensive experiments on multiple KG benchmark datasets demonstrate the effectiveness of our method to generate less duplicated and more accurate keyphrases<sup>1</sup>.
Keyphrases are short phrases that indicate the core information of a document. As shown in Figure 1, the keyphrase generation (KG) problem focuses on automatically producing a keyphrase set (a set of keyphrases) for the given document. Because of the condensed expression, keyphrases can benefit various downstream applications including opinion mining (Berend, 2011; Wilson et al., 2005), doc
Input Document: ... A noninvasive diagnostic device was developed to assess the vascular origin and severity of penile dysfunction. It was designed and studied using both a mathematical model of penile hemodynamics and preliminary experiments on healthy young volunteers. ... Simulations using a mathematical model show that the device is capable of differentiating between arterial insufficiency and venous leak and indicate the severity of each. ...
Keyphrases:  
{erectile dysfunction; arterial insufficiency; venous leak; veno-occlusive mechanism; mathematical model; hemodynamics}
Figure 1: An example of an input document and its expected keyphrase output for keyphrase generation problem. Present keyphrases that appear in the document are underlined.
ument clustering (Hulth and Megyesi, 2006), and text summarization (Wang and Cardie, 2013).
Keyphrases of a document can be categorized into two groups: present keyphrase that appears in the document and absent keyphrase that does not appear in the document. Recent generative methods for KG apply the attentional encoder-decoder framework (Luong et al., 2015; Bahdanau et al., 2014) with copy mechanism (Gu et al., 2016; See et al., 2017) to predict both present and absent keyphrases. To generate multiple keyphrases for an input document, these methods first use beam search to generate a huge number of keyphrases (e.g., 200) and then pick the top  $N$  ranked keyphrases as the final prediction. Thus, in other words, these methods can only predict a fixed number of keyphrases for all documents.
However, in a practical situation, the appropriate number of keyphrases varies according to the content of the input document. To simultaneously predict keyphrases and determine the suitable number of keyphrases, Yuan et al. (2018) adopts a sequential decoding method with greedy search to generate one sequence consisting of the predicted keyphrases and separators. For example, the produced sequence may be "hemodynamics [sep] erectile dysfunction [sep] ..., where "[sep]" is the sep
1Our code is available at https://github.com/ Chen-Wang-CUHK/ExHiRD-DKG.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1095-1105 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
arator. After producing an ending token, the decoding process terminates. The final keyphrase predictions are obtained after splitting the sequence by separators. However, there are two drawbacks to this method. First, the sequential decoding method ignores the hierarchical compositionality existing in a keyphrase set (a keyphrase set is composed of multiple keyphrases and each keyphrase consists of multiple words). In this work, we examine the hypothesis that a generative model can predict more accurate keyphrases by incorporating the knowledge of the hierarchical compositionality in the decoder architecture. Second, the sequential decoding method tends to generate duplicated keyphrases. It is simple to design specific post-processing rules to remove the repeated keyphrases, but generating and then removing repeated keyphrases wastes time and computing resources. To address these two limitations, we propose a novel exclusive hierarchical decoding framework for KG, which includes a hierarchical decoding process and an exclusion mechanism.
Our hierarchical decoding process is designed to explicitly model the hierarchical compositionality of a keyphrase set. It is composed of phrase-level decoding (PD) and word-level decoding (WD). A PD step determines which aspect of the document to summarize based on both the document content and the aspects summarized by previously-generated keyphrases. The hidden representation of the captured aspect is employed to initialize the WD process. Then, a new WD process is conducted under the PD step to generate a new keyphrase word by word. Both PD and WD repeat until meeting the stop conditions. In our method, both PD and WD attend the document content to gather contextual information. Moreover, the attention score of each WD step is rescaled by the corresponding PD attention score. The purpose of the attention rescaling is to indicate which aspect is focused on by the current PD step.
We also propose two kinds of exclusion mechanisms (i.e., a soft one and a hard one) to avoid generating duplicated keyphrases. Either the soft one or the hard one is used in our hierarchical decoding process. Both of them are used in the WD process of our hierarchical decoding. Besides, both of them collect the previously-generated  $K$  keyphrases, where  $K$  is a predefined window size. The soft exclusion mechanism is incorporated in the training stage, where an exclusive loss is em
ployed to encourage the model to generate a different first word of the current keyphrase with the first words of the collected  $K$  keyphrases. However, the hard exclusion mechanism is used in the inference stage, where an exclusive search is used to force WD to produce a different first word with the first words of the collected  $K$  keyphrases. Our motivation is from the statistical observation that in  $85\%$  of the documents on the largest KG benchmark, the keyphrases of each individual document have different first words. Moreover, since a keyphrase is usually composed of only two or three words, the predicted first word significantly affects the prediction of the following keyphrase words. Thus, our exclusion mechanisms can boost the diversity of the generated keyphrases. In addition, generating fewer duplications will also improve the chance to produce correct keyphrases that have not been predicted yet.
We conduct extensive experiments on four popular real-world benchmarks. Empirical results demonstrate the effectiveness of our hierarchical decoding process. Besides, both the soft and the hard exclusion mechanisms significantly reduce the number of duplicated keyphrases. Furthermore, after employing the hard exclusion mechanism, our model consistently outperforms all the SOTA sequential decoding baselines on the four benchmarks.
We summarize our main contributions as follows: (1) to our best knowledge, we are the first to design a hierarchical decoding process for the keyphrase generation problem; (2) we propose two novel exclusion mechanisms to avoid generating duplicated keyphrases as well as improve the generation accuracy; and (3) our method consistently outperforms all the SOTA sequential decoding methods on multiple benchmarks under the new setting.
2.1 Keyphrase Extraction
Most of the traditional extractive methods (Witten et al., 1999; Mihalcea and Tarau, 2004) focus on extracting present keyphrases from the input document and follow a two-step framework. They first extract plenty of keyphrase candidates by handcrafted rules (Medelyan et al., 2009). Then, they score and rank these candidates based on either unsupervised methods (Mihalcea and Tarau, 2004) or supervised learning methods (Nguyen and Kan, 2007; Hulth, 2003). Recently, neural-based se
quence labeling methods (Gollapalli et al., 2017; Luan et al., 2017; Zhang et al., 2016) are also explored in keyphrase extraction problem. However, these extractive methods cannot predict absent keyphrase which is also an essential part of a keyphrase set.
2.2 Keyphrase Generation
To produce both present and absent keyphrases, Meng et al. (2017) introduced a generative model, CopyRNN, which is based on an attentional encoder-decoder framework (Bahdanau et al., 2014) incorporating with a copy mechanism (Gu et al., 2016). A wide range of extensions of CopyRNN are recently proposed (Chen et al., 2018, 2019b; Ye and Wang, 2018; Chen et al., 2019a; Zhao and Zhang, 2019). All of them rely on beam search to over-generate lots of keyphrases with large beam size and then select the top  $N$  (e.g., five or ten) ranked ones as the final prediction. That means these over-generated methods will always predict  $N$  keyphrases for any input documents. Nevertheless, in a real situation, the keyphrase number should be determined by the document content and may vary among different documents.
To this end, Yuan et al. (2018) introduced a new setting that the KG model should predict multiple keyphrases and simultaneously decide the suitable keyphrase number for the given document. Two models with a sequential decoding process, catSeq and catSeqD, are proposed in Yuan et al. (2018). The catSeq is also an attentional encoder-decoder model (Bahdanau et al., 2014) with copy mechanism (See et al., 2017), but adopting new training and inference setup to fit the new setting. The catSeqD is an extension of catSeq with orthogonal regularization (Bousmalis et al., 2016) and target encoding. Lately, Chan et al. (2019) proposed a reinforcement learning based fine-tuning method, which fine-tunes the pre-trained models with adaptive rewards for generating more sufficient and accurate keyphrases. We follow the same setting with Yuan et al. (2018) and propose an exclusive hierarchical decoding method for the KG problem. To the best of our knowledge, this is the first time the hierarchical decoding is explored in the KG problem. Different from the hierarchical decoding in other areas (Fan et al., 2018; Yarats and Lewis, 2018; Tan et al., 2017; Chen and Zhuge, 2018), we rescale the attention score of each WD step with the corresponding PD attention score to provide aspect
guidance when generating keyphrases. Moreover, either a soft or a hard exclusion mechanism is innovatively incorporated in the decoding process to improve generation diversity.
3 Notations and Problem Definition
We denote vectors and matrices with bold lowercase and uppercase letters respectively. Sets are denoted with calligraphy letters. We use  $\mathbf{W}$  to represent a parameter matrix.
We define the keyphrase generation problem as follows. The input is a document  $\mathbf{x}$ , the output is a keyphrase set  $\mathcal{Y} = \{\mathbf{y}^i\}_{i=1,\dots,|\mathcal{Y}|}$ , where  $|\mathcal{Y}|$  is the keyphrase number of  $\mathbf{x}$ . Both the  $\mathbf{x}$  and each  $\mathbf{y}^i$  are sequences of words, i.e.,  $\mathbf{x} = [x_1,\dots,x_{l_{\mathbf{x}}}]$  and  $\mathbf{y}^i = [y_1^i,\dots,y_{l_{\mathbf{y}^i}}^i]$ , where  $l_{\mathbf{x}}$  and  $l_{\mathbf{y}^i}$  are the word numbers of  $\mathbf{x}$  and  $\mathbf{y}^i$  correspondingly.
4 Our Methodology
We first encode each word of the document into a hidden state and then employ our exclusive hierarchical decoding shown in Figure 2 to produce keyphrases for the given document. Our hierarchical decoding process consists of phrase-level decoding (PD) and word-level decoding (WD). Each PD step decides an appropriate aspect to summarize based on both the context of the document and the aspects summarized by previous PD steps. Then, the hidden representation of the captured aspect is employed to initialize the WD process to generate a new keyphrase word by word. The WD process terminates when producing a “[eowd]” token. If the WD process output a “[eopd)” token, the whole hierarchical decoding process stops. Both PD and WD attend the document content. The PD attention score is used to re-weight the WD attention score to provide aspect guidance. To improve the diversity of the predicted keyphrases, we incorporate either an exclusive loss when training (i.e., the soft exclusion mechanism) or an exclusive search mechanism when inference (i.e., the hard exclusion mechanism).
4.1 Sequential Encoder
To obtain the context-aware representation of each document word, we employ a two-layered bidirectional GRU (Cho et al., 2014) as the document encoder:  $\mathbf{m}_k = \mathrm{BiGRU}(\mathbf{e}_{x_k},\overleftrightarrow{\mathbf{m}}_{k - 1},\overleftarrow{\mathbf{m}}_{k + 1})$  where  $k = 1,2,\ldots ,l_{\mathbf{x}}$  and  $\mathbf{e}_{x_k}$  is the embedding vector of  $x_{k}$  with  $d_{e}$  dimensions.  $\mathbf{m}_k = [\overrightarrow{\mathbf{m}}_k;\overleftarrow{\mathbf{m}}_k]\in \mathbb{R}^d$
Figure 2: Illustration of our exclusive hierarchical decoding.  $\mathbf{h}_i$  is the hidden state of  $i$ -th PD step.  $\mathbf{h}_{i,j}$  is the corresponding  $j$ -th WD hidden state. The “[neopd)” token means PD does not end. The “[eowd)” token means WD terminates. The “[eopd)” token means PD ends and the whole decoding process finishes. “[m1, ..., m $l_{\mathbf{x}}$ ] represents the encoded hidden states from the document. “PD-Attention” and “WD-Attention” are the attention mechanisms in PD and WD respectively. “ $\beta_i$ ” is the PD attention score at  $i$ -th step.  $\tilde{\mathbf{h}}_{i,j}$  is the WD attentional vector. “EL/ES” indicates either the exclusive loss or the exclusive search is incorporated.
is the encoded context-aware representation of  $x_{k}$ . Here, “ $[\cdot ;\cdot ]$ ” means concatenation.
4.2 Hierarchical Decoder
Our hierarchical decoding process is controlled by the hierarchical decoder, which utilizes a phrase-level decoder and a word-level decoder to handle the PD process and the WD process respectively. We present our hierarchical decoder first and then introduce the exclusion mechanisms. In our decoders, all the hidden states and attentional vectors are  $d$ -dimensional vectors.
4.2.1 Phrase-level Decoder
We adopt a unidirectional GRU layer as our phrase-level decoder. After the WD process under last PD step is finished, the phrase-level decoder will update its hidden state as follows:
$$
\mathbf {h} _ {i} = \overrightarrow {\mathrm {G R U}} _ {1} \left(\tilde {\mathbf {h}} _ {i - 1, e n d}, \mathbf {h} _ {i - 1}\right), \tag {1}
$$
where  $\tilde{\mathbf{h}}_{i - 1, end}$  is the attentional vector for the ending WD step under the (i-1)-th PD step (e.g.,  $\tilde{\mathbf{h}}_{2,2}$  in Figure 2(b)).  $\mathbf{h}_i$  is regarded as the hidden representation of the captured aspect at the  $i$ -th PD step.  $\mathbf{h}_0$  is initialized as the document representation  $[\overline{\mathbf{m}}_{l_{\mathbf{x}}};\overleftarrow{\mathbf{m}}_1]$ .  $\tilde{\mathbf{h}}_{0, end}$  is initialized with zeros.
In PD-Attention process, the PD attentional score  $\beta_{i} = [\beta_{i,1},\beta_{i,2},\dots ,\beta_{i,l_{\mathbf{x}}}]$  is computed from the following attention mechanism employing  $\mathbf{h}_i$
as the query vector:
$$
\beta_ {i, k} = \exp (s _ {i, k}) / \sum_ {n = 1} ^ {l _ {\mathbf {x}}} \exp (s _ {i, n}), \tag {2}
$$
$$
s _ {i, n} = \left(\mathbf {h} _ {i}\right) ^ {T} \mathbf {W} _ {1} \mathbf {m} _ {n}. \tag {3}
$$
4.2.2 Word-level Decoder
We choose another unidirectional GRU layer to conduct word-level decoding. Under the  $i$ -th PD step, the word-level decoder updates its hidden state first:
$$
\mathbf {h} _ {i, j} = \overrightarrow {\mathrm {G R U}} _ {2} \left(\left[ \tilde {\mathbf {h}} _ {i, j - 1}; \mathbf {e} _ {y _ {j - 1} ^ {i}} \right], \mathbf {h} _ {i, j - 1}\right), \tag {4}
$$
where  $\tilde{\mathbf{h}}_{i,j - 1}$  is the WD attentional vector of the (j-1)-th WD step and  $\mathbf{e}_{y_{j - 1}^i}$  is the  $d_{e}$ -dimensional embedding vector of the  $y_{j - 1}^{i}$  token. We define  $\mathbf{h}_{i,0} = \overrightarrow{\mathrm{GRU}_2} ([\mathbf{0};\mathbf{e}_s],\mathbf{h}_i)$ , where  $\mathbf{h}_i$  is the current hidden state of the phrase-level decoder,  $\mathbf{0}$  is a zero vector, and  $\mathbf{e}_s$  is the embedding of the start token. Then, the WD attentional vector is computed:
$$
\tilde {\mathbf {h}} _ {i, j} = \tanh  \left(\mathbf {W} _ {2} \left[ \mathbf {h} _ {i, j}; \mathbf {a} _ {i, j} \right]\right), \tag {5}
$$
$$
\mathbf {a} _ {i, j} = \sum_ {k = 1} ^ {l _ {\mathbf {x}}} \bar {\alpha} _ {(i, j), k} \mathbf {m} _ {k}, \tag {6}
$$
$$
\bar {\alpha} _ {(i, j), k} = \frac {\alpha_ {(i , j) , k} \times \beta_ {i , k}}{\sum_ {n = 1} ^ {l _ {\mathbf {x}}} \alpha_ {(i , j) , n} \times \beta_ {i , n}}, \tag {7}
$$
where  $\alpha_{(i,j),k}$  is the original WD attention score which is computed similar to  $\beta_{i,k}$  except that a
new parameter matrix is used and  $\mathbf{h}_{i,j}$  is employed as the query vector. The purpose of the rescaling operation in Eq. (7) is to indicate the focused aspect of the current PD step for each WD step.
Finally, the  $\tilde{\mathbf{h}}_{i,j}$  is utilized to predict the probability distribution of current keyword with the copy mechanism (See et al., 2017):
$$
P _ {j} ^ {i} = \left(1 - g _ {j} ^ {i}\right) P _ {j, \mathcal {V}} ^ {i} + g _ {j} ^ {i} P _ {j, \mathcal {X}} ^ {i}, \tag {8}
$$
where  $g_{j}^{i} = \mathrm{sigmoid}(\mathbf{w}_{g}^{T}\tilde{\mathbf{h}}_{i,j} + b_{g})\in \mathbb{R}$  is the copy gate.  $P_{j,\mathcal{V}}^i = \mathrm{softmax}(\mathbf{W}_3\tilde{\mathbf{h}}_{i,j} + \mathbf{b}_{\mathcal{V}})\in \mathbb{R}^{|\mathcal{V}|}$  is the probability distribution over a predefined vocabulary  $\nu$ $P_{j,\mathcal{X}}^i = \sum_{k:x_k = y_j^i}\bar{\alpha}_{(i,j),k}\in \mathbb{R}^{|\mathcal{X}|}$  is the copying probability distribution over  $\mathcal{X}$  which is a set of all the words that appeared in the document.  $P_{j}^{i}\in \mathbb{R}^{|\mathcal{V}\cup \mathcal{X}|}$  is the final predicted probability distribution. Finally, greedy search is applied to produce the current token.
The WD process terminates when producing a "[eowd]" token. The whole hierarchical decoding process ends if the word-level decoder produces a "[eopd]" token at the 0-th step, i.e.,  $y_0^i$  is predicted as "[eopd]".
4.3 Training
A standard negative log-likelihood loss is employed as the generation loss to train our hierarchical decoding model:
$$
\mathcal {L} _ {g} = - \sum_ {i = 1} ^ {| \bar {y} |} \sum_ {j = 0} ^ {l _ {\bar {y}} i} \log P _ {j} ^ {i} \left(\bar {y} _ {j} ^ {i} | \mathbf {x}; \bar {\mathbf {Y}} ^ {i - 1}; \bar {\mathbf {y}} _ {j - 1} ^ {i}\right), \tag {9}
$$
where  $\bar{\mathbf{Y}}^{i - 1} = \bar{\mathbf{y}}^1,\dots ,\bar{\mathbf{y}}^{i - 1}$  are the target keyphrases of previously-finished PD steps and  $\bar{\mathbf{y}}_{j - 1}^{i} = \bar{y}_{0}^{i},\ldots ,\bar{y}_{j - 1}^{i}$  are target keyphrase words of previous WD steps under the  $i$  -th PD step. When training, each original target keyphrase is extended with a "[neopd]" token and a "[eowd]" token, i.e.,  $\bar{\mathbf{y}}^i = ["[\mathrm{neopd}]^\prime ,y_1^i,\dots ,y_{l_\mathbf{y}^i}^i,"[\mathrm{eowd}]^\prime ]$  . Besides, a "[eopd]" token is also incorporated into the targets to indicate the ending of whole decoding process. Teacher forcing is employed when training.
4.4 Soft and Hard Exclusion Mechanisms
To alleviate the duplication generation problem, we propose a soft and a hard exclusion mechanisms. Either of them can be incorporated into our hierarchical decoding process to form one kind of exclusive hierarchical decoding method.
Soft Exclusion Mechanism. An exclusive loss (EL) is introduced in the training stage as shown
Algorithm 1 Training with Exclusive Loss
Require: The window size  $K_{EL}$ . The target keyphrases  $[\bar{\mathbf{y}}^1, \dots, \bar{\mathbf{y}}^i, \dots, \bar{\mathbf{y}}^{|\hat{\mathcal{Y}}|}]$ . The predicted probability distribution  $P_j^i$  for the  $j$ -th WD step under the  $i$ -th PD step where  $i = 1, \dots, |\bar{\mathcal{Y}}|$  and  $j = 0, 1, \dots, l_{\bar{\mathbf{y}}^i}$ .
1: Firstly, the exclusive loss of the  $j$ -th WD step under the  $i$ -th PD step is computed as follows.
2:  $K_{EL} \gets \min \{K_{EL}, i - 1\}$
3: if  $K_{EL} > 0$  and  $j == 1$  then
4:  $\mathcal{L}_{EL}^{i,j} = \sum_{idx = i - K_{EL},\bar{y}_j^{idx}\neq \bar{y}_j^i}^{i - 1} - \log (1 - P_j^i (\bar{y}_j^{idx}))$
5: else
6:  $\mathcal{L}_{EL}^{i,j} = 0.0$
7: end if
8: Secondly, the exclusive loss for the whole decoding process is calculated as  $\mathcal{L}_{EL} = \sum_{i,j}\mathcal{L}_{EL}^{i,j}$ .
9: Finally, the joint loss  $\mathcal{L} = \mathcal{L}_g + \mathcal{L}_{EL}$  is employed to train the model.
Algorithm 2 Inference with Exclusive Search
Require: The window size  $K_{ES}$ . The first words of previously-predicted keyphrases  $[y_1^1, \ldots, y_1^{i-1}]$ . The current WD step index  $j$ . The predicted probability distribution  $P_j^i$  for current WD step.
1:  $K_{ES} \gets \min \{K_{ES}, i - 1\}$
2: if  $K_{ES} > 0$  and  $j == 1$  then
3: for  $idx = i - K_{ES}, i - K_{ES} + 1, \ldots, i - 1$  do
4:  $P_{j}^{i}(y_{j}^{idx})\gets 0.0$
5: end for
6: end if
7: Return  $y_{j}^{i} = \arg \max (P_{j}^{i})$  as the predicted word for current WD step.
in Algorithm 1. "  $j == 1$ " in line "3" means the current WD step is predicting the first word of a keyphrase. In short, the exclusive loss punishes the model for the tendency to generate the same first word of the current keyphrase with the first words of previously-generated keyphrases within the window size  $K_{EL}$ .
Hard Exclusion Mechanism. An exclusive search (ES) is introduced in the inference stage as shown in Algorithm 2. The exclusive search mechanism forces the word-level decoding to predict a different first word with the first words of previously-predicted keyphrases within the window size  $K_{ES}$ .
Since a keyphrase usually has only two or three words, the first word significantly affects the prediction of the following words. Therefore, both the soft and the hard exclusion mechanisms can improve the diversity of generated keyphrases.
Our model implementations are based on the OpenNMT system (Klein et al., 2017) using PyTorch (Paszke et al., 2017). Experiments of all
models are repeated with three different random seeds and the averaged results are reported.
We employ four scientific article benchmark datasets to evaluate our models, including KP20k (Meng et al., 2017), Inspec (Hulth, 2003), Krapivin (Krapivin et al., 2009), and SemEval (Kim et al., 2010). Following previous work (Yuan et al., 2018; Chen et al., 2019a), we use the training set of KP20k to train all the models. After removing the duplicated data, we maintain 509,818 data samples in the training set, 20,000 in the validation set, and 20,000 in the testing set. After training, we test all the models on the testing datasets of these four benchmarks. The dataset statistics are shown in Table 1.
Dataset	Total	Validation	Testing
Inspec	2,000	1,500	500
Krapivin	2,303	1,903	400
SemEval	244	144	100
KP20k	549,818	20,000	20,000
Table 1: The statistics of validation and testing datasets.
We focus on the comparisons with state-of-the-art decoding methods and choose the following generation models under the new setting as our baselines: