SlowGuess/NAACL_2021_OCR huggingface dataset:
Jing Zhao, Junwei Bao, Yifan Wang, Youzheng Wu, Xiaodong He, Bowen Zhou JD AI Research
{zhaojing857,baojunwei, wangyifan15,wuyouzheng1}@jd.com {xiaodong.he,bowen.zhou}@jd.com
Keyphrases, that concisely summarize the high-level topics discussed in a document, can be categorized into present keyphrase which explicitly appears in the source text, and absent keyphrase which does not match any contiguous subsequence but is highly semantically related to the source. Most existing keyphrase generation approaches synchronously generate present and absent keyphrases without explicitly distinguishing these two categories. In this paper, a Select-Guide-Generate (SGG) approach is proposed to deal with present and absent keyphrase generation separately with different mechanisms. Specifically, SGG is a hierarchical neural network which consists of a pointing-based selector at low layer concentrated on present keyphrase generation, a selection-guided generator at high layer dedicated to absent keyphrase generation, and a guider in the middle to transfer information from selector to generator. Experimental results on four keyphrase generation benchmarks demonstrate the effectiveness of our model, which significantly outperforms the strong baselines for both present and absent keyphrases generation. Furthermore, we extend SGG to a title generation task which indicates its extensibility in natural language generation tasks.
Automatic keyphrase prediction recommends a set of representative phrases that are related to the main topics discussed in a document (Liu et al., 2009). Since keyphrases can provide a high-level topic description of a document, they are beneficial for a wide range of natural language processing (NLP) tasks, such as information extraction (Wan and Xiao, 2008), text summarization (Wang and Cardie, 2013) and question generation (Subramanian et al., 2018).
Figure 1: An example of keyphrase prediction by SGG.
Existing methods for keyphrase prediction can be categorized into extraction and generation approaches. Specifically, keyphrase extraction methods identify important consecutive words from a given document as keyphrases, which means that the extracted keyphrases (denoted as present keyphrases) must exactly come from the given document. However, some keyphrases (denoted as absent keyphrases) of a given document do not match any contiguous subsequence but are highly semantically related to the source text. The extraction methods fail to predict these absent keyphrases. Therefore, generation methods have been proposed to produce a keyphrase verbatim from a predefined vocabulary, no matter whether the generated keyphrase appears in the source text. Compared with conventional extraction methods, generation methods have the ability of generating absent keyphrases as well as present keyphrases.
CopyRNN (Meng et al., 2017) is the first to employ the sequence-to-sequence (Seq2Seq) framework (Sutskever et al., 2014) with the copying mechanism (Gu et al., 2016) to generate keyphrases for the given documents. Following the CopyRNN, several Seq2Seq-based keyphrase generation approaches have been proposed to improve the generation performance (Chen et al., 2018; Ye and Wang, 2018; Chen et al., 2019; Zhao and Zhang, 2019; Wang et al., 2019; Yuan et al., 2020). All these existing methods generate present and absent keyphrases synchronously without ex
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5717-5726 June 6-11, 2021. ©2021 Association for Computational Linguistics
Training(%)	Test(%)
Inspec	Krapiven	NUS	SemEval
49.79	13.12	11.74	11.30	11.25
Table 1: Proportions of absent keyphrases in training set and predictions of CopyRNN on four commonly used datasets, where top-10 predictions are considered.
plicity, identifying these two different categories of keyphrases, which leads to two problems: (1) They complicate the identification of present keyphrases. Specifically, they search for words over the entire predefined vocabulary containing a vast amount of words (e.g., 50,000 words) to generate a present keyphrase verbatim, which is overparameterized since a present keyphrase can be simply selected from a continuous subsequence of the source text containing limited words (e.g., less than 400 words). (2) They weaken the generation of absent keyphrases. Existing models for absent keyphrase generation are usually trained on datasets mixed with a large proportion of present keyphrases. Table 1 shows that nearly half of the training data are present keyphrases, which leads to the extremely low proportions of absent keyphrases generated by such a model, i.e., CopyRNN. The above observation demonstrates that these methods are biased towards replicating words from source text for present keyphrase generation, which will inevitably affect the performance on generating absent keyphrases.
To address the aforementioned problems, we propose a Select-Guide-Generate (SGG) approach, which deals with present and absent keyphrase generation separately with different stages based on different mechanisms. Figure 1 illustrates an example of keyphrase prediction by SGG. The motivation behind is to solve keyphrase generation problem from selecting to generating, and use the selected results to guide the generation. Specifically, our SGG is implemented with a hierarchical neural network which performs Seq2Seq learning by applying a multi-task learning strategy. This network consists of a selector at low layer, a generator at high layer, and a guider at middle layer for information transfer. The selector generates present keyphrases through a pointing mechanism (Vinyals et al., 2015), which adopts attention distributions to select a sequence of words from the source text as output. The generator further generates the absent keyphrases through a pointing-generating (PG) mechanism (See et al., 2017). Since present
keyphrases have already been generated by the selector, they should not be generated again by the generator. Therefore, a guider is designed to memorize the generated present keyphrases from the selector, and then fed into the attention module of the generator to constrain it to focus on generating absent keyphrases. We summarize our main contributions as follows:
- We propose a SGG approach which models present and absent keyphrase generation separately in different stages, i.e., select, guide, and generate, without sacrificing the end-to-end training through back-propagation.
- Extensive experiments are conducted to verify the effectiveness of our model, which not only improves present keyphrase generation but also dramatically boosts the performance of absent keyphrase generation.
- Furthermore, we adopt SGG to a title generation task, and the experiment results indicate the extensibility and effectiveness of our SGG approach on generation tasks.
As mentioned in Section 1, the extraction and generation methods are two different research directions in the field of keyphrase prediction. The existing extraction methods can be broadly classified into supervised and unsupervised approaches. The supervised approaches treat keyphrase extraction as a binary classification task, which train the models with the features of labeled keyphrases to determine whether a candidate phrase is a keyphrase (Witten et al., 1999; Medelyan et al., 2009; Gollapalli et al., 2017). In contrast, the unsupervised approaches treat keyphrase extraction as a ranking task, scoring each candidate using some different ranking metrics, such as clustering (Liu et al., 2009), or graph-based ranking (Mihalcea and Tarau, 2004; Wang et al., 2014; Gollapalli and Caragea, 2014; Zhang et al., 2017).
This work is mainly related to keyphrase generation approaches which have demonstrated good performance on keyphrase prediction task. Following CopyRNN (Meng et al., 2017), several extensions have been proposed to boost the generation capability. In CopyRNN, model training heavily relies on large amount of labeled data, which is often unavailable especially for the new domains. To address this problem, Ye and Wang (2018) proposed
a semi-supervised keyphrase generation model that utilizes both abundant unlabeled data and limited labeled data. CopyRNN uses the concatenation of article title and abstract as input, ignoring the leading role of the title. To address this deficiency, Chen et al. (2019) proposed a title-guided Seq2Seq network to sufficiently utilize the already summarized information in title. In addition, some research attempts to introduce external knowledge into keyphrase generation, such as syntactic constraints (Zhao and Zhang, 2019) and latent topics (Wang et al., 2019).
These approaches do not consider the one-to-many relationship between the input text and target keyphrases, and thus fail to model the correlation among the multiple target keyphrases. To overcome this drawback, Chen et al. (2018) incorporated the review mechanism into keyphrase generation and proposed a model CorrRNN with correlation constraints. Similarly, SGG separately models one-to-many relationship between the input text and present keyphrases and absent keyphrases. To avoid generating duplicate keyphrases, Chen et al. (2020) proposed an exclusive hierarchical decoding framework that includes a hierarchical decoding process and either a soft or a hard exclusion mechanism. For the same purpose, our method deploys a guider to avoid the generator generating duplicate present keyphrases. Last but most important, all these methods do not consider the difference between present and absent keyphrases. We are the first to discriminately treat present and absent keyphrases in keyphrase generation task.
Given a dataset including  $K$  data samples, where the  $j$ -th data item  $\langle x^{(j)}, y^{(j,p)}, y^{(j,a)} \rangle$  consists of a source text  $x^{(j)}$ , a set of present keyphrases  $y^{(j,p)}$  and a set of absent keyphrases  $y^{(j,a)}$ . Different from CopyRNN (Meng et al., 2017) splitting each data item into multiple training examples, each of which contains only one keyphrase as target, we regard each data item as one training example by concatenating its present keyphrases as one target and absent keyphrases as another one. Specifically, assume that the  $j$ -th data item consists of  $m$  present keyphrases  $\{y_1^{(j,p)}, \dots, y_m^{(j,p)}\}$  and  $n$  absent keyphrases  $\{y_1^{(j,a)}, \dots, y_n^{(j,a)}\}$ , the target present keyphrases  $y^{(j,p)}$  and target absent
keyphrases  $y^{(j,a)}$  are represented as:
$$
y ^ {(j, p)} = y _ {1} ^ {(j, p)} | | y _ {2} ^ {(j, p)} | | \dots | | y _ {m} ^ {(j, p)}
$$
$$
y ^ {(j, a)} = y _ {1} ^ {(j, a)} | | y _ {2} ^ {(j, a)} | | \dots | | y _ {n} ^ {(j, a)}
$$
where  $||$  is a special splitter to separate the keyphrases. We then get the source text  $x^{(j)}$ , the present keyphrases  $y^{(j,p)}$  and the absent keyphrases  $y^{(j,a)}$  all as word sequences. Under this setting, our model is capable of generating multiple keyphrases in one sequence as well as capturing the mutual relations between these keyphrases. A keyphrase generation model is to learn the mapping from the source text  $x^{(j)}$  to the target keyphrases  $(y^{(j,p)}, y^{(j,a)})$ . For simplicity,  $(x, y^p, y^a)$  is used to denote each item in the rest of this paper, where  $x$  denotes a source text sequence,  $y^p$  denotes its present keyphrase sequence and  $y^a$  denotes its absent keyphrase sequence.
The architecture of our proposed Select-Guide-Generate (SGG) approach is illustrated in Figure 2. Our model is the extension of Seq2Seq framework which consists of a text encoder, a selector, a guider, and a generator. The text encoder converts the source text  $x$  into a set of hidden representation vectors  $\{\mathbf{h}_i\}_{i=1}^L$  with a bidirectional Long Short-term Memory Network (bi-LSTM) (Hochreiter and Schmidhuber, 1997), where  $L$  is the length of source text sequence. The selector is a uni-directional LSTM, which predicts the present keyphrase sequence  $y^p$  based on the attention distribution over source words. After selecting present keyphrases, a guider is produced by a guider to memorize the prediction information of the selector, and then fed to the attention module of a generator to adjust the information it pays attention to. The selection-guided generator is also implemented as a uni-directional LSTM, which produces the absent keyphrase sequence  $y^a$  based on two distributions over predefined-vocabulary and source words, respectively. At the same time, a soft switch gate  $p_{gen}$  is employed as a trade-off between the above two distributions.
3.3 Text Encoder
The goal of a text encoder is to provide a series of dense representations  $\{\mathbf{h}_i\}_{i = 1}^L$  of the source text. In our model, the text encoder is implemented as a bi-LSTM (Hochreiter and Schmidhuber, 1997) which reads an input sequence  $x = \{x_{i}\}_{i = 1}^{L}$  from
Figure 2: The architecture of the proposed SGG which is implemented with a hierarchical neural network.
two directions and outputs a sequence of forward hidden states  $\{\vec{\mathbf{h}}_i\}_{i = 1}^L$  and backward hidden states  $\{\overleftarrow{\mathbf{h}_i}\}_{i = 1}^L$  by iterating the following equations:
$$
\overrightarrow {\mathbf {h} _ {i}} = \operatorname {L S T M} \left(x _ {i}, \mathbf {h} _ {i - 1}\right) \tag {1}
$$
$$
\overleftarrow {\mathbf {h} _ {i}} = \operatorname {L S T M} \left(x _ {i}, \mathbf {h} _ {i + 1}\right) \tag {2}
$$
The final hidden representation  $\mathbf{h}_i$  of the  $i$ -th source word is the concatenation of forward and backward hidden states, i.e.,  $\mathbf{h}_i = [\vec{\mathbf{h}}_i; \vec{\mathbf{h}}_i]$ .
3.4 Selector
A selector is designed to generate present keyphrase sequences through the pointer mechanism (Vinyals et al., 2015), which adopts the attention distribution as a pointer to select words from the source text as output. Specifically, given source text sequence  $x$  and previously generated words  $\{y_1^p,\dots,y_{t - 1}^p\}$ , the probability distribution of predicting next word  $y_{t}^{p}$  in present keyphrases is:
$$
\begin{array}{l} \mathcal {P} \left(y _ {t} ^ {p} \mid y _ {<   t} ^ {p}, x\right) = \alpha^ {p, t} = \operatorname {s o f t m a x} (\mathbf {u} ^ {p, t}) \tag {3} \\ u _ {i} ^ {p, t} = \mathbf {V} _ {p} ^ {T} \mathrm {t a n h} (\mathbf {W} _ {p} [ \mathbf {s} _ {t} ^ {p}; \mathbf {h} _ {i} ] + \mathbf {b} _ {p}) \quad (4) \\ \end{array}
$$
where  $\alpha^{p,t}$  is the attention (Bahdanau et al., 2015) distribution at decoding time step  $t$ ,  $i \in (1, \dots, L)$ , and  $\mathbf{V}_p$ ,  $\mathbf{W}_p$  and  $\mathbf{b}_p$  are trainable parameters of the model.  $\mathbf{u}^{p,t}$  can be viewed as the degree of matching between input at position  $i$  and output at
position  $t$ .  $\mathbf{s}_t^p$  represents the hidden state at deciding time step  $t$ , and is updated by equation:
$$
\mathbf {s} _ {t} ^ {p} = \operatorname {L S T M} \left(y _ {t - 1} ^ {p}, \mathbf {s} _ {t - 1} ^ {p}, \mathbf {c} _ {t - 1} ^ {p}\right) \tag {5}
$$
where context vector  $\mathbf{c}_{t - 1}^p = \sum_{i = 1}^L\alpha_i^{p,t - 1}\mathbf{h}_i$  is the weighted sum of source hidden states.
3.5 Guider
A guider is designed to fully utilize the attention information of the selector to guide the generator on absent keyphrase generation. The idea behind is to utilize a guider  $\mathbf{r}$  to softly indicate which words in source text have been generated by the selector. This is important for helping the generator to focus on generating the absent keyphrases. Specifically,  $\mathbf{r}$  is constructed through the accumulation of the attention distributions over all decoding time steps of the selector, computed as:
$$
\mathbf {r} = \sum_ {t = 1} ^ {M} \alpha^ {p, t} \tag {6}
$$
where  $M$  is the length of present keyphrase sequence.  $\mathbf{r}$  is an unnormalized distribution over the source words. As the attention distribution of selector is equal to the probability distribution over the source words,  $\mathbf{r}$  represents the possibility that these words have been generated by the selector. The calculation of guider is inspired by the coverage vector (Tu et al., 2016) that is sequentially updated during the decoding process. In contrast to this, the guider here is a static vector which is capable of memorizing a global information.
3.6 Selection-Guided Generator
A generator aims to predict an absent keyphrase sequence based on the guidance of the selection information from the guider. Unlike present keyphrases, most words in absent keyphrases do not appear in source text. Therefore, the generator generates absent keyphrases by picking up words from both a predefined large scale vocabulary and the source text (See et al., 2017; Gu et al., 2016). The probability distribution of predicting next word  $y_{t}^{a}$  in absent keyphrases is defined as:
$$
\begin{array}{l} \mathcal {P} (y _ {t} ^ {a} \mid y _ {<   t} ^ {a}, x) \\ = p _ {g e n} \mathcal {P} _ {v o c a b} \left(y _ {t} ^ {a}\right) + \left(1 - p _ {g e n}\right) \sum_ {i: y _ {t} ^ {a} = x _ {i}} \alpha_ {i} ^ {a, t} \tag {7} \\ \end{array}
$$
where  $\mathcal{P}_{\text{vocab}}$  is the probability distribution over the predefined vocabulary, which is zero if  $y_{t}^{a}$  is an out-of-vocabulary (OOV) word. Similarly, if  $y_{t}^{a}$  does
not appear in the source text, then  $\sum_{i:y_t^a = x_i}\alpha_i^{a,t}$  is zero.  $\mathcal{P}_{\text{vocab}}$  is computed as:
$$
\mathcal {P} _ {\text {v o c a b}} \left(y _ {t} ^ {a}\right) = \operatorname {s o f t m a x} \left(\mathbf {W} \left[ \mathbf {s} _ {t} ^ {a}; \mathbf {c} _ {t} ^ {a} \right] + \mathbf {b}\right) \tag {8}
$$
where  $\mathbf{W}$  and  $\mathbf{b}$  are learnable parameters,  $\mathbf{s}_t^a$  is the hidden state of generator, and  $\mathbf{c}_t^a$  is the context vector for generating absent keyphrase sequence, computed by the following equations:
$$
\mathbf {c} _ {t} ^ {a} = \sum_ {i = 1} ^ {L} \alpha_ {i} ^ {a, t} \mathbf {h} _ {i} \tag {9}
$$
$$
\alpha^ {a, t} = \operatorname {s o f t m a x} (\mathbf {u} ^ {a, t}) \tag {10}
$$
$$
u _ {i} ^ {a, t} = \mathbf {V} _ {a} ^ {T} \tanh  \left(\mathbf {W} _ {a} \left[ \mathbf {s} _ {t} ^ {a}; \mathbf {h} _ {i}; \mathbf {r} \right] + \mathbf {b} _ {a}\right) \tag {11}
$$
where  $\mathbf{V}_a$ ,  $\mathbf{W}_a$  and  $\mathbf{b}_a$  are learnable parameters.  $\mathbf{r}$  is a vector produced by the guider. The generation probability  $p_{gen}$  at time step  $t$  is computed as:
$$
p _ {g e n} = \sigma \left(\mathbf {W} _ {g e n} \left[ \mathbf {c} _ {t} ^ {a}; \mathbf {s} _ {t} ^ {a}; \operatorname {e m b} \left(y _ {t - 1} ^ {a}\right) \right] + \mathbf {b} _ {g e n}\right) \tag {12}
$$
where  $\mathbf{W}_{gen}$  and  $\mathbf{b}_{gen}$  are learnable parameters,  $\sigma (\cdot)$  represents a sigmoid function and  $\operatorname {emb}(y_{t - 1}^a)$  is the embedding of  $y_{t - 1}^{a}$ . In addition,  $p_{gen}$  in formula (7) is used as a soft switch to choose either generating words over vocabulary or copying words from source text based on distribution  $\alpha^{a,t}$ .
3.7 Training
Given the set of data pairs  $\{x^{(j)},y^{(j,p)},y^{(j,a)}\}_{j = 1}^{K}$ , the loss function of the keyphrase generation consists of two parts of cross entropy losses:
$$
\mathcal {L} _ {p} (\theta) = - \sum_ {j = 1} ^ {K} \sum_ {i = 1} ^ {M} \log \left(\mathcal {P} \left(y _ {i} ^ {(j, p)} \mid x ^ {(j)}; \theta\right)\right) \tag {13}
$$
$$
\mathcal {L} _ {a} (\theta) = - \sum_ {j = 1} ^ {K} \sum_ {i = 1} ^ {N} \log \left(\mathcal {P} \left(y _ {i} ^ {(j, a)} \mid x ^ {(j)}; \theta\right)\right) \tag {14}
$$
where  $\mathcal{L}_p$  and  $\mathcal{L}_a$  are the losses of generating present and absent keyphrases, respectively.  $N$  is the word sequence length of absent keyphrases, and  $\theta$  are the parameters in our model. The training objective is to jointly minimize the two losses:
$$
\mathcal {L} = \mathcal {L} _ {p} + \mathcal {L} _ {a}. \tag {15}
$$
We use the dataset collected by Meng et al. (2017) from various online digital libraries, which contains approximately 570K samples, each of which
contains a title and an abstract of a scientific publication as source text, and author-assigned keywords as target keyphrases. We randomly select the example which contains at least one present keyphrase to construct the training set. Then, a validation set containing 500 samples will be selected from the remaining examples. In order to evaluate our proposed model comprehensively, we test models on four widely used public datasets from the scientific domain, namely Inspec (Hulth and Megyesi, 2006), Krapivin (Krapivin et al., 2009), SemEval2010 (Kim et al., 2010) and NUS (Nguyen and Kan, 2007), the statistic information of which are summarized in Table 2.
Dataset	#Abs	#PKPs	#AKPs
Test	Inspec	500	3,654	1,349
Krapivin	400	1,299	1,040
NUS	211	1,333	1,128
SemEval	100	625	841
Validation	500	1,158	1,418
Training	453,757	1,082,285	1,073,404
Table 2: Statistics of the dataset. #Abs, #PKPs, #AKPs denote the number of abstracts, present keyphrases, and absent keyphrases, respectively.
4.2 Baselines and Evaluation Metrics
For present keyphrase prediction, we compare our model with both extraction and generation approaches. Extraction approaches include two unsupervised extraction methods: TF-IDF, TextRank (Mihalcea and Tarau, 2004) and one classic supervised extraction method KEA (Witten et al., 1999). For the generation baselines, some models, such as CopyRNN, split each data item into multiple training examples, each of which only contains one keyphrase, while the other models concatenate all keyphrases as target. To simplicity, the pattern of training model only with one keyphrase is denoted as one-to-one and with the concatenation of all keyphrases as one-to-many. The generation baselines are the following state-of-the-art encoder-decoder models:
- CopyRNN(one-to-one) (Meng et al., 2017) represents a RNN-based encoder-decoder model incorporating the copying mechanism.
- CopyTrans(one-to-many) is a transformer-based (Vaswani et al., 2017) encoder-decoder model incorporating the copying mechanism.
Method	Inspec	Krapivin	NUS	SemEval
F1@5	F1@10	F1@5	F1@10	F1@5	F1@10	F1@5	F1@10
TF-IDF	22.1	31.3	12.9	16.0	13.6	18.4	12.8	19.4
TextRank	22.3	28.1	18.9	16.2	19.5	19.6	17.6	18.7
KEA	9.8	12.6	11.0	15.2	6.9	8.4	2.5	2.6
CopyRNN	27.8	34.2	31.1	26.6	33.4	32.6	29.3	30.4
\( CopyTrans^† \)	21.1	16.2	26.4	20.5	35.1	28.2	29.5	26.3
CorrRNN	-	-	31.8	27.8	35.8	33.0	32.0	32.0
CatSeq	29.0	30.0	30.7	27.4	35.9	34.9	30.2	30.6
SGG	30.6	35.9	28.8	25.3	36.3	35.8	33.8	33.6
Table 3: F1@5/10 results of predicting present keyphrases of different models on four datasets. The best and second best performance in each column are highlighted with bold and underline respectively.  $\dagger$  indicates that the model is reimplemented.
Method	Inspec	Krapivin	NUS	SemEval
CopyRNN	10.0	20.2	11.6	6.7
CopyTrans†	5.6	16.9	8.9	4.1
CorrRNN†	8.5	15.2	8.0	3.5
CatSeq	2.9	7.4	3.1	2.5
SGG	11.0	23.5	12.4	4.9
Table 4: Recall@50 results of predicting absent keyphrases of different models on four datasets. The CorrRNN is retrained following the implementation details in Chen et al. (2018) as they did not report the Recall@50 results.
- CorrRNN(one-to-many) (Chen et al., 2018) is an extension of CopyRNN incorporating the coverage mechanism (Tu et al., 2016).
- CatSeq(one-to-many) (Yuan et al., 2020) has the same model structure as CopyRNN. The difference is CatSeq is trained by one-to-many.
The baseline CopyTrans has not been reported in existing papers and thus is retrained. The implementation of Transformer is based on open source tool OpenNMT<sup>1</sup>. For our experiments of absent keyphrase generation, only generation methods are chosen as baselines. The copying mechanism used in all implemented generation models is based on the version (See et al., 2017), which is slightly different from the implementations by version (Meng et al., 2017; Gu et al., 2016). SGG indicates the full version of our proposed model, which contains a selector, a guider, and a generator. Note that SGG is also trained under one-to-many pattern.
Same as CopyRNN, we adopt top-  $N$  macroaveraged  $F$ -measure (F1) and recall as our evalua