SlowGuess/ACL_2020_OCR huggingface dataset:
Xingdi Yuan†* Tong Wang†* Rui Meng‡* Khushboo Thaker‡  
Peter Brusilovsky‡ Daqing He‡ Adam Trischler†
†Microsoft Research, Montréal
$^{\ddagger}$ School of Computing and Information, University of Pittsburgh
{eric.yuan, tong.wang}  $@$  microsoft.com rui.meng@pitt.edu
Different texts shall by nature correspond to different number of keyphrases. This desideratum is largely missing from existing neural keyphrase generation models. In this study, we address this problem from both modeling and evaluation perspectives.
We first propose a recurrent generative model that generates multiple keyphrases as delimiter-separated sequences. Generation diversity is further enhanced with two novel techniques by manipulating decoder hidden states. In contrast to previous approaches, our model is capable of generating diverse keyphrases and controlling number of outputs.
We further propose two evaluation metrics tailored towards the variable-number generation. We also introduce a new dataset (STACKEX) that expands beyond the only existing genre (i.e., academic writing) in keyphrase generation tasks. With both previous and new evaluation metrics, our model outperforms strong baselines on all datasets.
Keyphrase generation is the task of automatically predicting keyphrases given a source text. Desired keyphrases are often multi-word units that summarize the high-level meaning and highlight certain important topics or information of the source text. Consequently, models that can successfully perform this task should be capable of not only distilling high-level information from a document, but also locating specific, important snippets therein.
To make the problem even more challenging, a keyphrase may or may not be a substring of the source text (i.e., it may be present or absent). Moreover, a given source text is usually associated with
Dataset	#Train	#Valid	#Test	Mean	Var	%Pre
KP20K	≈514k	≈20k	≈20k	5.3	14.2	63.3%
INSPEC	-	1500	500	9.6	22.4	78.5%
KRAPIVIN	-	1844	460	5.2	6.6	56.2%
NUS	-	-	211	11.5	64.6	51.3%
SEMEVAL	-	144	100	15.7	15.1	44.5%
STACKEX	≈298k	≈16k	≈16k	2.7	1.4	57.5%
Table 1: Statistics of various datasets. Mean and Var indicate the mean and variance of target phrase numbers, %Pre denotes percentage of present keyphrases.
a set of multiple keyphrases. Thus, keyphrase generation is an instance of the set generation problem, where both the size of the set and the size (i.e., the number of tokens in a phrase) of each element can vary depending on the source.
Similar to summarization, keyphrase generation is often formulated as a sequence-to-sequence (Seq2Seq) generation task in most prior studies (Meng et al., 2017; Chen et al., 2018a; Ye and Wang, 2018; Chen et al., 2018b). Conditioned on a source text, Seq2Seq models generate phrases individually or as a longer sequence jointed by delimiting tokens. Since standard Seq2Seq models generate only one sequence at a time, thus to generate multiple phrases, a common approach is to over-generate using beam search (Reddy et al., 1977) with a large beam width. Models are then evaluated by taking a fixed number of top predicted phrases (typically 5 or 10) and comparing them against the ground truth keyphrases.
Though this approach has achieved good empirical results, we argue that it suffers from two major limitations. Firstly, models that use beam search to generate multiple keyphrases generally lack the ability to determine the dynamic number of keyphrases needed for different source texts. Meanwhile, the parallelism in beam search also fails to model the inter-relation among the generated phrases, which can often result in diminished diversity in the output. Although certain existing models
* These authors contributed equally. The order is determined by a fidget spinner.
7961
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7961-7975 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
take output diversity into consideration during training (Chen et al., 2018a; Ye and Wang, 2018), the effort is significantly undermined during decoding due to the reliance on over-generation and phrase ranking with beam search.
Secondly, the current evaluation setup is rather problematic, since existing studies attempt to match a fixed number of outputs against a variable number of ground truth keyphrases. Empirically, the number of keyphrases can vary drastically for different source texts, depending on a plethora of factors including the length or genre of the text, the granularity of keyphrase annotation, etc. For the several commonly used keyphrase generation datasets, for example, the average number of keyphrases per data point can range from 5.3 to 15.7, with variances sometimes as large as 64.6 (Table 1). Therefore, using an arbitrary, fixed number  $k$  to evaluate entire datasets is not appropriate. In fact, under this evaluation setup, the F1 score for the oracle model on the KP20K dataset is 0.858 for  $k = 5$  and 0.626 for  $k = 10$ , which apparently poses serious normalization issues as evaluation metrics.
To overcome these problems, we propose novel decoding strategies and evaluation metrics for the keyphrase generation task. The main contributions of this work are as follows:
1. We propose a Seq2Seq based keyphrase generation model capable of generating diverse keyphrases and controlling number of outputs.
2. We propose new metrics based on commonly used  $F_{1}$  score under the hypothesis of variable-size outputs from models, which results in improved empirical characteristics over previous metrics based on a fixed  $k$ .
3. An additional contribution of our study is the introduction of a new dataset for keyphrase generation: STACKEX.
With its marked difference in genre, we expect the dataset to bring added heterogeneity to keyphrase generation evaluation.
2.1 Keyphrase Extraction and Generation
Traditional keyphrase extraction has been studied extensively in past decades. In most existing literature, keyphrase extraction has been formulated as a two-step process. First, lexical features such as part-of-speech tags are used to determine a list
of phrase candidates by heuristic methods (Witten et al., 1999; Liu et al., 2011; Wang et al., 2016; Yang et al., 2017). Second, a ranking algorithm is adopted to rank the candidate list and the top ranked candidates are selected as keyphrases. A wide variety of methods were applied for ranking, such as bagged decision trees (Medelyan et al., 2009; Lopez and Romary, 2010), Multi-Layer Perceptron, Support Vector Machine (Lopez and Romary, 2010) and PageRank (Mihalcea and Tarau, 2004; Le et al., 2016; Wan and Xiao, 2008). Recently, Zhang et al. (2016); Luan et al. (2017); Gollapalli et al. (2017) used sequence labeling models to extract keyphrases from text; Subramanian et al. (2017) used Pointer Networks to point to the start and end positions of keyphrases in a source text; Sun et al. (2019) leveraged graph neural networks to extract keyphrases.
The main drawback of keyphrase extraction is that sometimes keyphrases are absent from the source text, thus an extractive model will fail predicting those keyphrases. Meng et al. (2017) first proposed the CopyRNN, a neural model that both generates words from vocabulary and points to words from the source text. Based on the CopyRNN architecture, Chen et al. (2018a); Zhao and Zhang (2019) leveraged attention to help reducing duplication and improving coverage. Ye and Wang (2018) proposed semi-supervised methods by leveraging both labeled and unlabeled data for training. Chen et al. (2018b); Ye and Wang (2018) proposed to use structure information (e.g., title of source text) to improve keyphrase generation performance. Chan et al. (2019) introduced RL to the keyphrase generation task. Chen et al. (2019a) retrieved similar documents from training data to help producing more accurate keyphrases.
2.2 Sequence to Sequence Generation
Sequence to Sequence (Seq2Seq) learning was first introduced by Sutskever et al. (2014); together with the soft attention mechanism of (Bahdanau et al., 2014), it has been widely used in natural language generation tasks. Gülçehre et al. (2016); Gu et al. (2016) used a mixture of generation and pointing to overcome the problem of large vocabulary size. Paulus et al. (2017); Zhou et al. (2017) applied Seq2Seq models on summary generation tasks, while Du et al. (2017); Yuan et al. (2017) generated questions conditioned on documents and answers from machine comprehension
datasets. Seq2Seq was also applied on neural sentence simplification (Zhang and Lapata, 2017) and paraphrase generation tasks (Xu et al., 2018).
Given a piece of source text, our objective is to generate a variable number of multi-word phrases. To this end, we opt for the sequence-to-sequence (Seq2Seq) (Sutskever et al., 2014) framework as the basis of our model, combined with attention and pointer softmax mechanisms in the decoder.
Since each data example contains one source text sequence and multiple target phrase sequences (dubbed ONE2MANY, and each sequence can be of multi-word), two paradigms can be adopted for training Seq2Seq models. The first one (Meng et al., 2017) is to divide each ONE2MANY data example into multiple ONE2ONE examples, and the resulting models (e.g., CopyRNN) can generate one phrase at once and must rely on beam search technique to produce more unique phrases.
To enable models to generate multiple phrases and control the number to output, we propose the second training paradigm ONE2SEQ, in which we concatenate multiple phrases into a single sequence with a delimiter  $\langle \mathrm{sep}\rangle$ , and this concatenated sequence is then used as the target for sequence generation during training. An overview of the model's structure is shown in Figure 1.
In the following subsections, we use  $w$  to denote input text tokens,  $x$  to denote token embeddings,  $h$  to denote hidden states, and  $y$  to denote output text tokens. Superscripts denote time-steps in a sequence, and subscripts  $e$  and  $d$  indicate whether a variable resides in the encoder or the decoder of the model, respectively. The absence of a superscript indicates multiplicity in the time dimension.  $L$  refers to a linear transformation and  $L^{f}$  refers to it followed by a non-linear activation function  $f$ . Angled brackets,  $\langle \rangle$ , denote concatenation.
3.1 Sequence to Sequence Generation
We develop our model based on the standard Seq2Seq (Sutskever et al., 2014) model with attention mechanism (Bahdanau et al., 2014) and pointer softmax (Gulçehre et al., 2016). Due to
space limit, we describe this basic Seq2Seq model in Appendix A.
3.2 Mechanisms for Diverse Generation
There are usually multiple keyphrases for a given source text because each keyphrase represents certain aspects of the text. Therefore keyphrase diversity is desired for the keyphrase generation. Most previous keyphrase generation models generate multiple phrases by over-generation, which is highly prone to generate similar phrases due to the nature of beam search. Given our objective to generate variable numbers of keyphrases, we need to adopt new strategies for achieving better diversity in the output.
Recall that we represent variable numbers of keyphrases as delimiter-separated sequences. One particular issue we observed during error analysis is that the model tends to produce identical tokens following the delimiter token. For example, suppose a target sequence contains  $n$  delimiter tokens at time-steps  $t_1, \ldots, t_n$ . During training, the model is rewarded for generating the same delimiter token at these time-steps, which presumably introduces much homogeneity in the corresponding decoder states  $h_d^{t_1}, \ldots, h_d^{t_n}$ . When these states are subsequently used as inputs at the time-steps immediately following the delimiter, the decoder naturally produces highly similar distributions over the following tokens, resulting in identical tokens being decoded. To alleviate this problem, we propose two plug-in components for the sequential generation model.
3.2.1 Semantic Coverage
We propose a mechanism called semantic coverage that focuses on the semantic representations of generated phrases. Specifically, we introduce another uni-directional recurrent model GRU<sub>SC</sub> (dubbed target encoder) which encodes decoder-generated tokens  $y^{\tau}$ , where  $\tau \in [0, t)$ , into hidden states  $h_{\mathrm{SC}}^{t}$ . This state is then taken as an extra input to the decoder GRU, modifying equation of the decoder GRU to:
$$
h _ {d} ^ {t} = \operatorname {G R U} _ {d} \left(\left\langle x _ {d} ^ {t}, h _ {\mathrm {S C}} ^ {t} \right\rangle , h _ {d} ^ {t - 1}\right). \tag {1}
$$
If the target encoder were to be updated with the training signal from generation (i.e., backpropagating error from the decoder GRU to the target encoder), the resulting decoder is essentially a 2-layer GRU with residual connections. Instead, inspired
1We release the code, datasets and model outputs for reproducing our results in https://github.com/memray/OpenNMT-kpg-release.
7963
Figure 1: The architecture of the proposed model for improving keyphrase diversity.  $A$  represents last states of a bi-directional source encoder;  $B$  represents the last state of target encoder;  $C$  indicates decoder states where target tokens are either delimiters or end-of-sentence tokens. During orthogonal regularization, all  $C$  states are used; during target encoder training, we maximize mutual information between states  $A$  with  $B$ . Red dash arrow indicates a detached path, i.e., no back-propagation through such path.
by previous representation learning works (Logeswaran and Lee, 2018; van den Oord et al., 2018; Hjelm et al., 2018), we train the target encoder in an self-supervised fashion (Figure 1). Specifically, due to the autoregressive nature of the RNN-based decoder, we follow Contrastive Predictive Coding (CPC) (van den Oord et al., 2018), where a Noise-Contrastive Estimation(NCE) loss is used to maximize a lower bound on mutual information. That is, we extract target encoder's final hidden state vector  $h_{\mathrm{SC}}^{M}$ , where  $M$  is the length of target sequence, and use it as a general representation of the target phrases. We train by maximizing the mutual information between these phrase representations and the final state of the source encoder  $h_e^T$  as follows. For each phrase representation vector  $h_{\mathrm{SC}}^{M}$ , we take the encodings  $H_e^T = \{h_{e,1}^T,\dots,h_{e,N}^T\}$  of  $N$  different source texts, where  $h_{e, true}^T$  is the encoder representation for the current source text, and the remaining  $N - 1$  are negative samples (sampled at random) from the training data. The target encoder is trained to minimize the classification loss:
$$
\mathcal {L} _ {\mathrm {S C}} = - \log \frac {g \left(h _ {e , t r u e} ^ {T} , h _ {\mathrm {S C}} ^ {M}\right)}{\sum_ {i \in [ 1 , N ]} g \left(h _ {e , i} ^ {T} , h _ {\mathrm {S C}} ^ {M}\right)}, \tag {2}
$$
$$
g (h _ {a}, h _ {b}) = \exp (h _ {a} ^ {\top} B h _ {b})
$$
where  $B$  is bi-linear transformation.
The motivation here is to constrain the overall representation of generated keyphrase to be semantically close to the overall meaning of the source text. With such representations as input to the decoder, the semantic coverage mechanism can potentially help to provide useful keyphrase information and guide generation.
3.2.2 Orthogonal Regularization
We also propose orthogonal regularization, which explicitly encourages the delimiter-generating decoder states to be different from each other. This is inspired by Bousmalis et al. (2016), who use orthogonal regularization to encourage representations across domains to be as distinct as possible. Specifically, we stack the decoder hidden states corresponding to delimiters together to form matrix  $H = \langle h_d^{t_1},\dots ,h_d^{t_n}\rangle$  and use the following equation as the orthogonal regularization loss:
$$
\mathcal {L} _ {\mathrm {O R}} = \left\| H ^ {\top} H \odot \left(\mathbf {1} - I _ {n}\right) \right\| _ {2}, \tag {3}
$$
where  $H^{\top}$  is the matrix transpose of  $H$ ,  $I_{n}$  is the identity matrix of rank  $n$ ,  $\odot$  indicates element wise multiplication,  $\| M\| _2$  indicates  $L^2$  norm of each element in a matrix  $M$ . This loss function prefers orthogonality among the hidden states  $h_d^{t_1},\ldots ,h_d^{t_n}$  and thus improves diversity in the tokens following the delimiters.
3.2.3 Training Loss
We adopt the widely used negative log-likelihood loss in our sequence generation model, denoted as  $\mathcal{L}_{\mathrm{NLL}}$ . The overall loss we use for optimization is:
$$
\mathcal {L} = \mathcal {L} _ {\mathrm {N L L}} + \lambda_ {\mathrm {O R}} \cdot \mathcal {L} _ {\mathrm {O R}} + \lambda_ {\mathrm {S C}} \cdot \mathcal {L} _ {\mathrm {S C}}, \tag {4}
$$
where  $\lambda_{\mathrm{OR}}$  and  $\lambda_{\mathrm{SC}}$  are hyper-parameters.
3.3 Decoding Strategies
According to different task requirements, various decoding methods can be applied to generate the target sequence  $y$ . Prior studies Meng et al. (2017); Yang et al. (2017) focus more on generating excessive number of phrases by leveraging beam
7964
search to proliferate the output phrases. In contrast, models trained under ONE2SEQ paradigm are capable of determining the proper number of phrases to output. In light of previous research in psychology (Van Zandt and Townsend, 1993; Forster and Bednall, 1976), we name these two decoding/search strategies as Exhaustive Decoding and Self-terminating Decoding, respectively, due to their resemblance to the way humans behave in serial memory tasks. Simply speaking, the major difference lies in whether a model is capable of controlling the number of phrases to output. We describe the detailed decoding strategies used in this study as follows:
3.3.1 Exhaustive Decoding
As traditional keyphrase tasks evaluate models with a fixed number of top-ranked predictions (say F-score @5 and @10), existing keyphrase generation studies have to over-generate phrases by means of beam search (commonly with a large beam size, e.g., 150 and 200 in (Chen et al., 2018b; Meng et al., 2017), respectively), a heuristic search algorithm that returns  $K$  approximate optimal sequences. For the ONE2ONE setting, each returned sequence is a unique phrase itself. But for ONE2SEQ, each produced sequence contains several phrases and additional processes (Ye and Wang, 2018) are needed to obtain the final unique (ordered) phrase list.
It is worth noting that the time complexity of beam search is  $O(Bm)$ , where  $B$  is the beam width, and  $m$  is the maximum length of generated sequences. Therefore the exhaustive decoding is generally very computationally expensive, especially for ONE2SEQ setting where  $m$  is much larger than in ONE2ONE. It is also wasteful as we observe that less than  $5\%$  of phrases generated by ONE2SEQ models are unique.
3.3.2 Self-terminating Decoding
An innate characteristic of keyphrase tasks is that the number of keyphrases varies depending on the document and dataset genre, therefore dynamically outputting a variable number of phrases is a desirable property for keyphrase generation models. Since our model is trained to generate a variable number of phrases as a single sequence joined by delimiters, we can obtain multiple phrases by simply decoding a single sequence for each given
source text. The resulting model thus implicitly performs the additional task of dynamically estimating the proper size of the target phrase set: once the model believes that an adequate number of phrases have been generated, it outputs a special token  $\langle /s\rangle$  to terminate the decoding process.
One notable attribute of the self-terminating decoding strategy is that, by generating a set of phrases in a single sequence, the model conditions its current generation on all previously generated phrases. Compared to the exhaustive strategy (i.e., phrases being generated independently by beam search in parallel), our model can model the dependency among its output in a more explicit fashion. Additionally, since multiple phrases are decoded as a single sequence, decoding can be performed more efficiently than exhaustive decoding by conducting greedy search or beam search on only the top-scored sequence.
4 Evaluating Keyphrase Generation
Formally, given a source text, suppose that a model predicts a list of unique keyphrases  $\hat{\mathcal{V}} = (\hat{y}_1,\dots ,\hat{y}_m)$  ordered by the quality of the predictions  $\hat{y}_i$  and that the ground truth keyphrases for the given source text is the oracle set  $\mathcal{V}$ . When only the top  $k$  predictions  $\hat{\mathcal{V}}_{:k} = (\hat{y}_1,\dots ,\hat{y}_{\min (k,m)})$  are used for evaluation, precision, recall, and  $F_{1}$  score are consequently conditioned on  $k$  and defined as:
$$
P @ k = \frac {| \hat {\mathcal {Y}} _ {: k} \cap \mathcal {Y} |}{| \hat {\mathcal {Y}} _ {: k} |}, \qquad R @ k = \frac {| \hat {\mathcal {Y}} _ {: k} \cap \mathcal {Y} |}{| \mathcal {Y} |},
$$
$$
\mathbf {F} _ {1} @ k = \frac {2 * P @ k * R @ k}{P @ k + R @ k}. \tag {5}
$$
As discussed in Section 1, the number of generated keyphrases used for evaluation can have a critical impact on the quality of the resulting evaluation metrics. Here we compare three choices of  $k$  and the implications on keyphrase evaluation for each choice:
-  $\mathbf{F}_1@k$ : where  $k$  is a pre-defined constant (usually 5 or 10). Due to the high variance of the number of ground truth keyphrases, it is often that  $|\hat{\mathcal{Y}}_{:k}| \leq k < |\mathcal{Y}|$ , and thus  $R@k$  — and in turn  $F_1@k$  — of an oracle model can be smaller than 1. This undesirable property is unfortunately prevalent in the evaluation metrics adopted by all existing keyphrase generation studies to our knowledge.
A simple remedy is to set  $k$  as a variable number which is specific to each data example. Here we define two new metrics:
2Note this is fundamentally different from other NLG tasks. In specific, the number of keyphrases is variable, the length of each keyphrase is also variable.
7965
Kp20K	Inspec	Krapivin	NUS	SemEval
Model	@5	@10	@O	@5	@10	@O	@5	@10	@O	@5	@10	@O	@5	@10	@O
Abstractive Neural
CopyRNN (Meng et al.)	32.8	25.5	-	29.2	33.6	-	30.2	25.2	-	34.2	31.7	-	29.1	29.6	-
CopyRNN*	31.7	27.3	33.5	24.4	28.9	29.0	30.5	26.6	32.5	37.6	35.2	40.6	31.8	31.8	31.7
CorrRNN (Chen et al.)	-	-	-	-	-	-	31.8	27.8	-	35.8	33.0	-	32.0	32.0	-
ParaNetT +CoAtt (Zhao and Zhang)	36.0	28.9	-	29.6	35.7	-	32.9	28.2	-	36.0	35.0	-	31.1	31.2	-
catSeqTG-2RFI† (Chan et al.)	32.1	-	35.7	25.3	-	28.0	30.0	-	34.8	37.5	-	25.5	28.7	-	29.8
KG-KE-KR-M† (Chen et al.)	31.7	28.2	38.8	25.7	28.4	31.4	27.2	25.0	31.7	28.9	28.6	38.4	20.2	22.3	30.3
CatSeq (Ours)	31.4	27.3	31.9	29.0	30.0	30.7	30.7	27.4	32.4	35.9	34.9	38.3	30.2	30.6	31.0
CatSeqD (Ours)	34.8	29.8	35.7	27.6	33.3	33.1	32.5	28.5	37.1	37.4	36.6	40.6	32.7	35.2	35.7
Extractive IR
TfIdf (Hasan and Ng)	7.2	9.4	6.3	16.0	24.4	20.8	6.7	9.3	6.8	11.2	14.0	12.2	8.8	14.7	11.3
TextRank (Mihalcea and Tarau)	18.1	15.1	18.4	28.6	33.9	33.5	18.5	16.0	21.1	23.0	21.6	23.8	21.7	22.6	22.9
KEA (Witten et al.)	4.6	4.4	5.1	2.2	2.2	2.2	1.8	1.7	1.7	7.3	7.1	8.1	6.8	6.5	6.6
Maui (Medelyan et al.)	0.5	0.5	0.4	3.5	4.6	3.9	0.5	0.7	0.6	0.4	0.6	0.6	1.1	1.4	1.1
Extractive Neural
DivGraphPointer (Sun et al.)	36.8	29.2	-	38.6	41.7	-	46.0	40.2	-	40.1	38.9	-	36.3	29.7	-
w/ Additional Data
Semi-Multi (Ye and Wang)	32.8	26.4	-	32.8	31.8	-	32.3	25.4	-	36.5	32.6	-	31.9	31.2	-
TG-Net (Chen et al.)	37.2	31.5	-	31.5	38.1	-	34.9	29.5	-	40.6	37.0	-	31.8	32.2	-
Table 2: Performance  $(\mathrm{F_1}$  -score) of present keyphrase prediction on scientific publications datasets. Best/secondbest performing score in each column is highlighted with bold/underline. We also list results from literature where models that are not directly comparable (i.e., models leverage additional data and pure extractive models). Note model names with  $\dagger$  represent its  $\mathbf{F}_1@O$  is computed by us using existing works' released keyphrase predictions.3