Venue: NAACL 2021

# An Empirical Study on Neural Keyphrase Generation

Rui Meng\({}^{\dagger}\) Xingdi Yuan\({}^{\dagger}\) Tong Wang\({}^{\ddagger}\) Sanqiang Zhao\({}^{\dagger}\) Adam Trischler\({}^{\ddagger}\) Daqing He\({}^{\dagger}\)

\({}^{\dagger}\)School of Computing and Information, University of Pittsburgh

\({}^{\ddagger}\)Microsoft Research, Montreal

rui.meng@pitt.edu

###### Abstract

Recent years have seen a flourishing of neural keyphrase generation (KPG) works, including the release of several large-scale datasets and a host of new models to tackle them. Model performance on KPG tasks has increased significantly with evolving deep learning research. However, there lacks a comprehensive comparison among different model designs, and a thorough investigation on related factors that may affect a KPG system's generalization performance. In this empirical study, we aim to fill this gap by providing extensive experimental results and analyzing the most crucial factors impacting the generalizability of KPG models. We hope this study can help clarify some of the uncertainties surrounding the KPG task and facilitate future research on this topic.

+
Footnote †: _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4985–5007 June 6–11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

Keyphrases are phrases that summarize and highlight important information in a piece of text. Keyphrase generation (KPG) is the task of automatically predicting such keyphrases given the source text. The task can be (and has often been) easily misunderstood and trivialized as yet another natural language generation task like summarization and translation, failing to recognize one key aspect that distinguishes kPG: the multiplicity of generation targets; for each input sequence, a KPG system is expected to output _multiple_ keyphrases, each a mini-sequence of multiple word tokens.

Despite this unique nature, kPG has been essentially "brute-forced" into the sequence-to-sequence (Seq2Seq) Sutskever et al. (2014) framework in the existing literature Meng et al. (2017); Chen et al. (2018); Ye and Wang (2018); Chen et al. (2019b); Yuan et al. (2020); Chan et al. (2019); Zhao and Zhang (2019); Chen et al. (2019a).The community has approached the unique challenges with much ingenuity in problem formulation, model design, and evaluation. For example, multiple target phrases have been reformulated by either splitting into one phrase per data point or joining into a single sequence with delimiters (Figure 1), both allowing straightforward applications of existing neural techniques such as Seq2Seq. In accordance with the tremendous success and demonstrated effectiveness of neural approaches, steady progress has been made in the past few years -- at least empirically -- across various domains, including sub-areas where it was previously shown to be rather difficult (e.g., in generating keyphrases that are not present in the source text).

Meanwhile, with the myriad of kPG's unique challenges comes an ever-growing collection of studies that, albeit novel and practical, may quickly proliferate and overwhelm. We are therefore motivated to present this study as -- to the best of our knowledge -- the first systematic investigation on such challenges as well as the effect of interplay among their solutions. We hope this study can serve as a practical guide to help researchers to gain a more holistic view on the task, and to profit from the empirical results of our investigations on a variety of topics in kPG including model design, evaluation, and hyper-parameter selection.

The rest of the paper is organized as follows. We first enumerate specific challenges in kPG due to the multiplicity of its target, and describe general setups for the experiments. We subsequently present experimental results and discussions to answer three main questions:

1. How well do kPG models generalize to various testing distributions?

2. Does the order of target keyphrases matter while training One2Seq?

3. Are larger training data helpful? How to better make use of them?

## 2 Unique Challenges in kpg

Due to the multiplicity of the generation targets, kPG is unique compared to other NLG tasks such

as summarization and translation. In this section, we start from providing background knowledge of the KPG problem setup. Then we enumerate the unique aspects in KPG model designing and training that we focus on in this work.

Problem DefinitionFormally, the task of keyphrase generation (KPG) is to generate a _set_ of keyphrases \(\{p_{1},\dots,p_{n}\}\) given a source text \(t\) (a sequence of words). Semantically, these phrases summarize and highlight important information contained in \(t\), while syntactically, each keyphrase may consist of multiple words. A keyphrase is defined as _present_ if it is a sub-string of the source text, or as _absent_ otherwise.

Training ParadigmsTo tackle the unique challenge of generating multiple targets, existing neural KPG approaches can be categorized under one of two training paradigms: One2One Meng et al. (2017) or One2Seq Yuan et al. (2020), both based on the Seq2Seq framework. Their main difference lies in how target keyphrase multiplicity is handled in constructing data points (Figure 1).

Specifically, with multiple target phrases \(\{p_{1},\dots,p_{n}\}\), One2One takes one phrase at a time and pairs it with the source text \(t\) to form \(n\) data points \((t,p_{i})_{i=1:n}\). During training, a model learns a one-to-many mapping from \(t\) to \(p_{i}\)'s, i.e., the same source string usually has multiple corresponding target strings. In contrast, One2Seq concatenates all ground-truth keyphrases \(p_{i}\) into a single string: \(P=\texttt{<bos>}p_{1}\texttt{<sep>}\cdots\texttt{<sep>}p_{n}\texttt{<cos>}\) (i.e., prefixed with \(\texttt{<obs>}\), joint with \(\texttt{<sep>}\), and suffixed with \(\texttt{<eqs>}\)), thus forming a single data point \((t,P)\). A system is then trained to predict the concatenated sequence \(P\) given \(t\). By default, we construct \(P\) follow the ordering strategy proposed in Yuan et al. (2020). Specifically, we sort present phrases by their first occurrences in source text, and append absent keyphrases at the end. This ordering is denoted as Pres-Abs in SS4.

ArchitectureIn this paper, we adopt the architecture used in both Meng et al. (2017) and Yuan et al. (2020), using RNN to denote it. RNN is a GRU-based Seq2Seq model Cho et al. (2014) with a copy mechanism Gu et al. (2016) and a coverage mechanism See et al. (2017). We also consider a more recent architecture, Transformer Vaswani et al. (2017), which is widely used in encoder-decoder language generation literature Gehrmann et al. (2018). We replace both the encoder GRU and decoder GRU in RNN by Transformer blocks, and denote this architecture variant as TRANS. Both the RNN and TRANS models can be trained with either the One2One or One2Seq paradigm.

In recent years, a host of auxiliary designs and mechanisms have been proposed and developed based on either One2One or One2Seq (see SS6). In this study, however, we focus only on the "vanilla" version of them and we show that given a set of carefully chosen architectures and training strategies, base models can achieve comparable, if not better performance than state-of-the-art methods. We assume that KPG systems derived from either One2One or One2Seq model would be affected by these factors of model designing in similar ways.

Decoding StrategiesKPG is distinct from other NLG tasks since it expects a _set_ of multi-word phrases (rather than a single sequence) as model predictions. Depending on the preference of po

Figure 1: **Top: comparison between One2One (left) and One2Seq (right) paradigms on the same data point. Bottom: demonstration of the decoding process for One2One (left) and One2Seq (mid/right) models. One2Seq can apply both beam search (mid) and greedy decoding (right).**



tential downstream tasks, a KPG system can utilize different decoding strategies. For applications that favor _high recall_ (e.g., generating indexing terms for retrieval systems), a common practice is to utilize beam search and take predictions from _all_ beams1. This is applicable in both One2One- and One2Seq-based models to proliferate the number of predicted phrases at inference time. In this work, we use a beam width of 200 and 50 for One2One and One2Seq, respectively. On the contrary, some other applications favor _high precision_ and small number of predictions (e.g., KG construction), a One2Seq-based model is capable of decoding greedily, thanks to its nature of generating multiple keyphrases in a sequential manner.

Footnote 1: This is in contrast to only taking the _single_ top beam as in typical NLG tasks.

As an example, we illustrate the two decoding strategies in Figure 1. Specifically, a One2One model typically collects output keyphrases from all beams and use the top \(k\) phrases as the model output (\(k=5\) in the example). In One2Seq, either beam search or greedy decoding can be applied. For beam search, we use both the order of phrases within a beam and the rankings of beams to rank the outputs. In the shown example, top 5 beam search outputs are obtained from the 2 beams with highest rankings. As for greedy decoding, the decoder uses a beam size of 1, and takes all phrases from the single beam as outputs. In this way, the One2Seq model can determine the number of phrases to output by itself conditioned on \(t\).

EvaluationDue to the multiplicity of targets in KPG task, the evaluation protocols are distinct from typical NLG tasks. A spectrum of evaluation metrics have been used to evaluate KPG systems, including metrics that truncate model outputs at a fixed number such as **F\({}_{1}\)@5** and **F\({}_{1}\)@10**Meng et al. (2017); metrics that evaluate a model's ability of generating variant number of phrases such as **F\({}_{1}\)@\(\mathcal{O}\)** and **F\({}_{1}\)@\(\mathcal{M}\)**van et al. (2020); metrics that evaluate absent keyphrases such as Recall@50 (**R@50**). Detailed definitions of the metrics are provided in Appendix A. Due to space limit, we mainly discuss **F\({}_{1}\)@\(\mathcal{O}\)**, **F\({}_{1}\)@10** and **R@50** in the main content, complete results with all common metrics are included in Appendix E. We save model checkpoints for every 5,000 training steps and report test performance using checkpoints that produce the best **F\({}_{1}\)@\(\mathcal{O}\)**or R@50** on the KP20v validation set.

DatasetsA collection of datasets in the domain of scientific publication (KP20K, INSPEC, KRAPIVIN, NUS, and SEMEVAL) and news articles (DUC) have been widely used to evaluate KPG task. Following previous work, we train models using the training set of KP20K since its size is sufficient to support the training of deep neural networks. Evaluation is performed on KP20v's test set as well as all other datasets without fine-tuning. Details of the datasets are shown in Appendix B.

## 3 Generalizability

In this section, we show and analyze the generalization performance of KPG systems from 2 dimensions: model architecture and training paradigm. Specifically, we compare the two model architectures (i.e., RNN and TRANS) as described in SS2. For each model architecture, we train the KPG model using either of the training paradigms (i.e., One2One or One2Seq) also as described in SS2.

To better understand model variants' generalization properties, we categorize the 6 testing sets into 3 classes according to their distribution similarity with the training data (KP20K), as shown in Table 1. Concretely, KP20x and KRAPIVIN are in-distribution test sets (denoted as D\({}_{0}\)), since they both contain scientific paper abstracts paired with keyphrases provided by their authors. INspec, NUS and SEMEVAL are out-of-distribution test sets

\begin{table}
\begin{tabular}{c c|l l l l l l l l l l l l} \hline \hline  & \multicolumn{4}{c|}{Present (\textbf{F${}_{1}\text@$\varnothing$})} & \multicolumn{4}{c|}{Present (\textbf{F${}_{1}\text@$\varnothing$})} & \multicolumn{4}{c|}{Present (\textbf{F${}_{1}\text@$\varnothing$})} \\  & \multicolumn{4}{c|}{One2One} & \multicolumn{4}{c|}{One2Seq} & \multicolumn{4}{c|}{One2One} & \multicolumn{4}{c|}{One2Seq} \\  & \multicolumn{4}{c|}{One2Seq} \\  & \multicolumn{4}{c|}{RNN} & \multicolumn{4}{c|}{TRANS} & \multicolumn{4}{c|}{RUN} & \multicolumn{4}{c|}{TRANS} & \multicolumn{4}{c|}{RUN} & \multicolumn{4}{c|}{TRANS} & \multicolumn{4}{c|}{RUN} & \multicolumn{4}{c|}{TRANS} & \multicolumn{4}{c|}{RUN} & \multicolumn{4}{c|}{TRANS} & \multicolumn{4}{c|}{RUN} & \multicolumn{4}{c|}{TRANS} \\ \cline{3-18}  & \multirow{2}{*}{RF20K} & \multirow{2}{*}{KAPIVIN} & 35.3 & 37.4 & 31.2 & 36.2 & 27.9 & 28.9 & 26.1 & 29.0 & 13.1 & 22.1 & 3.2 & 15.0 \\  & & 35.5 & 33.0 & 33.5 & 36.4 & 27.0 & 26.4 & 26.9 & 28.1 & 13.7 & 23.8 & 3.3 & 16.6 \\  & & D\({}_{0}\) Average & 35.4 & 35.2 & 32.3 & **36.3** & 27.4 & 27.7 & 26.5 & **28.5** & 13.4 & **23.0** & 3.2 & 15.8 \\ \hline \multirow{6}{*}{D\({}_{1}\)} & \multirow{2}{*}{NBSPC} & \multirow{2}{*}{NBSPC} & \multirow{2}{*}{33.7} & 33.6 & 38.8 & 36.9 & 32.5 & 30.8 & 38.7 & 36.6 & 8.2 & 9.2 & 3.7 & 6.7 \\  & & 43.4 & 41.1 & 39.2 & 42.3 & 35.9 & 36.1 & 36.6 & 37.3 & 11.2 & 18.9 & 2.9 & 12.5 \\  & & 35.2 & 35.1 & 36.2 & 34.8 & 34.6 & 33.0 & 35.0 & 34.2 & 6.1 & 18.9 & 1.7 & 12.5 \\  & & 37.4 & 36.3 & **38.1** & 38.0 & 34.4 & 33.3 & **36.7** & 36.0 & 8.5 & **12.7** & 2.8 & 9.2 \\ \hline D\({}_{2}\) & \multirow{2}{*}{DOC} & 13.4 & 7.8 & **15.0** & 11.0 & 13.7 & 8.4 & **16.0** & 11.4 & 0.0 & **0.2** & 0.0 & 0.0 \\ \cline{1-1}  & All & Average & 32.8 & 31.2 & 32.3 & **32.9** & 28.6 & 27.3 & **29.9** & 29.4 & 8.7 & **14.0** & 2.5 & 9.8 \\ \hline \end{tabular}
\end{table}
Table 1: Testing scores across different model architectures, training paradigms, and datasets. In which, D\({}_{0}\): individution; D\({}_{1}\): out-of-distribution, and D\({}_{2}\): out-of-domain. We provide the average score over each category.

 

(denoted as D\({}_{1}\)), they share same type of source text with D\({}_{0}\), but with additionally labeled keywords by third-party annotators. DUC is a special test set which uses news articles as its source text. Because it shares the least domain knowledge and vocabulary with all the other test sets, we call it out-of-domain test set (denoted as D\({}_{2}\)).

Model Architecture: RNN vs TRANSThe first thing to notice is that on present KPG, the models show consistent trends between **F\({}_{1}\)@10** and **F\({}_{1}\)@\(\mathcal{O}\)**. We observe that TRANS models significantly outperform RNN models when trained with the One2Seq paradigm on D\({}_{0}\) test sets. However, when test data distribution shift increases, on D\({}_{1}\) test sets, RNN models starts to outperform TRANS; eventually, when dealing with D\({}_{2}\) test set, RNN outperforms TRANS by a large margin. On models trained with One2One paradigm, we observe a similar trend. On D\({}_{0}\) data, TRANS models achieve comparable **F\({}_{1}\)@10** and **F\({}_{1}\)@\(\mathcal{O}\)** scores with RNN, when data distribution shift increases, RNN models produce better results.

On the contrary, for absent KPG, TRANS outperforms RNN by a significant margin in all experiment settings. This is especially obvious when models are trained with One2Seq paradigm, where RNN models barely generalize to any of the testing data and produce an average **R@50** of 2.5. In the same setting, TRANS models get an average **R@50** of 9.8, which is 4\(\times\) higher than RNN.

To further study the different generation behaviors between RNN and TRANS, we investigate the average number of unique predictions generated by either of the models. As shown in Figure 12 in Appendix D, comparing results of order Pres-Ahs in sub-figure a/b (RNN) with sub-figure c/d (TRANS), we observe that TRANS is consistently generating more unique predictions than RNN, in both cases of greedy decoding (4.5 vs 4.2) and beam search (123.3 vs 96.8). We suspect that generating a more diverse set of keyphrases may have a stronger effect on in-distribution test data. The generated outputs during inference are likely to represent the distribution learned from the training data, when the test data share the same (or similar) distribution, a larger set of unique predictions leads to a higher recall -- which further contributes to their F-scores. In contrast, on test sets which data distribution is far from training distribution, the extra predictions may not be as useful, and even hurts precision. Similarly, because we evaluate absent KPG by the models' recall, TRANS models -- produce more unique predictions -- can always outperform RNN models.2

Footnote 2: Our TRANS and RNN models follow Vaswani et al. (2017) and Meng et al. (2017)’s hyper-parameter settings respectively. RNN is significantly lighter than TRAIN. We conduct experiments with a much larger RNN but only observe marginal performance boost against Meng et al. (2017)’s setting.

Training Paradigm: One2One vs One2SeqWe observe that on present KPG tasks, models trained with the One2Seq paradigm outperforms One2One in most settings, this is particularly clear on D\({}_{1}\) and D\({}_{2}\) test sets. We believe this is potentially due to the unique design of the One2Seq training paradigm where at every generation step, the model conditions its decision making on all previously generated tokens (phrases). Compared to the One2One paradigm where multiple phrases can only be generated independently by beam search in parallel, the One2Seq paradigm can model the dependencies among tokens and the dependencies among phrases more explicitly.

However, on absent KPG, One2One consistently outperforms One2Seq. Furthermore, only when trained with One2One paradigm, an RNN-based model can achieve **R@50** scores close to TRANS-based models. This may because a One2Seq model tends to produce more duplicated predictions during beam search inference. By design, every beam is a string that contains multiple phrases that concatenated by the delimiter <sep>, there is no guarantee that the phrase will not appear in multiple beams. In the example shown in Figure 1, "topic tracking" is such a duplicate prediction that appears in multiple beams. In fact, the proportion of duplicates in One2Seq predictions

Figure 2: Present KPG testing scores (**F\({}_{1}\)@\(\mathcal{M}\)**). Colors represent the relative performance, normalized per row.

 

is more than 90%. This is in contrast with beam search on One2One models, where each beam only contains a single keyphrase thus has a much lower probability of generating duplication.3

Footnote 3: Due to post-processing such as stemming, One2One model may still produce duplication.

## 4 Does Order Matter in One2Seq?

In the One2One paradigm (as shown in Figure 1), each data example is split to multiple equally weighted data pairs, thus it generates phrases without any prior on the order. In contrast, One2Seq training has the unique capability of generating a varying number of keyphrases in a single sequence. This inductive bias enables a model to learn dependencies among keyphrases, and also to implicitly estimate the number of target phrases conditioned on the source text. However, the One2Seq approach introduces a new complication. During training, the Seq2Seq decoder takes the concatenation of multiple target keyphrases as target. As pointed out by Vinyals et al. (2016), order matters in sequence modeling tasks; yet the ordering among the target keyphrases has not been fully investigated and its effect to the models' performance remains unclear. Several studies have noted this problem Ye and Wang (2018); Yuan et al. (2020) without further exploration.

Ordering DefinitionTo explore along this direction, we first define nine ordering strategies for concatenating target phrases.

* Random:Randomly shuffle the target phrases. Because of the _set generation_ nature of KPG, we expect randomly shuffled target sequences help to learn an order-invariant decoder.
* OriI: Keep phrases in their original order in the data (e.g., provided by the authors of source texts). This was used by Ye and Wang (2018).
* OriI-Rev: Reversed order of Ori.
* S=>L: Phrases sorted by lengths (number of tokens, from short to long).
* L=>S: Reversed order of S~>L.
* Alpha: Sort phrases by alphabetical order.
* Alpha-Rev: Reversed order of Alpha.
* Pres-Abs: Sort present phrases by their first occurrences in source text. Absent phrases are shuffled and appended to the end of the present phrase sequence. This was used by Yuan et al. (2020).
* Abs-Pres: Similar to Pres-Abs, but prepending absent phrases to the beginning.

Greedy DecodingIn Figure 2, we show the RNN and TRANS model's \({\bf F}_{1}@\mathcal{M}\) on present FPG task, equipped with greedy decoding. In this setting, the model simply chooses the token with the highest probability at every step, and terminates either upon generating the <eos> token or reaching the maximum target length limit (40). This means the model predicts phrases solely relying on its innate distribution learned from the training data, and thus this performance could somewhat reflect to which

Figure 3: Present KPG testing scores **(top: \({\bf F}_{1}@\texttt{10}\), bottom: \({\bf F}_{1}@\mathcal{O}\))**. Colors represent the relative performance, normalized per row.

 

degree the model fits the training distribution and understands the task.

Through this set of experiments, we first observe that each model demonstrates consistent performance across all six test datasets, indicating that ordering strategies play critical roles in training One2Seq models when greedy decoding is applied. When using the RNN architecture, Random consistently yields lower \({\bf F}_{1}@\mathcal{M}\) than other ordering strategies on all datasets. This suggests that a consistent order of the keyphrases is beneficial. However, TRANS models show a better resistance against randomly shuffled keyphrases and produce average tier performance with the Random ordering. Meanwhile, we observe that Pres-Abs outperforms other ordering strategies by significant margins. A possible explanation is that with this order (of occurrences in the source text), the current target phrase is always to the right of the previous one, which can serve as an effective prior for the attention mechanism throughout the One2Seq decoding process. We observe similar trends in greedy decoding models' \({\bf F}_{1}@\mathcal{O}\) and \({\bf F}_{1}@\boldsymbol{10}\), due to space limit, we refer readers to Figure 9, 10 in Appendix D.

Beam SearchNext, we show results obtained from the same set of models equipped with beam search (beam width is 50) in Figure 3 (a/b). Compared with greedy decoding (Figure 10, Appendix D), we can clearly observe the overall \({\bf F}_{1}@\boldsymbol{10}\) scores have positive correlation with the beam width (greedy decoding is a special case where beam width equals to 1). We observe that compared to the greedy decoding case, the pattern among different ordering strategies appears to be less clear, with the scores distributed more evenly across different settings (concretely, the absolute difference between max average score and min average score is lower).

We suspect that the uniformity among different ordering strategies with beam search may be due to the limitation of the evaluation metric \({\bf F}_{1}@\boldsymbol{10}\). The metric \({\bf F}_{1}@\boldsymbol{10}\) truncates a model's predictions to 10 top-ranked keyphrases. By investigation, we find that during greedy decoding, the number of predictions acts as a dominant factor, this number varies greatly among different ordering. With greedy decoding, Pres-Abs can generally predict more phrases than the others, which explains its performance advantage (Figure 13 (a/c), Appendix D). However, as the beam width increases, all models can predict more than 10 phrases (Figure 13 (b/d), Appendix D). In this case, the \({\bf F}_{1}@\boldsymbol{10}\) is contributed more by a model' ability of generating more high quality keyphrases within its top-10 outputs, rather than the amount of predictions. Therefore, the performance gap among ordering strategies is gradually narrowed in beam search. For instance, we observe that the \({\bf F}_{1}@\boldsymbol{10}\) difference between Pres-Abs and S->L produced by RNN is 3.5/2.0/1.0/0.2 when beam width is 1/10/25/50.

To validate our assumption, we further investigate the same set of models' performance on \({\bf F}_{1}@\mathcal{O}\), which strictly truncates the generated keyphrase list by the number of ground-truth keyphrases \(\mathcal{O}\) (where in most cases \(\mathcal{O}<10\)). Under this harsher criterion, a model is required to generate more high quality keyphrases within its top-\(\mathcal{O}\) outputs. From Figure 3 (c/d), we observe that the scores are less uniformly distributed, this indicates a larger difference between different order settings. Among all orders, Ori produces best average \({\bf F}_{1}@\mathcal{O}\) with RNN, whereas Alpha-Rev and Ori-Rev produce best average \({\bf F}_{1}@\mathcal{O}\) with TRANS.

In our curated list of order settings, there are 3 pairs of orderings with reversed relationship (i.e., S->L vs L->S, Alpha vsAlpha-Rev, Ori vs Ori-Rev). Interestingly, we observe that when beam search is applied, these orderings often show a non-negligible score difference with their counterparts. This also suggests that order matters since specific model architecture and training paradigm often has its own preference on the phrase ordering.

It is also worth mentioning that when we manually check the output sequences in test set produced by Alpha ordering, we notice that the model is actually able to retain alphabetical order among the predicted keyphrases, hinting that a Seq2Seq model might be capable of learning simple morphological dependencies even without access to any character-level representations.

Ordering in Absent KPGWe report the performance of the same set of models on the absent portion of data in Figure 11, Appendix D. Although achieving relatively low **R@50** in most settings, scores produced by various orderings show clear distinctions, normalized heat maps suggest that the rankings among different orderings tend to be consistent across all testing datasets. In general, Press-Abs produces better absent keyphrases across different model architectures. Due to the 

space limit, we encourage readers to check out Appendix D, which provides an exhaustive set of heat maps including all experiment settings and metrics discussed in this section.

## 5 Training with More Data

In this section, we further explore the possibility of improving KPG performance by scaling up the training data. Data size has been shown as one of the most effective factors for training language models Raffel et al. (2019); Ott et al. (2018) but it has yet been discussed in the context of KPG.

MagKP DatasetWe construct a new dataset, namely MAGKP, on the basis of Microsoft Academic Graph Sinha et al. (2015). We filter the original MAG v1 dataset (166 million papers, multiple domains) and only keep papers in _Computer Science_ and with at least one keyphrase. This results in 2.7 million data points (\(5\times\) larger than KP20k). This dataset remains noisy despite the stringent filtering criteria, this is because 1) the data is crawled from the web and 2) some keywords are labeled by automatic systems rather than humans. This noisy nature brings many interesting observations.

General ObservationsThe first thing we try is to train a KPG model with both KP20k and MAGKP. During training, the two dataset are fed to the model in an alternate manner, we denote this data mixing strategy as ALT. In Figure 4, we compare models' performance when trained on both KP20k and MAGKP against solely on KP20k. We observe the extra MAGKP data brings consistent improvement across most model architecture and training paradigm variants. This suggests that model KPG models discussed in this work can benefit from additional training data. Among all the settings, **F\({}_{1}\)@\(\mathcal{O}\)** of the TRANS+One2Seq is boosted by 3 points on present KPG, the resulting score outperforms other variants by a significant margin and even surpass a host of state-of-the-art models (see comparison in Appendix E). Again, the same setting obtains a 2.3 boost of **R@50** score on the absent KPG task, makes TRANS+One2Seq the setting that benefits the most from extra data. In contrast, the extra MAGKP data provide only marginal improvement to RNN-based models. On present KPG, RNN+One2Seq even has an **F\({}_{1}\)@\(\mathcal{O}\)** drop when trained with more data.

As mentioned in SS3, the RNN model is significantly lighter than TRANS. To investigate if an RNN with more parameters can benefit more from MAGKP, we conduct experiments which use a GRU with much larger hidden size (dubbed BIGRNN). Results (in Appendix E) suggest otherwise, extra training data leads to negative effect on One2One and only marginal gain on One2Seq. We thus believe the architecture difference between TRANS and RNN is the potential cause, for instance, the built-in self-attention mechanism may help TRANS models learning from noisy data.

Learning with Noisy DataTo further investigate the performance boost brought by the MAGKP dataset on TRANS+One2Seq, we are curious to know which portion of the noisy data helped the most. As a naturally way to cluster the MAGKP data, we define the noisiness by the number of keyphrases per data point. As shown in Figure 5, the distribution of MAGKP (black border) covers a

Figure 4: Comparing models trained solely with KP20k against with additional MAGKP data.

Figure 5: A histogram showing the distribution of #(kp per document) on KP20k, MAGKP and its subsets. Data points with more than 30 keyphrases are truncated.

 

much wider spectrum on the x-axis compared to KP20k (red). Because keyphrase labels are provided by human authors, a majority of its keyphrase numbers lie in the range of [3; 6]; however, only less than 20% of the MAGKP data overlaps with this number distribution.

We thus break MAGKP down into a set of smaller subset: 1) MAGKP-LN is a considerably Less Noisy subset that contains data points that have 3-6 phrases. 2) MAGKP-NLarge is the Noisy subset in which all data points have more than 10 keyphrases. 3) MAGKP-Nsmall is a randomly sampled subset of MAGKP-NLarge with the same size as MAGKP-LN.

We also define a set of data mixing strategies to compare against ALT: Only: models are trained solely on a single set (or subset) of data; MX: KP20k and MAGKP (or its subset) are split into shards (10k each) and they are randomly sampled during training; FT: models are pre-trained on MAGKP (or its subset) and fine-tuned on KP20k.

In Figure 6, we observe that none of the MAGKP subsets can match KP20k's performance in the Only setting. Because MAGKP-LN and MAGKP-Nsmall share similar data size with KP20k, this suggest the distributional shift between MAGKP and the 6 testing sets is significant. In the MX setting where KP20k is mixed with noisy data, we observe a notable performance boost compared to ONLY (yet still lower than ALT), however, we do not see clear patterns among the 4 MAGKp subsets in this setting. In the FT setting, we observe a surge in scores across all MAGKP subsets. In present RPG, both MAGKP and MAGKP-NLarge outperform the score achieved in the ALT setting; similarly, in absent KPG, MAGKP, MAGKP-NIlarge and MAGKP-Nsmall exceeds the ALT score. This is to our surprise that the subsets considered as noisy provide a greater performance boost, while they perform poorly if "Only" trained on these subsets.

To sum up, during our investigation on augmenting KP20k with the noisy MAGKP data, we obtain the best performance from a TRANS+One2Seq model that pre-trained on MAGKP and then finetuned on KP20k, and this performance has outperformed current state-or-the-art models. We conjecture that the performance gain may come from data diversity, because MAGKP contains a much wider distribution of data compared to the author keyword distribution as in KP20k. This inspires us to develop data augmentation techniques to exploit the diversity in unlabeled data.

## 6 Related Work

Traditional Keyphrase ExtractionKeyphrase extraction has been studied extensively for decades. A common approach is to formulate it as a two-step process. Specifically, a system first heuristically selects a set of candidate phrases from the text using some pre-defined features (Witten et al., 1999; Liu et al., 2011; Wang et al., 2016; Yang et al., 2017). Subsequently, a ranker is used to select the top ranked candidates following various criteria. The ranker can be bagged decision trees (Medelyan et al., 2009; Lopez and Romary, 2010), Multi-Layer Perceptron, Support Vector Machine (Lopez and Romary, 2010) or PageRank (Mihalcea and Tarau, 2004; Le et al., 2016; Wan and Xiao, 2008). Compared to the newly developed data driven approaches with deep neural networks, the above approaches suffer from poor performance and the need of dataset-specific heuristic design.

Neural Keyphrase ExtractionOn neural keyphrase extraction task, Zhang et al. (2016); Luan et al. (2017); Gollapalli et al. (2017) use sequence labeling approach; Subramanian et al. (2018) use pointer networks to select spans from source text; Sun et al. (2019) leverage graph neural networks. Despite improved over tradition approaches, the above methods do not have the capability of predicting absent keyphrases.

Meng et al. (2017) first propose the CopyRNN model, which both generates words from vocab

Figure 6: TRANS+One2Seq trained with KP20x and different subsets of MAGKP, using four data mixing strategy. Scores are averaged over all 6 test sets.

 

ularly and points to words from the source text -- overcoming the barrier of predicting absent keyphrases. Following this idea, Chen et al. (2018); Zhao and Zhang (2019) leverage the attention mechanism to help reducing duplication and improving coverage. Ye and Wang (2018) propose a semi-supervised training strategy. Yuan et al. (2020) propose One2Seq, which enables a model to generate variable number of keyphrases. Chen et al. (2019b); Ye and Wang (2018); Wang et al. (2019) propose to leverage extra structure information (e.g., title, topic) to guide the generation. Chan et al. (2019) propose an RL model, Swaminathan et al. (2020) propose using GAN for KPG. Chen et al. (2019a) retrieve similar documents from training data to help producing more accurate keyphrases. Chen et al. (2020) introduce hierarchical decoding and exclusion mechanism to prevent from generating duplication. Cano and Bojar (2019) also propose to utilize more data, but their goal is to bridge KPG with summarization.

## 7 Conclusion and Takeaways

We present an empirical study discussing neural KPG models from various aspects. Through extensive experiments and analysis, we answer the three questions (SS1). Results suggest that given a carefully chosen architecture and training strategy, a base model can perform comparable with fancy SOTA models. Further augmented with (noisy) data in the correct way, a base model can outperform SOTA models (Appendix E). We strive to provide a guideline on how to choose such architectures and training strategies, which hopefully can be proven valuable and helpful to the community.

We conclude our discussion with the following takeaways:

1. One2Seq excels at present KPG, while One2One performs better on absent KPG. See Section 3.
2. For present KPG, TRANS performs better on in-distribution data, when distribution or domain shift increase, RNN can outperform TRANS. See Section 3.
3. On absent KPG, TRANS is the clear winner. See Section 3.
4. For One2Seq, target ordering is important in greedy decoding (with Pres-Abs being an overall good choice). See Section 4.
5. The effect of target ordering tends to diminish when beam search is performed. See Section 4.
6. Large and noisy data can benefit KPG. Empirically, a decent way to leverage them is to pre-train on extra data then fine-tune on small in-domain data. See Section 5.
7. Copy mechanism helps present prediction while worsening absent performance. See Appendix C.1.
8. Larger beam width is beneficial, especially for absent KPG. However, on present KPG tasks, the benefit is diminished past a certain point and thus computational efficiency needs to be carefully considered. See Appendix C.2.

## Acknowledgments

RM was supported by the Amazon Research Awards for the project "Transferable, Controllable, Applicable Keyphrase Generation". This research was partially supported by the University of Pittsburgh Center for Research Computing through the resources provided. The authors thank the anonymous NAACL reviewers for their helpful feedback and suggestions.

## References

* Cano and Bojar (2019)E. Cano and O. Bojar. 2019. Keyphrase generation: A multi-aspect survey. In _2019 25th Conference of Open Innovations Association (FRUCT)_, pages 85-94. IEEE.
* Chan et al. (2019) Hou Pong Chan, Wang Chen, Lu Wang, and Irwin King. 2019. Neural keyphrase generation via reinforcement learning with adaptive rewards.
* Chen et al. (2018) Jun Chen, Xiaoming Zhang, Yu Wu, Zhao Yan, and Zhoujun Li. 2018. Keyphrase generation with correlation constraints. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 4057-4066, Brussels, Belgium. Association for Computational Linguistics.
* Chen et al. (2019a) Wang Chen, Hou Pong Chan, Piji Li, Lidong Bing, and Irwin King. 2019a. An integrated approach for keyphrase generation via exploring the power of retrieval and extraction. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 2846-2856, Minneapolis, Minnesota. Association for Computational Linguistics 

* Chen et al. (2020) Wang Chen, Hou Pong Chan, Piji Li, and Irwin King. 2020. Exclusive hierarchical decoding for deep keyphrase generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics.
* Chen et al. (2019b) Wang Chen, Yifan Gao, Jiani Zhang, Irwin King, and Michael R Lyu. 2019b. Title-guided encoding for keyphrase generation. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 33, pages 6268-6275.
* Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1724-1734, Doha, Qatar. Association for Computational Linguistics.
* Gehrmann et al. (2018) Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 4098-4109, Brussels, Belgium. Association for Computational Linguistics.
* Dolapalli et al. (2017) Sujatha Das Gollapalli, Xiaoli Li, and Peng Yang. 2017. Incorporating expert knowledge into keyphrase extraction. In _AAAI_, pages 3180-3187. AAAI Press.
* Gu et al. (2016) Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1631-1640, Berlin, Germany. Association for Computational Linguistics.
* Gulcehre et al. (2016) Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. 2016. Pointing the unknown words. _arXiv preprint arXiv:1603.08148_.
* Le et al. (2016) Tho Thi Ngoc Le, Minh Le Nguyen, and Akira Shimazu. 2016. Unsupervised keyphrase extraction: Introducing new kinds of words to keyphrases. _29th Australasian Joint Conference, Hobart, TAS, Australia, December 5-8, 2016_.
* Liu et al. (2011) Zhiyuan Liu, Xinxiong Chen, Yabin Zheng, and Maosong Sun. 2011. Automatic keyphrase extraction by bridging vocabulary gap. _the Fifteenth Conference on Computational Natural Language Learning_.
* Lopez and Romary (2010) Patrice Lopez and Laurent Romary. 2010. Humb: Automatic key term extraction from scientific articles in grobidp. _the 5th International Workshop on Semantic Evaluation_.
* Luan et al. (2017) Yi Luan, Mari Ostendorf, and Hannaneh Hajishirzi. 2017. Scientific information extraction with semi-supervised neural tagging. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2641-2651, Copenhagen, Denmark. Association for Computational Linguistics.
* Medelyan et al. (2009) Olena Medelyan, Eibe Frank, and Ian H. Witten. 2009. Human-competitive tagging using automatic keyphrase extraction. In _Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing_, pages 1318-1327, Singapore. Association for Computational Linguistics.
* Meng et al. (2017) Rui Meng, Sanqiang Zhao, Shuguang Han, Daqing He, Peter Brusilovsky, and Yu Chi. 2017. Deep keyphrase generation. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 582-592. Association for Computational Linguistics.
* Mihalcea and Tarau (2004) Rada Mihalcea and Paul Tarau. 2004. TextRank: Bringing order into text. In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_, pages 404-411, Barcelona, Spain. Association for Computational Linguistics.
* Ott et al. (2018) Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. _arXiv preprint arXiv:1806.00187_.
* Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. _arXiv preprint arXiv:1910.10683_.
* See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointergenerator networks. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
* Sinha et al. (2015) Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June Hsu, and Kuansan Wang. 2015. An overview of microsoft academic service (mas) and applications. In _Proceedings of the 24th international conference on world wide web_, pages 243-246.
* Strubell et al. (2019) Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3645-3650, Florence, Italy. Association for Computational Linguistics.
* Subramanian et al. (2018) Sandeep Subramanian, Tong Wang, Xingdi Yuan, Saizheng Zhang, Adam Trischler, and Yoshua Bengio. 2018. Neural models for key phrase extraction and question generation. In _Proceedings of the 

Workshop on Machine Reading for Question Answering_, pages 78-88, Melbourne, Australia. Association for Computational Linguistics.
* Sun et al. (2019) Zhiqing Sun, Jian Tang, Pan Du, Zhi-Hong Deng, and Jian-Yun Nie. 2019. Divgraphpointer: A graph pointer network for extracting diverse keyphrases. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_, SIGIR'19, page 755-764, New York, NY, USA. Association for Computing Machinery.
* Volume 2_, NIPS'14, pages 3104-3112, Cambridge, MA, USA. MIT Press.
* Swaminathan et al. (2020) Avinash Swaminathan, Haimin Zhang, Debanjan Mahata, Rakesh Gosangi, Rajiv Shah, and Amanda Stent. 2020. A preliminary exploration of gans for keyphrase generation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 8021-8030.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems 30_.
* Vinyals et al. (2016) Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. 2016. Order matters: Sequence to sequence for sets. In _International Conference on Learning Representations (ICLR)_.
* Wan and Xiao (2008) Xiaojun Wan and Jianguo Xiao. 2008. Single document keyphrase extraction using neighborhood knowledge. _AAAI_.
* Wang et al. (2016) Minmei Wang, Bo Zhao, and Yihua Huang. 2016. Prt: Phrase-based topical ranking for automatic keyphrase extraction in scientific publications. _23rd International Conference, ICONIP 2016_.
* Wang et al. (2019) Yue Wang, Jing Li, Hou Pong Chan, Irwin King, Michael R. Lyu, and Shuming Shi. 2019. Topic-aware neural keyp phrase generation for social media language. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 2516-2526, Florence, Italy. Association for Computational Linguistics.
* Witten et al. (1999) Ian H. Witten, Gordon W. Paynter, Eibe Frank, Carl Gutwin, and Craig G. Nevill-Manning. 1999. Kea: Practical automatic keyphrase extraction. In _Proceedings of the Fourth ACM Conference on Digital Libraries_, DL '99, pages 254-255, New York, NY, USA. ACM.
* Yang et al. (2017) Zhilin Yang, Junjie Hu, Ruslan Salakhutdinov, and William W. Cohen. 2017. Semi-supervised qa with generative domain-adaptive nets. In _the 55th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics.
* Ye and Wang (2018) Hai Ye and Lu Wang. 2018. Semi-supervised learning for neural keyphrase generation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 4142-4153, Brussels, Belgium. Association for Computational Linguistics.
* Yuan et al. (2020) Xingdi Yuan, Tong Wang, Rui Meng, Khushboo Thaker, Peter Brusilovsky Daqing He, and Adam Trischler. 2020. One size does not fit all: Generating and evaluating variable number of keyphrases. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics.
* Zhang et al. (2016) Qi Zhang, Yang Wang, Yeyun Gong, and Xuanjing Huang. 2016. Keyphrase extraction using deep recurrent neural networks on twitter. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 836-845. Association for Computational Linguistics.
* Zhao and Zhang (2019) Jing Zhao and Yuxiang Zhang. 2019. Incorporating linguistic constraints into keyphrase generation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 5224-5233.

 



#### Contents in Appendices:

* In Appendix A, we provide the formal definition of all evaluation metrics we used in this work.
* In Appendix B, we provide detailed statistics of all datasets used in this work.
* In Appendix C, we provide observation and analysis on additional factors that can affect a KPG system's performance.
* In Appendix D, show the set of heat maps that are not shown in the main content due to space limit.
* In Appendix E, we provide a complete set of numbers containing all results discussed in this work, compared with a set of SOTA models in existing literature.
* In Appendix F, we provide implementation details that helps to reproduce our experiments.

## Appendix A Evaluation Metric Definition

In this section, we provide definition of the metrics we use in this work. All metrics are adopted from [17] and [20]. To make the results easy to reproduce, we simply _report macro-average scores over all the data examples_ in a dataset (rather than removing examples that contain no present/absent phrases). Since some data examples contain no valid present/absent phrase and lead to zero scores, this causes _our results can be lower than previously reported results_.

Given a data example consisting a source text \({\cal X}\) and a list of target keyphrases \({\hat{\mathcal{Y}}}\), suppose that a model predicts a list of unique keyphrases \({\hat{\mathcal{Y}}}=(\hat{y}_{1},\ldots,\hat{y}_{m})\) ordered by the quality of the predictions \(\hat{y}_{i}\), and that the ground truth keyphrases for the given source text is the oracle set \({\cal Y}\). When only the top \(k\) predictions \({\hat{\mathcal{Y}}}_{:k}=(\hat{y}_{1},\ldots,\hat{y}_{\min(k,m)})\) are used for evaluation, _precision_, _recall_, and _F\({}_{1}\)_-score are consequently conditioned on \(k\) and defined as:

\[\begin{split}\textbf{P@}k&=\frac{|{\hat{\mathcal{Y}}} _{:k}\cap{\mathcal{Y}}|}{|{\hat{\mathcal{Y}}}_{:k}|},\qquad\textbf{R@}k=\frac{ |{\hat{\mathcal{Y}}}_{:k}\cap{\mathcal{Y}}|}{|{\mathcal{Y}}|},\\ \textbf{F}_{1}@k&=\frac{2*\textbf{P@}k*\textbf{R @} k}{\textbf{P@}k+\textbf{R@}k}.\end{split} \tag{1}\]

Thus the metrics are defined as:

* **F\({}_{1}\)@5**: **F\({}_{1}\)@\(k\)** when \(k=5\).
* **F\({}_{1}\)@10**: **F\({}_{1}\)@\(k\)** when \(k=10\).
* **F\({}_{1}\)@\({\cal O}\)**: \({\cal O}\) denotes the number of oracle (ground truth) keyphrases. In this case, \(k=|{\hat{\mathcal{Y}}}|\), which means for each data example, the number of predicted phrases taken for evaluation is the same as the number of ground-truth keyphrases.
* **F\({}_{1}\)@\({\cal M}\)**: \({\cal M}\) denotes the number of predicted keyphrases. In this case, \(k=|{\hat{\mathcal{Y}}}|\) and we simply take all the predicted phrases for evaluation without truncation.
* **R@50**: **R@\(k\)** when \(k=50\).

## Appendix B Statistics of Datasets

We provide details of datasets used in this work. We use KP20K-train[34] and MAGKP[69] for training keyphrase generation models, both are built on the basis of scientific publications in Computer Science domain. Nevertheless, their distributions are considerably different, e.g. MAGKP data contains 3 times more keyphrases on average than KP20K. This is because KP20K is constructed using real author keywords whereas MAGKP may contain a vast amount of keyphrases annotated by automatic systems. Detailed statistics are listed in Table 2. We also leave out certain amount of data points from KP20K for validation and testing (KP20K-VALID and KP20K-TEST).

We also utilize five other datasets for evaluation purposes, as shown in Table 3. All except DUC come from scientific publications in Computer Science domain. KRAPIVIN uses keywords provided by the authors as targets, which is the same as KP20K. INSPEC, NUS, and SEMEVAL contain author-assigned keywords and additional keyphrases provided by third-party annotators. DUC, different from all above, is a keyphrase dataset based on news articles. Since it represents a rather different distribution from scientific publication datasets, hypothetically, obtaining decent test score on DUC requires extra generalizability.

## Appendix C Other Model Designing Aspects

Besides the findings we discuss in the paper, there exist other important factors affect the general performance of KPG models. We provides two addi

tional empirical results that we think might be of interest to certain readers.

### Effect of Copy Mechanism

Copy mechanism (Gu et al., 2016) (also referred to as Pointer Generator See et al., 2017) or Pointer Softmax Gulcehre et al. (2016)) has demonstrated to play a critical role in tasks where texts on the source and target side may overlap, such as summarization See et al. (2017) and keyphrase generation Meng et al. (2017). Basically, it is an additional loss that enables models to extract information from the source side with the help of attentions. Prior studies Meng et al. (2017) have shown the importance of copy mechanism with RNN+One2One, but no further comparison has been made.

In Figure 7, we present the results of four KPG model variants, equipped with and without copy mechanism. The results show that copy mechanism leads to considerable improvements on present KPG, especially for RNN. TRANS benefits less from the copy, which may be because its multi-head attentions behave similarly to the copy mechanism even without explicit training losses. With regard to the absent KPG results, copy mechanism only helps RNN+One2One. This suggests that TRANS can achieve consistently better abstractiveness (absent performance) by disabling the copy mechanism at the cost of weaker extractiveness. This dilemma cautions researchers to use copy mechanism more wisely according to specific applications.

### Effect of Beam Width

As discussed in SS2, one unique challenge of the KPG task is due to its multiplicity of target outputs. As a result, a common strategy is to take multiple beams during decoding in order to obtain more phrases (as opposed to greedy decoding). This choice is at times not only practical but in fact necessary: under the One2One paradigm, for example, it is crucial to have multiple beams in order to generate multiple keyphrases for a given input.

Generally speaking, KPG and its evaluation metrics are in general favors higher recall. It is thus not totally unexpected that the high precision scores of greedy decoding are often undermined by notable disadvantages in recall, which in turn leads to losing by large margins in F-scores when compared to results of beam search (with multiple beams). Empirically, as shown in Figure 8, we observe that beam search can sometimes achieve a relative gain of more than \(10\%\) in present phrase generation , and a much larger performance boost in absent phrase generation, over greedy decoding.

We are also interested in seeing if there exists an optimal beam width. In Figure 8, we show models' testing performance when various beam widths are

Figure 8: Averaged scores of models using different widths of Beam Search.

\begin{table}
\begin{tabular}{r|c|c|c|c} Dataset & \#Data & AvgRp & \#uniquekp & \#(word) per kpc \\ \hline KP20k-train & 514K & 2.7M & 700K & 1.92 \\ MARSER & 2.7M & 41.6M & 6.9M & 3.42 \\ MAGRF-LN & 522K & 2.3M & 579K & 2.73 \\ MAGRF-NIraper & 1.5M & 35.5M & 5.8M & 3.38 \\ MAGRF-Nsmall & 522K & 12.2M & 2.2M & 3.37 \\ \end{tabular}
\end{table}
Table 2: Statistics of training datasets.

Figure 7: Averaged scores of models with and without utilizing copy mechanism.

 

used. In present KPG task with One2One (upper left), beam width of 16 already provides an optimal score, larger beam widths (even 200) do not show any further advantage. Replacing the training paradigm with One2Seq (upper right), we observe a positive correlation between beam width and testing score -- larger beam widths lead to marginally better testing scores. However, the improvement (from beam size of 10 to 50) is not significant.

On absent KPG task (lower), both One2One and One2Seq paradigms seem to benefit from larger beam widths. Testing score shows strong positive correlation with beam width. We observe that this trend is consistent across different model architectures.

Overall, a larger beam width provides better scores in most settings, but the performance gain diminishes quickly towards very large beam width. In addition, it is worth noting that larger beam width also comes with more intense computing demands, for both space and time. As an example, in Figure 8 (top left), we observe that with the One2One training paradigm, a beam width of 200 does not show a significant advantage over 16, however, in terms of computation, beam width of 200 takes about \(10\times\) of the resources compared to 16. There clearly exists a trade-off between beam width and computational efficiency (e.g., carbon footprint Strubell et al. (2019)). We thus hope our results can serve as a reference for researchers, to help them choose beam width more wisely depending on specific tasks.

## Appendix D Does Order Matter in One2Seq? -- Additional Results

In SS4, we show models' performance trained with the One2Seq paradigm using different target ordering strategies. Here we provide the complete set of heat maps.

In Figure 9, we show present KPG testing scores in **F\({}_{1}@\mathcal{O}\)**, when using either greedy decoding or beam search as decoding strategy.

In Figure 10, we show present KPG testing scores in **F\({}_{1}@\textbf{10}\)**, when using either greedy decoding or beam search as decoding strategy.

In Figure 11, we show absent KPG testing scores in **R\(@\textbf{50}\)**, when using either greedy decoding or beam search as decoding strategy.

In addition, we shown in Figure 12, 13, and 14 the number of unique predictions on all/present/absent KPG tasks.

## Appendix E Complete Results

In this section, we report the full set of our experimental results.

In Table 4, we report all the testing scores on present keyphrase generation tasks. For all experiments, we use **F\({}_{1}@\textbf{5}\)**, **F\({}_{1}@\textbf{10}\)**, **F\({}_{1}@\mathcal{O}\)** and **F\({}_{1}@\mathcal{M}\)** to evaluate a model's performance. Additionally, we provide an average score for each of the 4 metrics over all datasets (over each row in Table 4).

In Table 5, we report all the testing scores on absent keyphrase generation tasks. For all experiments, we use **R\(@\textbf{10}\)** and **R\(@\textbf{50}\)** to evaluate a model's performance. Additionally, we provide an average score for each of the 2 metrics over all datasets (over each row in Table 5).

In table 7, we report detailed present testing scores when model trained with One2Seq paradigm, using different ordering strategies. For all experiments, we use **F\({}_{1}@\textbf{5}\)**, **F\({}_{1}@\textbf{10}\)**, **F\({}_{1}@\mathcal{O}\)** and **F\({}_{1}@\mathcal{M}\)** to evaluate a model's performance.

In table 8, we report detailed absent testing scores when model trained with One2Seq paradigm, using different ordering strategies. For all experiments, we use **R\(@\textbf{10}\)** and **R\(@\textbf{50}\)** to evaluate a model's performance.

We also provide scores against all ground-truth phrases (without splitting present/absent) in Table 6 and 9 to avoid the inconsistency in data processing (present/absent split may vary by ways of tokenization).

 



\begin{table}
\begin{tabular}{l|c c



\begin{table}
\begin{tabular}{c|c c c c c c c c c c c c c c c c c c c c c c c c} \hline \hline  & \multicolumn{3}{c|}{**Average**} & \multicolumn{3}{c|}{**Kgilk**} & \multicolumn{3}{c|}{**Imper**} & \multicolumn{3}{c|}{**Exorbita**} & \multicolumn{3}{c|}{**NMS**} & \multicolumn{3}{c}{**Smellat**} & \multicolumn{3}{c}{**BMC**} \\ \hline Method & 19 & 20 & 86 & A/I & 180 & S/A & A/I & 180 & S/A & A/I & 180 & S/A & I/A & 180 & S/A & A/I & 180 & S/A & A/I & 180 & S/A \\ \hline
**On-Mode variance** & & & & & & & & & & & & & & & & & & & & & & & \\ \hline SWS-OX-OX-X20.8 & 4.5 & 87.2 & 10.6 & 6.7 & 13.3 & 29.6 & 7.8 & 13.7 & 20.7 & 4.2 & 5.2 & 13.6 & 4.7 & 13.2 & 17.5 & 24.4 & 6.1 & 100 & 00 & 00 & 00 & 03 \\ WEN-OX-X20.8 & 4.0 & 87.2 & 10.6 & 6.7 & 13.1 & 29.6 & 7.8 & 13.7 & 20.7 & 4.2 & 5.2 & 13.6 & 4.7 & 13.2 & 17.5 & 24.4 & 6.1 & 100 & 00 & 00 & 00 & 03 \\ WEN-OX-X20.8 & 4.0 & 87.2 & 10.6 & 6.7 & 5.3 & 13.1 & 29.3 & 14.6 & 25.7 & 5.3 & 5.4 & 5.4 & 13.5 & 17.0 & 33.8 & 18.7 & 5.2 & 5.2 & 100 & 00 & 00 & 01 & 00 \\ \hline RD



\begin{table} \end{tabular}
[ENDFOOTNOTE]

\begin{tabular}{c c



\begin{table} \end{tabular}

\begin{tabular}{c|c c



\begin{table}
\begin{tabular}{c|c c c c c c c c c c c c c c c c c c c c c c c c c c c c c} \hline \hline  & & & **Average** & & & & & & & & & & & & & & & & & & & & & & \\ \hline Model & **10** & **80** & **Ad** & **10** & **90** & **Ad** & **10** & **10** & **90** & **Ad** & **10** & **90** & **Ad** & **10** & **90** & **Ad** & **10** & **90** & **Ad** & **10** & **90** & **Ad** & **10** & **90** & **Ad** \\ \hline
**3967 Greedy** & & 0.2 & 0.5 & 0.2 & 0.5 & 0.2 & 0.4 & 0.4 & 0.4 & 0.1 & 0.1 & 0.1 & 0.1 & 0.2 & 0.2 & 0.2 & 0.3 & 0.3 & 0.3 & 0.0 & 0.0 & 0.0 \\  & 0.2 & 0.2 & 0.2 & 0.2 & 0.2 & 0.2 & 0.4 & 0.4 & 0.4 & 0.1 & 0.1 & 0.1 & 0.1 & 0.1 & 0.1 & 0.3 & 0.3 & 0.3 & 0.0 & 0.0 & 0.0 \\  & 0.2 & 0.2 & 0.2 & 0.2 & 0.2 & 0.2 & 0.2 & 0.4 & 0.4 & 0.4 & 0.1 & 0.1 & 0.1 & 0.1 & 0.1 & 0.1 & 0.3 & 0.3 & 0.3 & 0.0 & 0.0 & 0.0 \\  & 0.3 & 0.1 & 0.1 & 0.2 & 0.2 & 0.2 & 0.2 & 0.1 & 0.1 & 0.1 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\  & 0.5 & 0.1 & 0.1 & 0.1 & 0.2 & 0.2 & 0.2 & 0.1 & 0.1 & 0.1 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\  & 0.5 & 0.1 & 0.1 & 0.2 & 0.2 & 0.2 & 0.1 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\  & 0.1-0.0 & 0.1 & 0.1 & 0.2 & 0.2 & 0.2 & 0.1 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\  & 0.2+0.0 & 0.2 & 0.2 & 0.2 & 0.2 & 0.2 & 0.2 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\  & 0.3+0.0 & 0.1 & 0.1 & 0.2 & 0.2 & 0.2 & 0.2 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\  & 0.4+0.0 & 0.2 & 0.





\begin{table}
\begin{tabular}{c|c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c} \hline \hline  & 1 & & **Average** & 1 & & & **Seg./NR** & & & & & & & & & & & & & & & & & & & & & & & \\ \hline Model & 1 & 2 & 30 & 45 & 86 & 17 & 93 & 83 &


* [19] S.-C. Chen, J.-H. Wang, and X.-J. Zhang, "A new approach to the classification of deep neural networks for image recognition," in _IEEE Transactions on Image Processing Systems_, vol. 34, no. 2, pp. 507-518, Jan. 2016.
* [20] D. C. Kingma and A. Ba, "Adam: a method for stochastic optimization," in _Proceedings of the IEEE conference on computer vision and pattern recognition_, volume 1, pages 1-10, Dec. 2015.
* [21] M. Krizhevsky, I. Sutskever, and E. Hinton, "Deep learning with tiny data sets," _arXiv preprint arXiv:1412.6159_, 2014.
* [22] R. Girshick, L. Warde, and Y. Bengio, "Generative adversarial nets for large scale machine translation," _CoRR_, abs/1412.0558v1, 2014.
* [23] G. Karaprasov, V. Mnih, N. Bahdan, P. Dollar, and R. Darrell, "Very diverse convolutional networks for large scale machine translation," _Neural Networks_, 2(1):1-10, 2015.
* [24] H. Li, W. Liu, Z. Huang, and B. Yuille, "ImageNet: A simple framework for efficient image segmentation," in _CVPR_, 2015.

[MISSING_PAGE_POST]

 


* [10] S.-H. Wang, J.-L. Wu, and X.-J. Zhang, "A new approach to the detection of high-quality images using deep learning with convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 2378-2389, 2016.
* [11] M.-S. Huang, Y.-C. Chen, and L.-Y. Sun, "Deep residual learning for image recognition," _IEEE Transactions on Image Processing Systems_**54**, no. 1, pp. 1-10, Jan. 2016.
* [12] A.-M. Lin, R.-T. Ma, and C.-B. Yuille, "Image classification via sparse regression," _arXiv preprint arXiv:1503.04987_ (2015).
* [13] D. Liu, Z.-Q. Li, and H.-X. Tang, "Image classification via sparse regression," _IEEE transactions on pattern analysis and machine intelligence_**50**, no. 1, pp. 1-1, Feb. 2016.

[MISSING_PAGE_POST]

 

* [19] M. C. Bader, A. D. Krizhevsky, and S. Ghemawat, "A new approach to deep learning for image recognition in the wild: a survey of machine learning," _IEEE Transactions on Image Processing Systems_**53**(2):647-680, June 2002.
* [20] J. L. Friedman, R. E. Hinton, and P. H. Zisserman, "Deep learning with applications to image processing," _arXiv preprint arXiv:1407.6098_ (2014).
* [21] T. F. Goodfellow, I. Guyon, Y. Bengio, and J. Dean, "Generative adversarial nets for large data sets," _Advances in neural information processing systems_**10**(1):1-10, July 2014.

[MISSING_PAGE_POST]