Generating Diverse Numbers of Diverse Keyphrases

Existing keyphrase generation studies suffer from the problems of generating duplicate phrases and deficient evaluation based on a fixed number of predicted phrases. We propose a recurrent generative model that generates multiple keyphrases sequentially from a text, with specific modules that promote generation diversity. We further propose two new metrics that consider a variable number of phrases. With both existing and proposed evaluation setups, our model demonstrates superior performance to baselines on three types of keyphrase generation datasets, including two newly introduced in this work: StackExchange and TextWorld ACG. In contrast to previous keyphrase generation approaches, our model generates sets of diverse keyphrases of a variable number.

section_name: Introduction
paragraphs: Keyphrase generation is the task of automatically predicting keyphrases given a source text. Desired keyphrases are often multi-word units that summarize the high-level meaning and highlight certain important topics or information of the source text. Consequently, models that can successfully perform this task should be capable of not only distilling high-level information from a document, but also locating specific, important snippets therein.
To make the problem even more challenging, a keyphrase may or may not be a substring of the source text (i.e., it may be present or absent). Moreover, a given source text is usually associated with a set of multiple keyphrases. Thus, keyphrase generation is an instance of the set generation problem, where both the size of the set and the size (i.e., the number of tokens in a phrase) of each element can vary depending on the source.
Similar to summarization, keyphrase generation is often formulated as a sequence-to-sequence (Seq2Seq) generation task in most prior studies BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 . Conditioned on a source text, Seq2Seq models generate phrases individually or as a longer sequence jointed by delimiting tokens. Since standard Seq2Seq models generate only one sequence at a time, thus to generate multiple phrases, a common approach is to over-generate using beam search with a large beam width. Models are then evaluated by taking a fixed number of top predicted phrases (typically 5 or 10) and comparing them against the ground truth keyphrases.
Though this approach has achieved good empirical results, we argue that it suffers from two major limitations. Firstly, models that use beam search to generate multiple keyphrases generally lack the ability to determine the dynamic number of keyphrases needed for different source texts. Meanwhile, the parallelism in beam search also fails to model the inter-relation among the generated phrases, which can often result in diminished diversity in the output. Although certain existing models take output diversity into consideration during training BIBREF1 , BIBREF2 , the effort is significantly undermined during decoding due to the reliance on over-generation and phrase ranking with beam search.
Secondly, the current evaluation setup is rather problematic, since existing studies attempt to match a fixed number of outputs against a variable number of ground truth keyphrases. Empirically, the number of keyphrases can vary drastically for different source texts, depending on a plethora of factors including the length or genre of the text, the granularity of keyphrase annotation, etc. For the several commonly used keyphrase generation datasets, for example, the average number of keyphrases per data point can range from 5.3 to 15.7, with variances sometimes as large as 64.6 (Table TABREF1 ). Therefore, using an arbitrary, fixed number INLINEFORM0 to evaluate entire datasets is not appropriate. In fact, under this evaluation setup, the F1 score for the oracle model on the KP20k dataset is 0.858 for INLINEFORM1 and 0.626 for INLINEFORM2 , which apparently poses serious normalization issues as evaluation metrics.
To overcome these problems, we propose novel decoding strategies and evaluation metrics for the keyphrase generation task. The main contributions of this work are as follows:
section_name: Keyphrase Extraction and Generation
paragraphs: Traditional keyphrase extraction has been studied extensively in past decades. In most existing literature, keyphrase extraction has been formulated as a two-step process. First, lexical features such as part-of-speech tags are used to determine a list of phrase candidates by heuristic methods BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 . Second, a ranking algorithm is adopted to rank the candidate list and the top ranked candidates are selected as keyphrases. A wide variety of methods were applied for ranking, such as bagged decision trees BIBREF8 , BIBREF9 , Multi-Layer Perceptron, Support Vector Machine BIBREF9 and PageRank BIBREF10 , BIBREF11 , BIBREF12 . Recently, BIBREF13 , BIBREF14 , BIBREF15 used sequence labeling models to extract keyphrases from text. Similarly, BIBREF16 used Pointer Networks to point to the start and end positions of keyphrases in a source text.
The main drawback of keyphrase extraction is that sometimes keyphrases are absent from the source text, thus an extractive model will fail predicting those keyphrases. BIBREF0 first proposed the CopyRNN, a neural generative model that both generates words from vocabulary and points to words from the source text. Recently, based on the CopyRNN architecture, BIBREF1 proposed the CorrRNN, which takes states and attention vectors from previous steps into account in both encoder and decoder to reduce duplication and improve coverage. BIBREF2 proposed semi-supervised methods by leveraging both labeled and unlabeled data for training. BIBREF3 , BIBREF2 proposed to use structure information (e.g., title of source text) to improve keyphrase generation performance. Note that none of the above works are able to generate variable number of phrases, which is one of our contributions.
section_name: Sequence to Sequence Generation
paragraphs: Sequence to Sequence (Seq2Seq) learning was first introduced by BIBREF17 ; together with the soft attention mechanism of BIBREF18 , it has been widely used in natural language generation tasks. BIBREF19 , BIBREF20 used a mixture of generation and pointing to overcome the problem of large vocabulary size. BIBREF21 , BIBREF22 applied Seq2Seq models on summary generation tasks, while BIBREF23 , BIBREF24 generated questions conditioned on documents and answers from machine comprehension datasets. Seq2Seq was also applied on neural sentence simplification BIBREF25 and paraphrase generation tasks BIBREF26 .
Given a source text consisting of INLINEFORM0 words INLINEFORM1 , the encoder converts their corresponding embeddings INLINEFORM2 into a set of INLINEFORM3 real-valued vectors INLINEFORM4 with a bidirectional GRU BIBREF27 : DISPLAYFORM0 
Dropout BIBREF28 is applied to both INLINEFORM0 and INLINEFORM1 for regularization.
The decoder is a uni-directional GRU, which generates a new state INLINEFORM0 at each time-step INLINEFORM1 from the word embedding INLINEFORM2 and the recurrent state INLINEFORM3 : DISPLAYFORM0 
The initial state INLINEFORM0 is derived from the final encoder state INLINEFORM1 by applying a single-layer feed-forward neural net (FNN): INLINEFORM2 . Dropout is applied to both the embeddings INLINEFORM3 and the GRU states INLINEFORM4 .
When generating token INLINEFORM0 , in order to better incorporate information from the source text, an attention mechanism BIBREF18 is employed to infer the importance INLINEFORM1 of each source word INLINEFORM2 given the current decoder state INLINEFORM3 . This importance is measured by an energy function with a 2-layer FNN: DISPLAYFORM0 
The output over all decoding steps INLINEFORM0 thus define a distribution over the source sequence: DISPLAYFORM0 
These attention scores are then used as weights for a refined representation of the source encodings, which is then concatenated to the decoder state INLINEFORM0 to derive a generative distribution INLINEFORM1 : DISPLAYFORM0 
where the output size of INLINEFORM0 equals to the target vocabulary size. Subscript INLINEFORM1 indicates the abstractive nature of INLINEFORM2 since it is a distribution over a prescribed vocabulary.
We employ the pointer softmax BIBREF19 mechanism to switch between generating a token INLINEFORM0 (from a vocabulary) and pointing (to a token in the source text). Specifically, the pointer softmax module computes a scalar switch INLINEFORM1 at each generation time-step and uses it to interpolate the abstractive distribution INLINEFORM2 over the vocabulary (see Equation EQREF16 ) and the extractive distribution INLINEFORM3 over the source text tokens: DISPLAYFORM0 
where INLINEFORM0 is conditioned on both the attention-weighted source representation INLINEFORM1 and the decoder state INLINEFORM2 : DISPLAYFORM0 
section_name: Model Architecture
paragraphs: Given a piece of source text, our objective is to generate a variable number of multi-word phrases. To this end, we opt for the sequence-to-sequence framework (Seq2Seq) as the basis of our model, combined with attention and pointer softmax mechanisms in the decoder.
Since each data example contains one source text sequence and multiple target phrase sequences (dubbed One2Many, and each sequence can be of multi-word), two paradigms can be adopted for training Seq2Seq models. The first one BIBREF0 is to divide each One2Many data example into multiple One2One examples, and the resulting models (e.g. CopyRNN) can generate one phrase at once and must rely on beam search technique to produce more unique phrases.
To enable models to generate multiple phrases and control the number to output, we propose the second training paradigm One2Seq, in which we concatenate multiple phrases into a single sequence with a delimiter INLINEFORM0 SEP INLINEFORM1 , and this concatenated sequence is then used as the target for sequence generation during training. An overview of the model's structure is shown in Figure FIGREF8 .
section_name: Notations
paragraphs: In the following subsections, we use INLINEFORM0 to denote input text tokens, INLINEFORM1 to denote token embeddings, INLINEFORM2 to denote hidden states, and INLINEFORM3 to denote output text tokens. Superscripts denote time-steps in a sequence, and subscripts INLINEFORM4 and INLINEFORM5 indicate whether a variable resides in the encoder or the decoder of the model, respectively. The absence of a superscript indicates multiplicity in the time dimension. INLINEFORM6 refers to a linear transformation and INLINEFORM7 refers to it followed by a non-linear activation function INLINEFORM8 . Angled brackets, INLINEFORM9 , denote concatenation.
section_name: Mechanisms for Diverse Generation
paragraphs: There are usually multiple keyphrases for a given source text because each keyphrase represents certain aspects of the text. Therefore keyphrase diversity is desired for the keyphrase generation. Most previous keyphrase generation models generate multiple phrases by over-generation, which is highly prone to generate similar phrases due to the nature of beam search. Given our objective to generate variable numbers of keyphrases, we need to adopt new strategies for achieving better diversity in the output.
Recall that we represent variable numbers of keyphrases as delimiter-separated sequences. One particular issue we observed during error analysis is that the model tends to produce identical tokens following the delimiter token. For example, suppose a target sequence contains INLINEFORM0 delimiter tokens at time-steps INLINEFORM1 . During training, the model is rewarded for generating the same delimiter token at these time-steps, which presumably introduces much homogeneity in the corresponding decoder states INLINEFORM2 . When these states are subsequently used as inputs at the time-steps immediately following the delimiter, the decoder naturally produces highly similar distributions over the following tokens, resulting in identical tokens being decoded. To alleviate this problem, we propose two plug-in components for the sequential generation model.
We propose a mechanism called semantic coverage that focuses on the semantic representations of generated phrases. Specifically, we introduce another uni-directional recurrent model INLINEFORM0 (dubbed target encoder) which encodes decoder-generated tokens INLINEFORM1 , where INLINEFORM2 , into hidden states INLINEFORM3 . This state is then taken as an extra input to the decoder GRU, modifying Equation EQREF12 to: DISPLAYFORM0 
If the target encoder were to be updated with the training signal from generation (i.e., backpropagating error from the decoder GRU to the target encoder), the resulting decoder is essentially a 2-layer GRU with residual connections. Instead, inspired by previous representation learning works BIBREF29 , BIBREF30 , BIBREF31 , we train the target encoder in an self-supervised fashion (Figure FIGREF8 ). That is, we extract target encoder's final hidden state vector INLINEFORM0 , where INLINEFORM1 is the length of target sequence, and use it as a general representation of the target phrases. We train by maximizing the mutual information between these phrase representations and the final state of the source encoder INLINEFORM2 as follows. For each phrase representation vector INLINEFORM3 , we take the enocdings INLINEFORM4 of INLINEFORM5 different source texts, where INLINEFORM6 is the encoder representation for the current source text, and the remaining INLINEFORM7 are negative samples (sampled at random) from the training data. The target encoder is trained to minimize the classification loss: DISPLAYFORM0 
where INLINEFORM0 is bi-linear transformation.
The motivation here is to constrain the overall representation of generated keyphrase to be semantically close to the overall meaning of the source text. With such representations as input to the decoder, the semantic coverage mechanism can potentially help to provide useful keyphrase information and guide generation.
We also propose orthogonal regularization, which explicitly encourages the delimiter-generating decoder states to be different from each other. This is inspired by BIBREF32 , who use orthogonal regularization to encourage representations across domains to be as distinct as possible. Specifically, we stack the decoder hidden states corresponding to delimiters together to form matrix INLINEFORM0 and use the following equation as the orthogonal regularization loss: DISPLAYFORM0 
where INLINEFORM0 is the matrix transpose of INLINEFORM1 , INLINEFORM2 is the identity matrix of rank INLINEFORM3 , INLINEFORM4 indicates element wise multiplication, INLINEFORM5 indicates INLINEFORM6 norm of each element in a matrix INLINEFORM7 . This loss function prefers orthogonality among the hidden states INLINEFORM8 and thus improves diversity in the tokens following the delimiters.
We adopt the widely used negative log-likelihood loss in our sequence generation model, denoted as INLINEFORM0 . The overall loss we use in our model is DISPLAYFORM0 
where INLINEFORM0 and INLINEFORM1 are hyper-parameters.
section_name: Decoding Strategies
paragraphs: According to different task requirements, various decoding methods can be applied to generate the target sequence INLINEFORM0 . Prior studies BIBREF0 , BIBREF7 focus more on generating excessive number of phrases by leveraging beam search to proliferate the output phrases. In contrast, models trained under One2Seq paradigm are capable of determining the proper number of phrases to output. In light of previous research in psychology BIBREF33 , BIBREF34 , we name these two decoding/search strategies as Exhaustive Decoding and Self-terminating Decoding, respectively, due to their resemblance to the way humans behave in serial memory tasks. Simply speaking, the major difference lies in whether a model is capable of controlling the number of phrases to output. We describe the detailed decoding strategies used in this study as follows:
As traditional keyphrase tasks evaluate models with a fixed number of top-ranked predictions (say F-score @5 and @10), existing keyphrase generation studies have to over-generate phrases by means of beam search (commonly with a large beam size, e.g., 150 and 200 in BIBREF3 , BIBREF0 , respectively), a heuristic search algorithm that returns INLINEFORM0 approximate optimal sequences. For the One2One setting, each returned sequence is a unique phrase itself. But for One2Seq, each produced sequence contains several phrases and additional processes BIBREF2 are needed to obtain the final unique (ordered) phrase list.
It is worth noting that the time complexity of beam search is INLINEFORM0 , where INLINEFORM1 is the beam width, and INLINEFORM2 is the maximum length of generated sequences. Therefore the exhaustive decoding is generally very computationally expensive, especially for One2Seq setting where INLINEFORM3 is much larger than in One2One. It is also wasteful as we observe that less than 5% of phrases generated by One2Seq models are unique.
An innate characteristic of keyphrase tasks is that the number of keyphrases varies depending on the document and dataset genre, therefore dynamically outputting a variable number of phrases is a desirable property for keyphrase generation models. Since our proposed model is trained to generate a variable number of phrases as a single sequence joined by delimiters, we can obtain multiple phrases by simply decoding a single sequence for each given source text. The resulting model thus implicitly performs the additional task of dynamically estimating the proper size of the target phrase set: once the model believes that an adequate number of phrases have been generated, it outputs a special token INLINEFORM0 EOS INLINEFORM1 to terminate the decoding process.
One notable attribute of the self-terminating decoding strategy is that, by generating a set of phrases in a single sequence, the model conditions its current generation on all previously generated phrases. Compared to the exhaustive strategy (i.e., phrases being generated independently by beam search in parallel), our model can model the dependency among its output in a more explicit fashion. Additionally, since multiple phrases are decoded as a single sequence, decoding can be performed more efficiently than exhaustive decoding by conducting greedy search or beam search on only the top-scored sequence.
section_name: Evaluating Keyphrase Generation
paragraphs: Formally, given a source text, suppose that a model predicts a list of unique keyphrases INLINEFORM0 ordered by the quality of the predictions INLINEFORM1 , and that the ground truth keyphrases for the given source text is the oracle set INLINEFORM2 . When only the top INLINEFORM3 predictions INLINEFORM4 are used for evaluation, precision, recall, and F INLINEFORM5 score are consequently conditioned on INLINEFORM6 and defined as: DISPLAYFORM0 
As discussed in Section SECREF1 , the number of generated keyphrases used for evaluation can have a critical impact on the quality of the resulting evaluation metrics. Here we compare three choices of INLINEFORM0 and the implications on keyphrase evaluation for each choice:
A simple remedy is to set INLINEFORM0 as a variable number which is specific to each data example. Here we define two new metrics:
By simply extending the constant number INLINEFORM0 to different variables accordingly, both F INLINEFORM1 @ INLINEFORM2 and F INLINEFORM3 @ INLINEFORM4 are capable of reflecting the nature of variable number of phrases for each document, and a model can achieve the maximum INLINEFORM5 score of INLINEFORM6 if and only if it predicts the exact same phrases as the ground truth. Another merit of F INLINEFORM7 @ INLINEFORM8 is that it is independent from model outputs, therefore we can use it to compare existing models.
section_name: Datasets and Experiments
paragraphs: In this section, we report our experiment results on multiple datasets and compare with existing models. We use INLINEFORM0 to refer to the delimiter-concatenated sequence-to-sequences model described in Section SECREF3 ; INLINEFORM1 refers to the model augmented with orthogonal regularization and semantic coverage mechanism.
To construct target sequences for training INLINEFORM0 and INLINEFORM1 , ground truth keyphrases are sorted by their order of first occurrence in the source text. Keyphrases that do not appear in the source text are appended to the end. This order may guide the attention mechanism to attend to source positions in a smoother way. Implementation details can be found in Appendix SECREF9 .
We include four non-neural extractive models and CopyRNN BIBREF0 as baselines. We use CopyRNN to denote the model reported by BIBREF0 , CopyRNN* to denote our implementation of CopyRNN based on their open sourced code. To draw fair comparison with existing study, we use the same model hyperparameter setting as used in BIBREF0 and use exhaustive decoding strategy for most experiments. KEA BIBREF4 and Maui BIBREF8 are trained on a subset of 50,000 documents from either KP20k (Table TABREF35 ) or StackEx (Table TABREF37 ) instead of all documents due to implementation limits (without fine-tuning on target dataset).
In Section SECREF42 , we apply the self-terminating decoding strategy. Since no existing model supports such decoding strategy, we only report results from our proposed models. They can be used for comparison in future studies.
section_name: Experiments on Scientific Publications
paragraphs: Our first dataset consists of a collection of scientific publication datasets, namely KP20k, Inspec, Krapivin, NUS, and SemEval, that have been widely used in existing literature BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 . KP20k, for example, was introduced by BIBREF0 and comprises more than half a million scientific publications. For each article, the abstract and title are used as the source text while the author keywords are used as target. The other four datasets contain much fewer articles, and thus used to test transferability of our model (without fine-tuning).
We report our model's performance on the present-keyphrase portion of the KP20k dataset in Table TABREF35 . To compare with previous works, we provide compute INLINEFORM0 and INLINEFORM1 scores. The new proposed F INLINEFORM2 @ INLINEFORM3 metric indicates consistent ranking with INLINEFORM4 for most cases. Due to its target number sensitivity, we find that its value is closer to INLINEFORM5 for KP20k and Krapivin where average target keyphrases is less and closer to INLINEFORM6 for the other three datasets.
From the result we can see that the neural-based models outperform non-neural models by large margins. Our implemented CopyRNN achieves better or comparable performance against the original model, and on NUS and SemEval the advantage is more salient.
As for the proposed models, both INLINEFORM0 and INLINEFORM1 yield comparable results to CopyRNN, indicating that One2Seq paradigm can work well as an alternative option for the keyphrase generation. INLINEFORM2 outperforms INLINEFORM3 on all metrics, suggesting the semantic coverage and orthogonal regularization help the model to generate higher quality keyphrases and achieve better generalizability. To our surprise, on the metric F INLINEFORM4 @10 for KP20k and Krapivin (average number of keyphrases is only 5), where high-recall models like CopyRNN are more favored, INLINEFORM5 is still able to outperform One2One baselines, indicating that the proposed mechanisms for diverse generation are effective.
section_name: Experiments on The StackEx Dataset
paragraphs: Inspired by the StackLite tag recommendation task on Kaggle, we build a new benchmark based on the public StackExchange data. We use questions with titles as source, and user-assigned tags as target keyphrases.
Since oftentimes the questions on StackExchange contain less information than in scientific publications, there are fewer keyphrases per data point in StackEx. Furthermore, StackExchange uses a tag recommendation system that suggests topic-relevant tags to users while submitting questions; therefore, we are more likely to see general terminology such as Linux and Java. This characteristic challenges models with respect to their ability to distill major topics of a question rather than selecting specific snippets from the text.
We report our models' performance on StackEx in Table TABREF37 . Results show INLINEFORM0 performs the best; on the absent-keyphrase generation tasks, it outperforms INLINEFORM1 by a large margin.
section_name: Generating Variable Number Keyphrases
paragraphs: One key advantage of our proposed model is the capability of predicting the number of keyphrases conditioned on the given source text. We thus conduct a set of experiments on KP20k and StackEx present keyphrase generation tasks, as shown in Table TABREF39 , to study such behavior. We adopt the self-terminating decoding strategy (Section SECREF28 ), and use both F INLINEFORM0 @ INLINEFORM1 and F INLINEFORM2 @ INLINEFORM3 (Section SECREF4 ) to evaluate.
In these experiments, we use beam search as in most Natural Language Generation (NLG) tasks, i.e., only use the top ranked prediction sequence as output. We compare the results with greedy search. Since no existing model is capable of generating variable number of keyphrases, in this subsection we only report performance on such setting from INLINEFORM0 and INLINEFORM1 .
From Table TABREF39 we observe that in the variable number generation setting, greedy search outperforms beam search consistently. This may because beam search tends to generate short and similar sequences. We can also see the resulting F INLINEFORM0 @ INLINEFORM1 scores are generally lower than results reported in previous subsections, this suggests an over-generation decoding strategy may still benefit from achieving higher recall.
section_name: Ablation Study
paragraphs: We conduct an ablation experiment to study the effects of orthogonal regularization and semantic coverage mechanism on INLINEFORM0 . As shown in Table TABREF44 , semantic coverage provides significant boost to INLINEFORM1 's performance on all datasets. Orthogonal regularization hurts performance when is solely applied to INLINEFORM2 model. Interestingly, when both components are enabled ( INLINEFORM3 ), the model outperforms INLINEFORM4 by a noticeable margin on all datasets, this suggests the two components help keyphrase generation in a synergetic way. One future direction is to apply orthogonal regularization directly on target encoder, since the regularizer can potentially diversify target representations at phrase level, which may further encourage diverse keyphrase generation in decoder.
section_name: Visualizing Diversified Generation
paragraphs: To verify our assumption that target encoding and orthogonal regularization help to boost the diversity of generated sequences, we use two metrics, one quantitative and one qualitative, to measure diversity of generation.
First, we simply calculate the average unique predictions produced by both INLINEFORM0 and INLINEFORM1 in experiments shown in Section SECREF36 . The resulting numbers are 20.38 and 89.70 for INLINEFORM2 and INLINEFORM3 respectively. Second, from the model running on the KP20k validation set, we randomly sample 2000 decoder hidden states at INLINEFORM4 steps following a delimiter ( INLINEFORM5 ) and apply an unsupervised clustering method (t-SNE BIBREF35 ) on them. From the Figure FIGREF46 we can see that hidden states sampled from INLINEFORM6 are easier to cluster while hidden states sampled from INLINEFORM7 yield one mass of vectors with no obvious distinct clusters. Results on both metrics suggest target encoding and orthogonal regularization indeed help diversifying generation of our model.
section_name: Qualitative Analysis
paragraphs: To illustrate the difference of predictions between our proposed models, we show an example chosen from the KP20k validation set in Appendix SECREF10 . In this example there are 29 ground truth phrases. Neither of the models is able to generate all of the keyphrases, but it is obvious that the predictions from INLINEFORM0 all start with “test”, while predictions from INLINEFORM1 are diverse. This to some extent verifies our assumption that without the target encoder and orthogonal regularization, decoder states following delimiters are less diverse.
section_name: Conclusion and Future Work
paragraphs: We propose a recurrent generative model that sequentially generates multiple keyphrases, with two extra modules that enhance generation diversity. We propose new metrics to evaluate keyphrase generation. Our model shows competitive performance on a set of keyphrase generation datasets, including one introduced in this work. In future work, we plan to investigate how target phrase order affects the generation behavior, and further explore set generation in an order invariant fashion.
section_name: Experiment Results on KP20k Absent Subset
paragraphs: Generating absent keyphrases on scientific publication datasets is a rather challenging problem. Existing studies often achieve seemingly good performance by measuring recall on tens and sometimes hundreds of keyphrases produced by exhaustive decoding with a large beam size — thus completely ignoring precision.
We report the models' R@10/50 scores on the absent portion of five scientific paper datasets in Table TABREF48 to be in line with previous studies.
The absent keyphrase prediction highly prefers recall-oriented models, therefore CopyRNN with beam size of 200 is innately proper for this task setting. Howerer, from the results we observe that with the help of exhaustive decoding and diverse mechanisms, INLINEFORM0 is able to perform comparably to CopyRNN model, and it generally works better for top predictions. Even though the trend of models' performance somewhat matches what we observe on the present data, we argue that it is hard to compare different models' performance on such scale. We argue that StackEx is better testbeds for absent keyphrase generation.
section_name: Implementation Details
paragraphs: Implemntation details of our proposed models are as follows. In all experiments, the word embeddings are initialized with 100-dimensional random matrices. The number of hidden units in both the encoder and decoder GRU are 150. The number of hidden units in target encoder GRU is 150. The size of vocabulary is 50,000.
The numbers of hidden units in MLPs described in Section SECREF3 are as follows. During negative sampling, we randomly sample 16 samples from the same batch, thus target encoding loss in Equation EQREF23 is a 17-way classification loss. In INLINEFORM0 , we set both the INLINEFORM1 and INLINEFORM2 in Equation EQREF27 to be 0.3. In all experiments, we use a dropout rate of 0.1.
We use Adam BIBREF36 as the step rule for optimization. The learning rate is INLINEFORM0 . The model is implemented using PyTorch BIBREF38 and OpenNMT BIBREF37 .
For exhaustive decoding, we use a beam size of 50 and a maximum sequence length of 40.
Following BIBREF0 , lowercase and stemming are performed on both the ground truth and generated keyphrases during evaluation.
We leave out 2,000 data examples as validation set for both KP20k and StackEx and use them to identify optimal checkpoints for testing. And all the scores reported in this paper are from checkpoints with best performances (F INLINEFORM0 @ INLINEFORM1 ) on validation set.
section_name: Example Output
paragraphs: See Table TABREF49 .

Questions and Answers:
question: How is keyphrase diversity measured?
answers: answer: unanswerable: no
extractive_spans: average unique predictions
illustrate the difference of predictions between our proposed models, we show an example chosen from the KP20k validation set
evidence: To verify our assumption that target encoding and orthogonal regularization help to boost the diversity of generated sequences, we use two metrics, one quantitative and one qualitative, to measure diversity of generation.
First, we simply calculate the average unique predictions produced by both INLINEFORM0 and INLINEFORM1 in experiments shown in Section SECREF36 . The resulting numbers are 20.38 and 89.70 for INLINEFORM2 and INLINEFORM3 respectively. Second, from the model running on the KP20k validation set, we randomly sample 2000 decoder hidden states at INLINEFORM4 steps following a delimiter ( INLINEFORM5 ) and apply an unsupervised clustering method (t-SNE BIBREF35 ) on them. From the Figure FIGREF46 we can see that hidden states sampled from INLINEFORM6 are easier to cluster while hidden states sampled from INLINEFORM7 yield one mass of vectors with no obvious distinct clusters. Results on both metrics suggest target encoding and orthogonal regularization indeed help diversifying generation of our model.
To illustrate the difference of predictions between our proposed models, we show an example chosen from the KP20k validation set in Appendix SECREF10 . In this example there are 29 ground truth phrases. Neither of the models is able to generate all of the keyphrases, but it is obvious that the predictions from INLINEFORM0 all start with “test”, while predictions from INLINEFORM1 are diverse. This to some extent verifies our assumption that without the target encoder and orthogonal regularization, decoder states following delimiters are less diverse.
question: How was the StackExchange dataset collected?
answers: answer: unanswerable: no
free_form_answer: they obtained computer science related topics by looking at titles and user-assigned tags
evidence: Inspired by the StackLite tag recommendation task on Kaggle, we build a new benchmark based on the public StackExchange data. We use questions with titles as source, and user-assigned tags as target keyphrases.
Since oftentimes the questions on StackExchange contain less information than in scientific publications, there are fewer keyphrases per data point in StackEx. Furthermore, StackExchange uses a tag recommendation system that suggests topic-relevant tags to users while submitting questions; therefore, we are more likely to see general terminology such as Linux and Java. This characteristic challenges models with respect to their ability to distill major topics of a question rather than selecting specific snippets from the text.
answer: unanswerable: yes
question: What does the TextWorld ACG dataset contain?
answers: answer: unanswerable: yes
question: What is the size of the StackExchange dataset?
answers: answer: unanswerable: yes
answer: unanswerable: no
free_form_answer: around 332k questions
evidence: FLOAT SELECTED: Table 1: Statistics of datasets we use in this work. Avg# and Var# indicate the mean and variance of numbers of target phrases per data point, respectively.
question: What were the baselines?
answers: answer: unanswerable: no
free_form_answer: CopyRNN (Meng et al., 2017), Multi-Task (Ye and Wang, 2018), and TG-Net (Chen et al., 2018b)
evidence: We report our model's performance on the present-keyphrase portion of the KP20k dataset in Table TABREF35 . To compare with previous works, we provide compute INLINEFORM0 and INLINEFORM1 scores. The new proposed F INLINEFORM2 @ INLINEFORM3 metric indicates consistent ranking with INLINEFORM4 for most cases. Due to its target number sensitivity, we find that its value is closer to INLINEFORM5 for KP20k and Krapivin where average target keyphrases is less and closer to INLINEFORM6 for the other three datasets.
FLOAT SELECTED: Table 2: Present keyphrase predicting performance on KP20K test set. Compared with CopyRNN (Meng et al., 2017), Multi-Task (Ye and Wang, 2018), and TG-Net (Chen et al., 2018b).
answer: unanswerable: no
extractive_spans: CopyRNN BIBREF0
KEA BIBREF4 and Maui BIBREF8
CopyRNN*
evidence: We include four non-neural extractive models and CopyRNN BIBREF0 as baselines. We use CopyRNN to denote the model reported by BIBREF0 , CopyRNN* to denote our implementation of CopyRNN based on their open sourced code. To draw fair comparison with existing study, we use the same model hyperparameter setting as used in BIBREF0 and use exhaustive decoding strategy for most experiments. KEA BIBREF4 and Maui BIBREF8 are trained on a subset of 50,000 documents from either KP20k (Table TABREF35 ) or StackEx (Table TABREF37 ) instead of all documents due to implementation limits (without fine-tuning on target dataset).
question: What two metrics are proposed?
answers: answer: unanswerable: no
extractive_spans: average unique predictions
randomly sample 2000 decoder hidden states at INLINEFORM4 steps following a delimiter ( INLINEFORM5 ) and apply an unsupervised clustering method (t-SNE BIBREF35 )
evidence: First, we simply calculate the average unique predictions produced by both INLINEFORM0 and INLINEFORM1 in experiments shown in Section SECREF36 . The resulting numbers are 20.38 and 89.70 for INLINEFORM2 and INLINEFORM3 respectively. Second, from the model running on the KP20k validation set, we randomly sample 2000 decoder hidden states at INLINEFORM4 steps following a delimiter ( INLINEFORM5 ) and apply an unsupervised clustering method (t-SNE BIBREF35 ) on them. From the Figure FIGREF46 we can see that hidden states sampled from INLINEFORM6 are easier to cluster while hidden states sampled from INLINEFORM7 yield one mass of vectors with no obvious distinct clusters. Results on both metrics suggest target encoding and orthogonal regularization indeed help diversifying generation of our model.

Figure and Table Captions:
caption: Figure 1: Overall structure of our proposed model. A represents last states of the bi-directional source encoder; B represents decoder states where target tokens are delimiters; C indicates target encoder states where input tokens are delimiters. During orthogonal regularization, all B states are used; during target encoder training, we maximize the mutual information between states A with each of the C states; red dash arrow indicates a detached path, i.e., do not back propagate through this path.
caption: Table 1: Statistics of datasets we use in this work. Avg# and Var# indicate the mean and variance of numbers of target phrases per data point, respectively.
caption: Table 2: Present keyphrase predicting performance on KP20K test set. Compared with CopyRNN (Meng et al., 2017), Multi-Task (Ye and Wang, 2018), and TG-Net (Chen et al., 2018b).
caption: Table 3: Model performance on transfer learning datasets.
caption: Table 4: Model performance on STACKEXCHANGE dataset.
caption: Table 5: Model performance on TEXTWORLD ACG dataset.
caption: Table 6: Example from KP20K validation set, predictions generated by catSeq and catSeqD models.
caption: Figure 2: t-SNE results on decoder hidden states. Upper row: catSeq; lower row: catSeqD; column k shows hidden states sampled from tokens at k steps following a delimiter.
caption: Table 7: Average number of generated keyphrase on KP20K and TEXTWORLD ACG.
caption: Table 8: Example observations and admissible commands in TEXTWORLD ACG dataset.