SlowGuess/ACL_2020_OCR huggingface dataset:
-  $\mathbf{F}_1@O$ :  $O$  denotes the number of oracle (ground truth) keyphrases. In this case,  $k = |\mathcal{Y}|$ , which means for each data example, the number of predicted phrases taken for evaluation is the same as the number of ground truth keyphrases.
-  $\mathbf{F}_1@\mathcal{M}$ :  $\mathcal{M}$  denotes the number of predicted keyphrases. In this case,  $k = |\hat{\mathcal{Y}}|$  and we simply take all the predicted phrases for evaluation without truncation.
By simply extending the constant number  $k$  to different variables accordingly, both  $\mathbf{F}_1@O$  and  $\mathbf{F}_1@M$  are capable of reflecting the nature of variable number of phrases for each document, and a model can achieve the maximum  $F_1$  score of 1.0 if and only if it predicts the exact same phrases as the ground truth. Another merit of  $\mathbf{F}_1@O$  is that it is independent from model outputs, therefore we can use it to compare existing models.
5 Datasets and Experiments
In this section, we report our experiment results on multiple datasets and compare with existing models. We use CatSeq to refer to the delimiter
concatenated sequence-to-sequences model described in Section 3; CatSeqD refers to the model augmented with orthogonal regularization and semantic coverage mechanism.
To construct target sequences for training CatSeq and CatSeqD, ground truth keyphrases are sorted by their order of first occurrence in the source text. Keyphrases that do not appear in the source text are appended to the end. This order may guide the attention mechanism to attend to source positions in a smoother way. Implementation details can be found in Appendix D. As for the pre-processing and evaluation, we follow the same steps as in (Meng et al., 2017). More details are provided in Appendix E for reproducing our results.
We include a set of existing models (Meng et al., 2017; Chen et al., 2018a; Chan et al., 2019; Zhao and Zhang, 2019; Chen et al., 2019a) as baselines, they all share same behavior of abstractive keyphrase generation with our proposed model. Specially for computing existing model's scores with our proposed new metrics  $(\mathbf{F}_1@O$  and  $\mathbf{F}_1@\mathcal{M})$ , we implemented our own version of CopyRNN (Meng et al., 2017) based on their open sourced code, denoted as CopyRNN*. We also report the scores of models from Chan et al. and Chen et al. based on their publicly released outputs.
We also include a set of models that use sim
<sup>3</sup>We acknowledge that  $\mathbf{F}_1@\mathcal{O}$  scores of Chan et al. (2019) and Chen et al. (2019a) might be not completely comparable with ours. This is due to additional post-processing and filtering methods might have been applied in different work. We elaborate the data pre-processing and evaluation protocols used in this work in Appendix E.
7966
Present	Absent
Model	F1@5	F1@10	F1@O	R@10	R@50
TfIdf	8.0	8.9	5.2	-	-
TextRank	12.1	10.1	11.6	-	-
KEA	4.9	4.8	5.3	-	-
Maui	35.8	23.3	51.8	-	-
CopyRNN*	44.2	30.3	66.2	48.8	66.0
CatSeq	48.3	45.5	63.5	40.7	42.2
CatSeqD	48.7	43.9	65.6	54.8	65.7
ilar strategies but can not directly compare with. This includes four non-neural extractive models: TfIdf (Hasan and Ng, 2010), TextRank (Mihalcea and Tarau, 2004), KEA (Witten et al., 1999), and Maui (Medelyan et al., 2009); one neural extractive model (Sun et al., 2019); and two neural models that use additional data (e.g., title) (Ye and Wang, 2018; Chen et al., 2019b).
In Section 5.3, we apply the self-terminating decoding strategy. Since no existing model supports such decoding strategy, we only report results from our proposed models. They can be used for comparison in future studies.
5.1 Experiments on Scientific Publications
Our first dataset consists of a collection of scientific publication datasets, namely KP20K, INSPEC, KRAPIVIN, NUS, and SEMEVAL, that have been widely used in existing literature (Meng et al., 2017; Chen et al., 2018a; Ye and Wang, 2018; Chen et al., 2018b; Chan et al., 2019; Zhao and Zhang, 2019; Chen et al., 2019a; Sun et al., 2019). KP20K, for example, was introduced by Meng et al. (2017) and comprises more than half a million scientific publications. For each article, the abstract and title are used as the source text while the author keywords are used as target. The other four datasets contain much fewer articles, and thus used to test transferability of our model.
We report our model's performance on the present-keyphrase portion of the KP20K dataset in Table 2.4 To compare with previous works, we provide compute  $F_{1}@5$  and  $F_{1}@10$  scores. The new proposed  $\mathbf{F}_1@O$  metric indicates consistent ranking with  $F_{1}@5 / 10$  for most cases. Due to its target number sensitivity, we find that its value is closer to  $F_{1}@5$  for KP20K and KRAPIVIN where average target keyphrases is less and closer to  $F_{1}@10$  for the other three datasets.
Table 3: Model performance on STACKEx dataset.
KP20K	STACKEX
Model	F1@O	F1@M	F1@O	F1@M
Greedy Search
CatSeq	33.1	32.4	59.2	56.3
CatSeqD	33.4	33.9	59.6	59.3
Top Ranked Sequence in Beam Search
CatSeq	24.3	25.1	52.4	52.7
CatSeqD	31.9	33.4	56.5	57.0
Table 4:  $\mathbf{F}_1@ \mathcal{O}$  and  $\mathbf{F}_1@ \mathcal{M}$  when generating variable number of keyphrases (self-terminating decoding).
From the result we can see that our CatSeqD outperform existing abstractive models on most of the datasets. Our implemented CopyRNN* achieves better or comparable performance against the original model, and on NUS and SemEval the advantage is more salient.
As for the proposed models, both CatSeq and CatSeqD yield comparable results to CopyRNN, indicating that ONE2SEQ paradigm can work well as an alternative option for the keyphrase generation task. CatSeqD outperforms CatSeq on all metrics, suggesting the semantic coverage and orthogonal regularization help the model to generate higher quality keyphrases and achieve better generalizability. To our surprise, on the metric  $\mathrm{F}_1@10$  for KP20K and KRAPIVIN (average number of keyphrases is only 5), where high-recall models like CopyRNN are more favored, CatSeqD is still able to outperform ONE2ONE baselines, indicating that the proposed mechanisms for diverse generation are effective.
5.2 Experiments on The STACKEx Dataset
Inspired by the StackLite tag recommendation task on Kaggle, we build a new benchmark based on the public StackExchange data<sup>5</sup>. We use questions with titles as source, and user-assigned tags as target keyphrases. We provide details regarding our data collection in Appendix C.
Since oftentimes the questions on StackExchange contain less information than in scientific publications, there are fewer keyphrases per data point in STACKEX (statistics are shown in Table 1). Furthermore, StackExchange uses a tag recommendation system that suggests topic-relevant tags to users while submitting questions; therefore, we are more likely to see general terminology such as
<sup>4</sup>We show experiment results on absent data in Appendix B.
$^{5}$ https://archive.org/details/stackexchange, we choose 19 computer science related topics from Oct. 2017 dump.
7967
Model	KP20x	Inspec	Krapivin	NUS	SemEval
CatSeq	31.9	30.7	32.3	38.3	31.0
+ Orth. Reg.	31.1	29.3	31.0	36.5	29.5
+ Sem. Cov.	32.9	32.1	34.5	40.2	32.9
CatSeqD	35.7	33.1	37.1	40.6	35.7
Table 5: Ablation study with  $\mathbf{F}_1@\mathcal{O}$  scores on five scientific publication datasets.
Linux and Java<sup>6</sup>. This characteristic challenges models with respect to their ability to distill major topics of a question rather than selecting specific snippets from the text.
We report our models' performance on STACKEx in Table 3. Results show CatSeqD performs the best in general; on the absent-keyphrase generation tasks, it outperforms CatSeq by a large margin.
5.3 Generating Variable Number Keyphrases
One key advantage of our proposed model is the capability of predicting the number of keyphrases conditioned on the given source text. We thus conduct a set of experiments on KP20K and STACKEx present keyphrase generation tasks, as shown in Table 4, to study such behavior. We adopt the self-terminating decoding strategy (Section 3.3), and use both  $\mathbf{F}_1@O$  and  $\mathbf{F}_1@M$  (Section 4) to evaluate.
In these experiments, we use beam search as in most Natural Language Generation (NLG) tasks, i.e., only use the top ranked prediction sequence as output. We compare the results with greedy search. Since no existing model is capable of generating variable number of keyphrases, in this subsection we only report performance on such setting from CatSeq and CatSeqD.
From Table 4 we observe that in the variable number generation setting, greedy search outperforms beam search consistently. This may because beam search tends to generate short and similar sequences. We can also see the resulting  $\mathbf{F}_1@O$  scores are generally lower than results reported in previous subsections, this suggests an over-generation decoding strategy may still benefit from achieving higher recall.
We conduct an ablation experiment to study the effects of orthogonal regularization and semantic coverage mechanism on CatSeq. As shown in Table 5, semantic coverage provides significant boost to CatSeq's performance on all datasets. Orthogonal regularization hurts performance when is solely applied to CatSeq model. Interestingly, when both components are enabled (CatSeqD), the model outperforms CatSeq by a noticeable margin on all datasets, this suggests the two components help keyphrase generation in a synergistic way. One future direction is to apply orthogonal regularization directly on target encoder, since the regularizer can potentially diversify target representations at phrase level, which may further encourage diverse keyphrase generation in decoder.
6.2 Visualizing Diversified Generation
To verify our assumption that target encoding and orthogonal regularization help to boost the diversity of generated sequences, we use two metrics, one quantitative and one qualitative, to measure diversity of generation.
First, we simply calculate the average unique predicted phrases produced by both CatSeq and CatSeqD in experiments shown in Section 5.1 (beam size is 50). The resulting numbers are 20.38 and 89.70 for CatSeq and CatSeqD respectively. Second, from the model running on the KP20K validation set, we randomly sample 2000 decoder hidden states at  $k$  steps following a delimiter  $(k = 1,2,3)$  and apply an unsupervised clustering method (t-SNE (van der Maaten and Hinton, 2008)) on them. From the Figure 2 we can see that hidden states sampled from CatSeqD are easier to cluster while hidden states sampled from CatSeq yield one mass of vectors with no obvious distinct clusters. Results on both metrics suggest target encoding and orthogonal regularization indeed help diversifying generation of our model.
To illustrate the difference of predictions between our proposed models, we show an example chosen from the KP20K validation set in Appendix F. In this example there are 29 ground truth phrases. Neither of the models is able to generate all of the
One example is shown in Appendix F.
7968
Figure 2: t-SNE results on decoder hidden states. Upper row: CatSeq; lower row: CatSeqD; column  $k$  shows hidden states sampled from tokens at  $k$  steps following a delimiter.
keyphrases, but it is obvious that the predictions from CatSeq all start with "test", while predictions from CatSeqD are diverse. This to some extent verifies our assumption that without the target encoder and orthogonal regularization, decoder states following delimiters are less diverse.
We propose a recurrent generative model that sequentially generates multiple keyphrases, with two extra modules that enhance generation diversity. We propose new metrics to evaluate keyphrase generation. Our model shows competitive performance on a set of keyphrase generation datasets, including one introduced in this work. In future work, we plan to investigate how target phrase order affects the generation behavior, and further explore set generation in an order invariant fashion.
This work is supported by the National Science Foundation under grant No. 1525186. This research was also supported in part by the University of Pittsburgh Center for Research Computing through the resources provided. The authors thank the anonymous ACL reviewers for their helpful feedback and suggestions.
Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473.
Bousmalis, K., Trigeorgis, G., Silberman, N., Krishnan, D., and Erhan, D. (2016). Domain separation networks. CoRR, abs/1608.06019.
Chan, H. P., Chen, W., Wang, L., and King, I. (2019). Neural keyphrase generation via reinforcement learning with adaptive rewards. arXiv preprint arXiv:1906.04106.
Chen, J., Zhang, X., Wu, Y., Yan, Z., and Li, Z. (2018a). Keyphrase generation with correlation constraints. CoRR, abs/1808.07185.
Chen, W., Chan, H. P., Li, P., Bing, L., and King, I. (2019a). An integrated approach for keyphrase generation via exploring the power of retrieval and extraction. arXiv preprint arXiv:1904.03454.
Chen, W., Gao, Y., Zhang, J., King, I., and Lyu, M. R. (2018b). Title-guided encoding for keyphrase generation. CoRR, abs/1808.08575.
Chen, W., Gao, Y., Zhang, J., King, I., and Lyu, M. R. (2019b). Title-guided encoding for keyphrase generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6268-6275.
Cho, K., Van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014). Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078.
Du, X., Shao, J., and Cardie, C. (2017). Learning to ask: Neural question generation for reading comprehension. CoRR, abs/1705.00106.
Forster, K. I. and Bednall, E. S. (1976). Terminating and exhaustive search in lexical access. Memory & Cognition, 4(1):53-61.
Gollapalli, S. D., Li, X., and Yang, P. (2017). Incorporating expert knowledge into keyphrase extraction. In AAAI, pages 3180-3187. AAAI Press.
Gu, J., Lu, Z., Li, H., and Li, V. O. K. (2016). Incorporating copying mechanism in sequence-to-sequence learning. CoRR, abs/1603.06393.
Gulçehre, C., Ahn, S., Nallapati, R., Zhou, B., and Bengio, Y. (2016). Pointing the unknown words. CoRR, abs/1603.08148.
Hasan, K. S. and Ng, V. (2010). Conundrums in unsupervised keyphrase extraction: making sense of the state-of-the-art. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, pages 365-373. Association for Computational Linguistics.
Hjelm, R. D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., and Bengio, Y. (2018). Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670.
Kingma, D. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
7969
Klein, G., Kim, Y., Deng, Y., Senellart, J., and Rush, A. M. (2017). OpenNMT: Open-source toolkit for neural machine translation. In Proc. ACL.
Le, T. T. N., Nguyen, M. L., and Shimazu, A. (2016). Unsupervised keyphrase extraction: Introducing new kinds of words to keyphrases. 29th Australasian Joint Conference, Hobart, TAS, Australia, December 5-8, 2016.
Liu, Z., Chen, X., Zheng, Y., and Sun, M. (2011). Automatic keyphrase extraction by bridging vocabulary gap. ACL.
Logeswaran, L. and Lee, H. (2018). An efficient framework for learning sentence representations. CoRR, abs/1803.02893.
Lopez, P. and Romary, L. (2010). Humb: Automatic key term extraction from scientific articles in gro-bidp. the 5th International Workshop on Semantic Evaluation.
Luan, Y., Ostendorf, M., and Hajishirzi, H. (2017). Scientific information extraction with semi-supervised neural tagging. CoRR, abs/1708.06075.
Medelyan, O., Frank, E., and Witten, I. H. (2009). Human-competitive tagging using automatic keyphrase extraction. EMNLP.
Meng, R., Zhao, S., Han, S., He, D., Brusilovsky, P., and Chi, Y. (2017). Deep keyphrase generation. In ACL.
Mihalcea, R. and Tarau, P. (2004). Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing, pages 404-411.
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. (2017). Automatic differentiation in pytorch. In NIPS-W.
Paulus, R., Xiong, C., and Socher, R. (2017). A deep reinforced model for abstractive summarization. CoRR, abs/1705.04304.
Reddy, D. R. et al. (1977). Speech understanding systems: A summary of results of the five-year research effort. Department of Computer Science. CarnegieMell University, Pittsburgh, PA, 17.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2014). Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res.
Subramanian, S., Wang, T., Yuan, X., and Trischler, A. (2017). Neural models for key phrase detection and question generation. CoRR, abs/1706.04560.
Sun, Z., Tang, J., Du, P., Deng, Z.-H., and Nie, J.-Y. (2019). Divgraphpointer: A graph pointer network for extracting diverse keyphrases. arXiv preprint arXiv:1905.07689.
Sutskever, I., Vinyals, O., and Le, Q. V. (2014). Sequence to sequence learning with neural networks. In NIPS.
van den Oord, A., Li, Y., and Vinyals, O. (2018). Representation learning with contrastive predictive coding. CoRR, abs/1807.03748.
van der Maaten, L. and Hinton, G. (2008). Visualizing high-dimensional data using t-sne. Journal of Machine Learning Research, 9:2579-2605.
Van Zandt, T. and Townsend, J. T. (1993). Self-terminating versus exhaustive processes in rapid visual and memory search: An evaluative review. Perception & Psychophysics, 53(5):563-580.
Wan, X. and Xiao, J. (2008). Single document keyphrase extraction using neighborhood knowledge. AAAI.
Wang, M., Zhao, B., and Huang, Y. (2016).Ptr: Phrase-based topical ranking for automatic keyphrase extraction in scientific publications. ICONIP 2016.
Witten, I. H., Paynter, G. W., Frank, E., Gutwin, C., and Nevill-Manning, C. G. (1999). Kea: Practical automatic keyphrase extraction. In DL '99.
Xu, Q., Zhang, J., Qu, L., Xie, L., and Nock, R. (2018). D-page: Diverse paraphrase generation. CoRR.
Yang, Z., Hu, J., Salakhutdinov, R., and Cohen, W. W. (2017). Semi-supervised qa with generative domain-adaptive nets. In ACL.
Ye, H. and Wang, L. (2018). Semi-supervised learning for neural keyphrase generation. CoRR, abs/1808.06773.
Yuan, X., Wang, T., Gülçehre, C., Sordoni, A., Bachman, P., Subramanian, S., Zhang, S., and Trischler, A. (2017). Machine comprehension by text-to-text neural question generation. CoRR, abs/1705.02012.
Zhang, Q., Wang, Y., Gong, Y., and Huang, X. (2016). Keyphrase extraction using deep recurrent neural networks on twitter. In EMNLP.
Zhang, X. and Lapata, M. (2017). Sentence simplification with deep reinforcement learning. CoRR, abs/1703.10931.
Zhao, J. and Zhang, Y. (2019). Incorporating linguistic constraints into keyphrase generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5224-5233.
Zhou, Q., Yang, N., Wei, F., and Zhou, M. (2017). Selective encoding for abstractive sentence summarization. CoRR, abs/1704.07073.
A Sequence to Sequence Generation
A.1 The Encoder-Decoder Model
Given a source text consisting of  $N$  words  $w_{e}^{1},\ldots ,w_{e}^{N}$ , the encoder converts their corresponding embeddings  $x_{e}^{1},\ldots ,x_{e}^{N}$  into a set of  $N$  real-valued vectors  $h_e = (h_e^1,\dots,h_e^N)$  with a bidirectional GRU (Cho et al., 2014):
$$
\begin{array}{l} h _ {e, \mathrm {f w d}} ^ {t} = \mathrm {G R U} _ {e, \mathrm {f w d}} (x _ {e} ^ {t}, h _ {e, \mathrm {f w d}} ^ {t - 1}), \\ h _ {e, \mathrm {b w d}} ^ {t} = \operatorname {G R U} _ {e, \mathrm {b w d}} \left(x _ {e} ^ {t}, h _ {e, \mathrm {b w d}} ^ {t + 1}\right), \tag {6} \\ h _ {e} ^ {t} = \langle h _ {e, \mathrm {f w d}} ^ {t}, h _ {e, \mathrm {b w d}} ^ {t} \rangle . \\ \end{array}
$$
Dropout (Srivastava et al., 2014) is applied to both  $x_{e}$  and  $h_{e}$  for regularization.
The decoder is a uni-directional GRU, which generates a new state  $h_d^t$  at each time-step  $t$  from the word embedding  $x_d^t$  and the recurrent state  $h_d^{t-1}$ :
$$
h _ {d} ^ {t} = \operatorname {G R U} _ {d} \left(x _ {d} ^ {t}, h _ {d} ^ {t - 1}\right). ^ {7} \tag {7}
$$
The initial state  $h_d^0$  is derived from the final encoder state  $h_e^N$  by applying a single-layer feedforward neural net (FNN):
$$
h _ {d} ^ {0} = L _ {0} ^ {\tanh } \left(h _ {e} ^ {N}\right). \tag {8}
$$
Dropout is applied to both the embeddings  $x_{d}$  and the GRU states  $h_{d}$ .
A.2 Attentive Decoding
When generating token  $y^{t}$ , in order to better incorporate information from the source text, an attention mechanism (Bahdanau et al., 2014) is employed to infer the importance  $\alpha^{t,i}$  of each source word  $w_{e}^{i}$  given the current decoder state  $h_{d}^{t}$ . This importance is measured by an energy function with a 2-layer FNN:
$$
\operatorname {e n e r g y} \left(h _ {d} ^ {t}, h _ {e} ^ {i}\right) = L _ {1} \left(L _ {2} ^ {\tanh } \left(\left\langle h _ {d} ^ {t}, h _ {e} ^ {i} \right\rangle\right)\right). \tag {9}
$$
The output over all decoding steps  $t$  thus define a distribution over the source sequence:
$$
\alpha^ {t} = \operatorname {s o f t m a x} \left(\operatorname {e n e r g y} \left(h _ {d} ^ {t}, h _ {e}\right)\right). \tag {10}
$$
These attention scores are then used as weights for a refined representation of the source encodings, which is then concatenated to the decoder state  $h_d^t$  to derive a generative distribution  $p_a$ :
$$
p _ {a} \left(y ^ {t}\right) = L _ {3} ^ {\text {s o f t m a x}} \left(L _ {4} ^ {\operatorname {t a n h}} \left(\left\langle h _ {d} ^ {t}, \sum_ {i} \alpha^ {t, i} \cdot h _ {e} ^ {i} \right\rangle\right)\right), \tag {11}
$$
where the output size of  $L_{3}$  equals to the target vocabulary size. Subscript  $a$  indicates the abstractive nature of  $p_{a}$  since it is a distribution over a prescribed vocabulary.
A.3 Pointer Softmax
We employ the pointer softmax (Gulçehre et al., 2016) mechanism to switch between generating a token  $y^{t}$  (from a vocabulary) and pointing (to a token in the source text). Specifically, the pointer softmax module computes a scalar switch  $s^t$  at each generation time-step and uses it to interpolate the abstractive distribution  $p_a(y^t)$  over the vocabulary (see Equation 11) and the extractive distribution  $p_x(y^t) = \alpha^t$  over the source text tokens:
$$
p \left(y ^ {t}\right) = s ^ {t} \cdot p _ {a} \left(y ^ {t}\right) + \left(1 - s ^ {t}\right) \cdot p _ {x} \left(y ^ {t}\right), \tag {12}
$$
where  $s^t$  is conditioned on both the attention-weighted source representation  $\sum_{i} \alpha^{t,i} \cdot h_e^i$  and the decoder state  $h_d^t$ :
$$
s ^ {t} = L _ {5} ^ {\text {s i g m o i d}} \left(\tanh  \left(L _ {6} \left(\sum_ {i} \alpha^ {t, i} \cdot h _ {e} ^ {i}\right) + L _ {7} \left(h _ {d} ^ {t}\right)\right)\right). \tag {13}
$$
B Experiment Results on Kp20k Absent Subset
Generating absent keyphrases on scientific publication datasets is a rather challenging problem. Existing studies often achieve seemingly good performance by measuring recall on tens and sometimes hundreds of keyphrases produced by exhaustive decoding with a large beam size — thus completely ignoring precision.
We report the models' Recall@10/50 scores on the absent portion of five scientific paper datasets in Table 6 to be in line with previous studies.
The absent keyphrase prediction highly prefers recall-oriented models, therefore CopyRNN with beam size of 200 is innately proper for this task setting. However, from the results we observe that with the help of exhaustive decoding and diverse mechanisms, CatSeqD is able to perform comparably to CopyRNN model, and it generally works better for top predictions. Even though the trend of models' performance somewhat matches what we observe on the present data, we argue that it is hard to compare different models' performance on such scale. We argue that STACKEX is better testbeds for absent keyphrase generation.