SlowGuess/NAACL_2021_OCR huggingface dataset:
It is also worth mentioning that when we manually check the output sequences in test set produced by ALPHA ordering, we notice that the model is actually able to retain alphabetical order among the predicted keyphrases, hinting that a Seq2Seq model might be capable of learning simple morphological dependencies even without access to any character-level representations.
Ordering in Absent KPG We report the performance of the same set of models on the absent portion of data in Figure 11, Appendix D. Although achieving relatively low  $\mathbf{R}@\mathbf{50}$  in most settings, scores produced by various orderings show clear distinctions, normalized heat maps suggest that the rankings among different orderings tend to be consistent across all testing datasets. In general, PRES-ABS produces better absent keyphrases across different model architectures. Due to the
Figure 5: A histogram showing the distribution of  $\# (\mathrm{kp}$  per document) on KP20K, MAGKP and its subsets. Data points with more than 30 keyphrases are truncated.
space limit, we encourage readers to check out Appendix D, which provides an exhaustive set of heat maps including all experiment settings and metrics discussed in this section.
5 Training with More Data
In this section, we further explore the possibility of improving KPG performance by scaling up the training data. Data size has been shown as one of the most effective factors for training language models (Raffel et al., 2019; Ott et al., 2018) but it has yet been discussed in the context of KPG.
MagKP Dataset We construct a new dataset, namely MAGKP, on the basis of Microsoft Academic Graph (Sinha et al., 2015). We filter the original MAG v1 dataset (166 million papers, multiple domains) and only keep papers in Computer Science and with at least one keyphrase. This results in 2.7 million data points  $(5\times$  larger than KP20K). This dataset remains noisy despite the stringent filtering criteria, this is because 1) the data is crawled from the web and 2) some keywords are labeled by automatic systems rather than humans. This noisy nature brings many interesting observations.
General Observations The first thing we try is to train a KPG model with both KP20K and MAGKP. During training, the two dataset are fed to the model in an alternate manner, we denote this data mixing strategy as ALT. In Figure 4, we compare models' performance when trained on both KP20K and MAGKP against solely on KP20K. We observe the extra MAGKP data brings consistent improvement across most model architecture and training paradigm variants. This suggests that
Figure 4: Comparing models trained solely with KP20K against with additional MAGKP data.
model KPG models discussed in this work can benefit from additional training data. Among all the settings,  $\mathbf{F}_1@O$  of the  $\mathrm{TRANS + One2Seq}$  is boosted by 3 points on present KPG, the resulting score outperforms other variants by a significant margin and even surpass a host of state-of-the-art models (see comparison in Appendix E). Again, the same setting obtains a 2.3 boost of  $\mathbf{R}@\mathbf{50}$  score on the absent KPG task, makes  $\mathrm{TRANS + One2Seq}$  the setting that benefits the most from extra data. In contrast, the extra MAGKP data provide only marginal improvement to RNN-based models. On present KPG,  $\mathrm{RNN + One2Seq}$  even has an  $\mathbf{F}_1@O$  drop when trained with more data.
As mentioned in §3, the RNN model is significantly lighter than TRANS. To investigate if an RNN with more parameters can benefit more from MAGKP, we conduct experiments which use a GRU with much larger hidden size (dubbed BIGRNN). Results (in Appendix E) suggest otherwise, extra training data leads to negative effect on One2One and only marginal gain on One2Seq. We thus believe the architecture difference between TRANS and RNN is the potential cause, for instance, the built-in self-attention mechanism may help TRANS models learning from noisy data.
Learning with Noisy Data To further investigate the performance boost brought by the MAGKP dataset on TRANS+One2Seq, we are curious to know which portion of the noisy data helped the most. As a naturally way to cluster the MAGKP data, we define the noisiness by the number of keyphrases per data point. As shown in Figure 5, the distribution of MAGKP (black border) covers a
Figure 6:  $\mathrm{TRANS + One2Seq}$  trained with KP20K and different subsets of MAGKP, using four data mixing strategy. Scores are averaged over all 6 test sets.
much wider spectrum on the x-axis compared to KP20K (red). Because keyphrase labels are provided by human authors, a majority of its keyphrase numbers lie in the range of [3, 6]; however, only less than  $20\%$  of the MAGKP data overlaps with this number distribution.
We thus break MAGKP down into a set of smaller subset: 1) MAGKP-LN is a considerably Less Noisy subset that contains data points that have  $3\sim 6$  phrases. 2) MAGKP-Nlarge is the Noisy subset in which all data points have more than 10 keyphrases. 3) MAGKP-Nsmall is a randomly sampled subset of MAGKP-Nlarge with the same size as MAGKP-LN.
We also define a set of data mixing strategies to compare against ALT: ONLY: models are trained solely on a single set (or subset) of data; MX: KP20K and MAGKP (or its subset) are split into shards (10k each) and they are randomly sampled during training; FT: models are pre-trained on MAGKP (or its subset) and fine-tuned on KP20K.
In Figure 6, we observe that none of the MAGKP subsets can match KP20K's performance in the ONLY setting. Because MAGKP-LN and MAGKP-Nsmall share similar data size with KP20K, this suggests the distributional shift between MAGKP and the 6 testing sets is significant. In the MX setting where KP20K is mixed with noisy data, we observe a notable performance boost compared to ONLY (yet still lower than ALT), however, we do not see clear patterns among the 4 MAGKP subsets in this setting. In the FT setting, we observe a surge in scores across all
MAGKP subsets. In present KPG, both MAGKP and MAGKP-Nlarge outperform the score achieved in the ALT setting; similarly, in absent KPG, MAGKP, MAGKP-Nlarge and MAGKP-Nsmall exceeds the ALT score. This is to our surprise that the subsets considered as noisy provide a greater performance boost, while they perform poorly if "ONLY" trained on these subsets.
To sum up, during our investigation on augmenting KP20K with the noisy MAGKP data, we obtain the best performance from a TRANS+One2Seq model that pre-trained on MAGKP and then finetuned on KP20K, and this performance has outperformed current state-of-the-art models. We conjecture that the performance gain may come from data diversity, because MAGKP contains a much wider distribution of data compared to the author keyword distribution as in KP20K. This inspires us to develop data augmentation techniques to exploit the diversity in unlabeled data.
Traditional Keyphrase Extraction Keyphrase extraction has been studied extensively for decades. A common approach is to formulate it as a two-step process. Specifically, a system first heuristically selects a set of candidate phrases from the text using some pre-defined features (Witten et al., 1999; Liu et al., 2011; Wang et al., 2016; Yang et al., 2017). Subsequently, a ranker is used to select the top ranked candidates following various criteria. The ranker can be bagged decision trees (Medelyan et al., 2009; Lopez and Romary, 2010), Multi-Layer Perceptron, Support Vector Machine (Lopez and Romary, 2010) or PageRank (Mihalcea and Tarau, 2004; Le et al., 2016; Wan and Xiao, 2008). Compared to the newly developed data driven approaches with deep neural networks, the above approaches suffer from poor performance and the need of dataset-specific heuristic design.
Neural Keyphrase Extraction On neural keyphrase extraction task, Zhang et al. (2016); Luan et al. (2017); Gollapalli et al. (2017) use sequence labeling approach; Subramanian et al. (2018) use pointer networks to select spans from source text; Sun et al. (2019) leverage graph neural networks. Despite improved over tradition approaches, the above methods do not have the capability of predicting absent keyphrases.
Meng et al. (2017) first propose the CopyRNN model, which both generates words from vocab
ulary and points to words from the source text — overcoming the barrier of predicting absent keyphrases. Following this idea, Chen et al. (2018); Zhao and Zhang (2019) leverage the attention mechanism to help reducing duplication and improving coverage. Ye and Wang (2018) propose a semi-supervised training strategy. Yuan et al. (2020) propose One2Seq, which enables a model to generate variable number of keyphrases. Chen et al. (2019b); Ye and Wang (2018); Wang et al. (2019) propose to leverage extra structure information (e.g., title, topic) to guide the generation. Chan et al. (2019) propose an RL model, Swaminathan et al. (2020) propose using GAN for KPG. Chen et al. (2019a) retrieve similar documents from training data to help producing more accurate keyphrases. Chen et al. (2020) introduce hierarchical decoding and exclusion mechanism to prevent from generating duplication. Cano and Bojar (2019) also propose to utilize more data, but their goal is to bridge KPG with summarization.
7 Conclusion and Takeaways
We present an empirical study discussing neural KPG models from various aspects. Through extensive experiments and analysis, we answer the three questions (§1). Results suggest that given a carefully chosen architecture and training strategy, a base model can perform comparable with fancy SOTA models. Further augmented with (noisy) data in the correct way, a base model can outperform SOTA models (Appendix E). We strive to provide a guideline on how to choose such architectures and training strategies, which hopefully can be proven valuable and helpful to the community.
We conclude our discussion with the following takeaways:
1. One2Seq excels at present KPG, while One2One performs better on absent KPG. See Section 3.
2. For present KPG, TRANS performs better on in-distribution data, when distribution or domain shift increase, RNN can outperform TRANS. See Section 3.
3. On absent KPG, TRANS is the clear winner. See Section 3.
4. For One2Seq, target ordering is important in greedy decoding (with PRES-ABS being an overall good choice). See Section 4.
5. The effect of target ordering tends to diminish when beam search is performed. See Section 4.
6. Large and noisy data can benefit KPG. Empirically, a decent way to leverage them is to pre-train on extra data then fine-tune on small in-domain data. See Section 5.
7. Copy mechanism helps present prediction while worsening absent performance. See Appendix C.1.
8. Larger beam width is beneficial, especially for absent KPG. However, on present KPG tasks, the benefit is diminished past a certain point and thus computational efficiency needs to be carefully considered. See Appendix C.2.
RM was supported by the Amazon Research Awards for the project "Transferable, Controllable, Applicable Keyphrase Generation". This research was partially supported by the University of Pittsburgh Center for Research Computing through the resources provided. The authors thank the anonymous NAACL reviewers for their helpful feedback and suggestions.
Erion Cano and Ondrej Bojar. 2019. Keyphrase generation: A multi-aspect survey. In 2019 25th Conference of Open Innovations Association (FRUCT), pages 85-94. IEEE.
Hou Pong Chan, Wang Chen, Lu Wang, and Irwin King. 2019. Neural keyphrase generation via reinforcement learning with adaptive rewards.
Jun Chen, Xiaoming Zhang, Yu Wu, Zhao Yan, and Zhoujun Li. 2018. Keyphrase generation with correlation constraints. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4057-4066, Brussels, Belgium. Association for Computational Linguistics.
Wang Chen, Hou Pong Chan, Piji Li, Lidong Bing, and Irwin King. 2019a. An integrated approach for keyphrase generation via exploring the power of retrieval and extraction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2846-2856, Minneapolis, Minnesota. Association for Computational Linguistics.
Wang Chen, Hou Pong Chan, Piji Li, and Irwin King. 2020. Exclusive hierarchical decoding for deep keyphrase generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.
Wang Chen, Yifan Gao, Jiani Zhang, Irwin King, and Michael R Lyu. 2019b. Title-guided encoding for keyphrase generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6268-6275.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724-1734, Doha, Qatar. Association for Computational Linguistics.
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098-4109, Brussels, Belgium. Association for Computational Linguistics.
Sujatha Das Gollapalli, Xiaoli Li, and Peng Yang. 2017. Incorporating expert knowledge into keyphrase extraction. In AAAI, pages 3180-3187. AAAI Press.
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1631-1640, Berlin, Germany. Association for Computational Linguistics.
Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. 2016. Pointing the unknown words. arXiv preprint arXiv:1603.08148.
Tho Thi Ngoc Le, Minh Le Nguyen, and Akira Shimazu. 2016. Unsupervised keyphrase extraction: Introducing new kinds of words to keyphrases. 29th Australasian Joint Conference, Hobart, TAS, Australia, December 5-8, 2016.
Zhiyuan Liu, Xinxiong Chen, Yabin Zheng, and Maosong Sun. 2011. Automatic keyphrase extraction by bridging vocabulary gap. the Fifteenth Conference on Computational Natural Language Learning.
Patrice Lopez and Laurent Romary. 2010. Humb: Automatic key term extraction from scientific articles in grobidp. the 5th International Workshop on Semantic Evaluation.
Yi Luan, Mari Ostendorf, and Hannaneh Hajishirzi. 2017. Scientific information extraction with semisupervised neural tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2641-2651, Copenhagen, Denmark. Association for Computational Linguistics.
Olena Medelyan, Eibe Frank, and Ian H. Witten. 2009. Human-competitive tagging using automatic keyphrase extraction. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 1318-1327, Singapore. Association for Computational Linguistics.
Rui Meng, Sanqiang Zhao, Shuguang Han, Daqing He, Peter Brusilovsky, and Yu Chi. 2017. Deep keyphrase generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 582-592. Association for Computational Linguistics.
Rada Mihalcea and Paul Tarau. 2004. TextRank: Bringing order into text. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 404-411, Barcelona, Spain. Association for Computational Linguistics.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. arXiv preprint arXiv:1806.00187.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June Hsu, and Kuansan Wang. 2015. An overview of microsoft academic service (mas) and applications. In Proceedings of the 24th international conference on world wide web, pages 243-246.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645-3650, Florence, Italy. Association for Computational Linguistics.
Sandeep Subramanian, Tong Wang, Xingdi Yuan, Saizheng Zhang, Adam Trischler, and Yoshua Bengio. 2018. Neural models for key phrase extraction and question generation. In Proceedings of the
Workshop on Machine Reading for Question Answering, pages 78-88, Melbourne, Australia. Association for Computational Linguistics.
Zhiqing Sun, Jian Tang, Pan Du, Zhi-Hong Deng, and Jian-Yun Nie. 2019. Divgraphpointer: A graph pointer network for extracting diverse keyphrases. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'19, page 755-764, New York, NY, USA. Association for Computing Machinery.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS'14, pages 3104-3112, Cambridge, MA, USA. MIT Press.
Avinash Swaminathan, Haimin Zhang, Debanjan Mahata, Rakesh Gosangi, Rajiv Shah, and Amanda Stent. 2020. A preliminary exploration of gans for keyphrase generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8021-8030.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30.
Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. 2016. Order matters: Sequence to sequence for sets. In International Conference on Learning Representations (ICLR).
Xiaojun Wan and Jianguo Xiao. 2008. Single document keyphrase extraction using neighborhood knowledge. AAAI.
Minmei Wang, Bo Zhao, and Yihua Huang. 2016. *Ptr: Phrase-based topical ranking for automatic keyphrase extraction in scientific publications*. 23rd International Conference, ICONIP 2016.
Yue Wang, Jing Li, Hou Pong Chan, Irwin King, Michael R. Lyu, and Shuming Shi. 2019. Topi- aware neural keyphrase generation for social media language. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2516-2526, Florence, Italy. Association for Computational Linguistics.
Ian H. Witten, Gordon W. Paynter, Eibe Frank, Carl Gutwin, and Craig G. Nevill-Manning. 1999. Kea: Practical automatic keyphrase extraction. In Proceedings of the Fourth ACM Conference on Digital Libraries, DL '99, pages 254-255, New York, NY, USA. ACM.
Zhilin Yang, Junjie Hu, Ruslan Salakhutdinov, and William W. Cohen. 2017. Semi-supervised qa with generative domain-adaptive nets. In the 55th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.
Hai Ye and Lu Wang. 2018. Semi-supervised learning for neural keyphrase generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4142-4153, Brussels, Belgium. Association for Computational Linguistics.
Xingdi Yuan, Tong Wang, Rui Meng, Khushboo Thaker, Peter Brusilovsky Daqing He, and Adam Trischler. 2020. One size does not fit all: Generating and evaluating variable number of keyphrases. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.
Qi Zhang, Yang Wang, Yeyun Gong, and Xuanjing Huang. 2016. Keyphrase extraction using deep recurrent neural networks on twitter. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 836-845. Association for Computational Linguistics.
Jing Zhao and Yuxiang Zhang. 2019. Incorporating linguistic constraints into keyphrase generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5224-5233.
- In Appendix A, we provide the formal definition of all evaluation metrics we used in this work.
- In Appendix B, we provide detailed statistics of all datasets used in this work.
- In Appendix C, we provide observation and analysis on additional factors that can affect a KPG system's performance.
- In Appendix D, show the set of heat maps that are not shown in the main content due to space limit.
- In Appendix E, we provide a complete set of numbers containing all results discussed in this work, compared with a set of SOTA models in existing literature.
- In Appendix F, we provide implementation details that helps to reproduce our experiments.
A Evaluation Metric Definition
In this section, we provide definition of the metrics we use in this work. All metrics are adopted from (Meng et al., 2017) and (Yuan et al., 2020). To make the results easy to reproduce, we simply report macro-average scores over all the data examples in a dataset (rather than removing examples that contain no present/absent phrases). Since some data examples contain no valid present/absent phrase and lead to zero scores, this causes our results can be lower than previously reported results.
Given a data example consisting of source text  $\mathcal{X}$  and a list of target keyphrases  $\mathcal{V}$ , suppose that a model predicts a list of unique keyphrases  $\hat{\mathcal{V}} = (\hat{y}_1, \dots, \hat{y}_m)$  ordered by the quality of the predictions  $\hat{y}_i$ , and that the ground truth keyphrases for the given source text is the oracle set  $\mathcal{V}$ . When only the top  $k$  predictions  $\hat{\mathcal{V}}_{:k} = (\hat{y}_1, \dots, \hat{y}_{\min(k, m)})$  are used for evaluation, precision, recall, and  $F_1$ -score are consequently conditioned on  $k$  and defined as:
$$
\mathbf {P} @ k = \frac {| \hat {\mathcal {Y}} _ {: k} \cap \mathcal {Y} |}{| \hat {\mathcal {Y}} _ {: k} |}, \qquad \mathbf {R} @ k = \frac {| \hat {\mathcal {Y}} _ {: k} \cap \mathcal {Y} |}{| \mathcal {Y} |},
$$
$$
\mathbf {F} _ {1} @ k = \frac {2 * \mathbf {P} @ k * \mathbf {R} @ k}{\mathbf {P} @ k + \mathbf {R} @ k}. \tag {1}
$$
Thus the metrics are defined as:
-  $\mathbf{F}_1@5$ :  $\mathbf{F}_1@k$  when  $k = 5$ .
-  $\mathbf{F}_1@10$ :  $\mathbf{F}_1@k$  when  $k = 10$ .
-  $\mathbf{F}_1@O$ :  $O$  denotes the number of oracle (ground truth) keyphrases. In this case,  $k = |\mathcal{V}|$ , which means for each data example, the number of predicted phrases taken for evaluation is the same as the number of ground-truth keyphrases.
-  $\mathbf{F}_1@\mathcal{M}$ :  $\mathcal{M}$  denotes the number of predicted keyphrases. In this case,  $k = |\hat{\mathcal{Y}}|$  and we simply take all the predicted phrases for evaluation without truncation.
-  $\mathbf{R}@\mathbf{50}$ :  $\mathbf{R}@\mathbf{k}$  when  $k = 50$ .
B Statistics of Datasets
We provide details of datasets used in this work. We use KP20K-train (Meng et al., 2017) and MAGKP (Sinha et al., 2015) for training keyphrase generation models, both are built on the basis of scientific publications in Computer Science domain. Nevertheless, their distributions are considerably different, e.g. MAGKP data contains 3 times more keyphrases on average than KP20K. This is because KP20K is constructed using real author keywords whereas MAGKP may contain a vast amount of keyphrases annotated by automatic systems. Detailed statistics are listed in Table 2. We also leave out certain amount of data points from KP20K for validation and testing (KP20K-VALID and KP20K-TEST).