SlowGuess/NAACL_2022_OCR huggingface dataset:
previous works (Meng et al., 2017; Chen et al., 2019; Yuan et al., 2018). We use the PresAbs ordering of the keyphrases as that was shown to be the most effective representation in (Meng et al., 2021). Further, we only train a single model by fine-tuning on the KP20K dataset and perform inference on all the test datasets. Similar to (Meng et al., 2021) we use a beam width of 50 for beam search and restrict our maximum generated sequence length to 40 tokens. For our evaluation we use macroaveraged F1@5 and F1@M as in (Chan et al., 2019) and (Chen et al., 2020) for both present and absent keyphrase generation. F1@M evaluates all the keyphrases predicted by the model with the groundtruth keyphrases. F1@5, as the name suggests evaluates only the first 5 keyphrases, however when there are fewer than five keyphrases, random incorrect keyphrases are appended till it reaches five predictions. (Chan et al., 2019) show that without this appending F1@M is the same as F1@5, when predictions are fewer than five. (Ye et al., 2021) also present a ONE2SET training paradigm and for a fair comparison we compare to their Transformer (ONE2SEQ) results, since we also train in the ONE2SEQ paradigm and not ONE2SET.
In Table 4 and Table 5 we see that KeyBART is the most effective pre-training method achieving SOTA on most datasets for F1@M in present and absent KG. We believe our choice of perturbation of the input during the pre-training setup makes this model robust and helps it identify and generate keyphrases more effectively. We also observe that our results for F1@5 aren't as competitive as F1@M and we believe this is because our model tends to favor predicting fewer than 5 keyphrases and thus tends to suffer from the random addition of keyphrases for F1@5. More concretely, the average predicted keyphrases per document for SemEval is
Model	Inspec	NUS	Krapivin	SemEval	KP20k
F1@5	F1@M	F1@5	F1@M	F1@5	F1@M	F1@5	F1@M	F1@5	F1@M
catSeq (Yuan et al., 2018)	22.5	26.2	32.3	39.7	26.9	35.4	24.2	28.3	29.1	36.7
catSeqTG (Chen et al., 2019)	22.9	27	32.5	39.3	28.2	36.6	24.6	29.0	29.2	36.6
catSeqTG-2RF1 (Chan et al., 2019)	25.3	30.1	37.5	43.3	30	36.9	28.7	32.9	32.1	38.6
GANMR (Swaminathan et al., 2020)	25.8	29.9	34.8	41.7	28.8	36.9	-	-	30.3	37.8
ExHiRD-h (Chen et al., 2020)	25.3	29.1	-	-	28.6	34.7	28.4	33.5	31.1	37.4
Transformer (Ye et al., 2021)	28.15	32.56	37.07	41.91	31.58	36.55	28.71	32.52	33.21	37.71
BART*	23.59	28.46	35.00	42.65	26.91	35.37	26.72	31.91	29.25	37.51
KeyBART-DOC*	24.42	29.57	31.37	39.24	24.21	32.60	24.69	30.50	28.82	37.59
KeyBART*	24.49	29.69	34.77	43.57	29.24	38.62	27.47	33.54	30.71	39.76
KeyBART* (no finetune)	30.72	36.89	18.86	21.67	18.35	20.46	20.25	25.82	12.57	15.41
Table 4: Keyphrase Generation for Present Keyphrases. SOTA is marked in Bold and our best performing models as Bold-Italicized.
Model	Inspec	NUS	Krapivin	SemEval	KP20k
F1@5	F1@M	F1@5	F1@M	F1@5	F1@M	F1@5	F1@M	F1@5	F1@M
catSeq (Yuan et al., 2018)	0.4	0.8	1.6	2.8	1.8	3.6	1.6	2.8	1.5	3.2
catSeqTG (Chen et al., 2019)	0.5	1.1	1.1	1.8	1.8	3.4	1.1	1.8	1.5	3.2
catSeqTG-2RF1 (Chan et al., 2019)	1.2	2.1	1.9	3.1	3.0	5.3	2.1	3.0	2.7	5.0
GANMR (Swaminathan et al., 2020)	1.3	1.9	2.6	3.8	4.2	5.7	-	-	3.2	4.5
ExHiRD-h (Chen et al., 2020)	1.1	2.2	-	-	2.2	4.3	1.7	2.5	1.6	3.2
Transformer (Ye et al., 2021)	1.02	1.94	2.82	4.82	3.21	6.04	2.05	2.33	2.31	4.61
BART*	1.08	1.96	1.80	2.75	2.59	4.91	1.34	1.75	1.77	3.56
KeyBART-DOC*	0.99	2.03	1.39	2.74	2.40	4.58	1.07	1.39	1.69	3.38
KeyBART*	0.95	1.81	1.23	1.90	3.09	6.08	1.96	2.65	2.03	4.26
KeyBART* (no finetune)	1.83	2.92	1.46	2.19	1.29	2.09	1.12	1.45	0.70	1.14
Table 5: Keyphrase Generation for Absent Keyphrases. SOTA is marked in Bold and our best performing models as Bold-Italicized.
Model	F1
LSTM-CRF (Lample et al., 2016)	91.0
ELMo (Peters et al., 2018)	92.2
BERT (Devlin et al., 2018)	92.8
(Akbik et al., 2019)	93.1
(Baevski et al., 2019)	93.5
LUKE (Yamada et al., 2020)	94.3
LUKE w/o entity attention	94.1
RoBERTa (Yamada et al., 2020)	92.4
RoBERTa-extended*	92.54
KBI*	92.73
KBIR*	92.97
Table 6: Named Entity Recognition results on CONLL-2003. SOTA is marked in Bold and our best performing models as Bold-Italicized.
2.51, NUS is 2.86, Krapivin is 2.86, Inspec is 3.09 and KP20k is 2.73. The Inspec dataset is anomalous where the non-finetuned model performs significantly better, demonstrating the effectiveness of the KeyBART training strategy.
4.2.3 Named Entity Recognition
We report the performance of different models for the task of NER by conducting experiments on CoNLL-2003 dataset (Sang and De Meulder, 2003).
Table 6 demonstrate that KBI and KBIR have
performance gains over RoBERTa on CoNLL-2003. With RoBERTa-extended, we see that only continued pre-training with the MLM objective results in minor gains. However, when we inspect the results for KBI and KBIR, we see consistent jumps in performance showing how both these architectures contribute in learning richer representations that directly impact NER performance. We hypothesize that KBIR is more effective at NER than KBI because the additional keyphrase replacement classification task builds richer boundary token representations making entity identification potentially easier. The results are also fairly competitive with SOTA NER models in literature despite the fact that we did not attempt modeling entities explicitly like existing SOTA model (Yamada et al., 2020).
4.2.4 Relation Extraction
The relation extraction (RE) task predicts relations among pairs of entity mentions in a text. We finetuned our models for the sentence-level relation extraction task using the popular TACRED benchmark dataset (Zhang et al., 2017b). TACRED contains more than 100,000 sentences with entities that belong to 23 different fine-grained semantic types and with 42 different relations among entities.
Model	EM	F1
BERT (Devlin et al., 2018)	84.2	91.1
XLNet (Yang et al., 2019)	89.0	94.5
ALBERT (Lan et al., 2019)	89.3	94.8
LUKE (Yamada et al., 2020)	89.8	95.0
LUKE w/o entity attention	89.2	94.7
RoBERTa (Liu et al., 2019)	88.9	94.6
RoBERTa-extended*	88.88	94.55
KBI*	88.97	94.7
KBIR*	89.04	94.75
To fine-tune our models, we modified the input sequences to mark the start and end of the subject entity with @ and the object entity with #: We use the final layer representation of the [CLS] token as the input to a multi-class classifier.
The results in the top half of Table 8 are reported from the respective papers that use various input formatting strategy. Similar to (Zhou and Chen, 2021), we also observe that a model's performance depends heavily on the formatting of the input sequence. All models in the bottom half of the table are trained with the same input format mentioned above. We observe that our KBIR model performs slightly worse than the original RoBERTa model. We also observe similar trends for KBI and RoBERTa-extended models. We conjecture that the domain shift of the pre-training corpus is responsible for the slight performance degradation.
4.2.5 Question Answering
The relation between question answering (QA) and KE has been explored to some extent in (Subramanian et al., 2018), which leverages keyphrase extraction for question generation. Motivated by their work, we evaluate our models on SQuAD v1.1 (Rajpurkar et al., 2016) dataset for the extractive question answering task. For all the models, we use a maximum sequence length of 512 with a sliding window of size 128.
Table 7 reports the F1 and Exact Match (EM) scores achieved by different model architectures on the DEV set. We observe improved performance with KBI and KBIR as compared to RoBERTa. We have an interesting observation where RoBERTa-extended performs worse than RoBERTa and we conjecture that it is because of the domain shift in the pre-training data which comprises scientific articles. On the other hand, the models trained with keyphrase pre-training objectives are fairly com
Table 7: Question Answering results on SQuAD v1.1 on the DEV set. State-of-the-art is marked in Bold and our best performing models as Bold-Italicized.
Model	F1
BERT (Zhang et al., 2019)	66.0
C-GCN (Zhang et al., 2018)	66.4
ERNIE (Zhang et al., 2019)	68.0
SpanBERT (Joshi et al., 2020)	70.8
MTB (Baldini Soares et al., 2019)	71.5
KnowBERT (Peters et al., 2019)	71.5
KEPLER (Wang et al., 2019)	71.7
K-Adapter (Wang et al., 2021)	72.0
LUKE (Yamada et al., 2020)	72.7
LUKE w/o entity attention	72.2
RoBERTa (Wang et al., 2021)	71.3
RoBERTa-extended*	70.94
KBI*	70.71
KBIR*	71.0
Table 8: Relation Extraction results on TACRED. State-of-the-art is marked in Bold and our best performing models as Bold-Italicized.
Model	R1	R2	RL
BART (Lewis et al., 2019)	44.16	21.28	40.9
BART*	42.93	20.12	39.72
KeyBART-DOC*	42.92	20.07	39.69
KeyBART*	43.10	20.26	39.90
Table 9: Summarization results on CNN/DailyMail dataset. Our best performing models are marked as Bold-Italicized.
petitive with the SOTA QA models. We explicitly include LUKE w/o entity attention since that removes the entity-aware attention module, making it slightly more comparable to our setup. We observe that KBIR outperforms it by a slim margin in F1. However, the performance is slightly lower in the EM scores. Note that our model does not yield the similar performance in EM as it does in F1 when compared to SOTA. A potential reason for this is that our model is more likely to identify keyphrases as answers.
4.2.6 Summarization
We fine-tune BART (Lewis et al., 2019), KeyBART-DOC and KeyBART on the CNN DailyMail (Hermann et al., 2015) summarization dataset (SUM). Keyphrase Generation is also considered as an extreme form of summarization and therefore, we expect to see improved performance for the summarization task. Since we were unable to reproduce the original BART scores for R1, R2 and RLSum, we used the reported hyperparameters to reproduce the results to best of our ability, accounting for minor implementation differences in framework versions. We hope this provides a more fair comparison with our model results. We do not claim SOTA for summarization models, rather want to
demonstrate that there are potential performance gains by training on a keyphrase specific objective. This is demonstrated in Table 9 where we see that the standard denoising autoencoder setup results in marginal losses. However, training with the keyphrase generation objective improves the ROUGE scores across the board when compared with BART trained on the same dataset.
We perform a qualitative analysis on the SemEval2010 dataset as it is the only common dataset between KE and KG tasks by leveraging predictions from the best performing models. We present examples in Table 10, in the Appendix, which captures the ground truth, extracted keyphrases and generated keyphrases (present and absent) for a given document from the SemEval dataset using our best performing models on the respective tasks. We observe that when the model tends to generate more keyphrases, it typically relies on the copy mechanism and hence most of the generated keyphrases are present in the text itself (Example 1). We also observe that when absent keyphrases are generated accompanied by a large number of generated keyphrases, they are usually a combination of two or more words directly present in the text such as 'user study' (Example 3). The example discusses how the authors study user behavior, potentially making 'user study' a fair prediction, however the ground truth would penalize the model if this was in the training phase. Finally, we observe more generated keyphrases when the model isn't able to identify keyphrases in text and doesn't rely heavily on the copy mechanism, but on its understanding of the text. This results in keyphrases such as 'natural language processing' (Example 2). Although the prediction is not in the ground-truth, it aligns with the mentions of 'question answering' and 'linguistics'. This demonstrates that the model is indeed able to generate meaningful absent keyphrases. However, we observe that the model is not able to learn or infer world knowledge required to produce the absent keyphrases in the ground-truth. For keyphrase extraction, we see that the model tends to tag phrases more frequently than previous models, improving recall. We hypothesize that it is due to the model having a better understanding of keyphrases in a document because of the keyphrase masking perturbation and also the KRC task.
6 Other Experiments
We also attempted to use a combination of the Wikipedia (English) dump and S2ORC (Lo et al., 2019) corpora for pre-training our models. In order to obtain keyphrases tags for data at a large scale, we employed TextRank (Mihalcea and Tarau, 2004) on each document in the corpora. We set the maximum number of keyphrases to 10 for the TextRank algorithm and considered all keyphrases tagged by TextRank. We created random splits for our dataset to generate a train and development (dev) set. However, we found that keyphrases tagged in this manner added a lot of noise to the dataset and resulted in only marginal overall gains.
To train a keyphrase specific language model, we also used a combined generative and discriminative approach as introduced in ELECTRA (Clark et al., 2020). The generative approach made the model predict the masked tokens that are part of a keyphrase. In the discriminative approach, the original sequence was perturbed by replacing the tokens of a keyphrase with another semantically unrelated keyphrase. We were unable to stabilize the training for such a setup and didn't get promising results.
We explored LMs capable of learning rich representations of keyphrases that achieve SOTA performance across multiple datasets for keyphrase extraction and generation tasks. Towards this effort, we proposed a new pre-training objective KBIR and a new training setup KeyBART. The trained LMs demonstrate their effectiveness by achieving SOTA or near SOTA performance for various downstream NLP tasks when fine-tuned on benchmark datasets spanning across multiple domains. As a next step, we would like to probe our LMs to understand them more and also gauge their effectiveness for the tasks of cross-domain keyphrase extraction and generation. We would also like to explore scaling these approaches to more datasets by revisiting more sophisticated unsupervised keyphrase tagging methods.
Alan Akbik, Tanja Bergmann, and Roland Vollgraf. 2019. Pooled contextualized embeddings for named entity recognition. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 724-728, Minneapolis, Minnesota. Association for Computational Linguistics.
Rabah Alzaidy, Cornelia Caragea, and C Lee Giles. 2019. Bi-lstm-crf sequence labeling for keyphrase extraction from scholarly documents. In *The world wide web conference*, pages 2551-2557.
Isabelle Augenstein, Mrinal Das, Sebastian Riedel, Lakshmi Vikraman, and Andrew McCallum. 2017. Semeval 2017 task 10: Scienceie-extracting keyphrases and relations from scientific publications. arXiv preprint arXiv:1704.02853.
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. CoRR, abs/1607.06450.
Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, and Michael Auli. 2019. Cloze-driven pretraining of self-attention networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5360-5369, Hong Kong, China. Association for Computational Linguistics.
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2895-2905, Florence, Italy. Association for Computational Linguistics.
Erion Cano and Ondrej Bojar. 2019a. Keyphrase generation: A multi-aspect survey. In 2019 25th Conference of Open Innovations Association (FRUCT), pages 85-94. IEEE.
Erion Cano and Ondrej Bojar. 2019b. Keyphrase generation: A text summarization struggle. arXiv preprint arXiv:1904.00110.
Erion Cano and Ondrej Bojar. 2020. Two huge title and keyword generation corpora of research articles. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 6663-6671, Marseille, France. European Language Resources Association.
Hou Pong Chan, Wang Chen, Lu Wang, and Irwin King. 2019. Neural keyphrase generation via reinforcement learning with adaptive rewards. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2163-2174, Florence, Italy. Association for Computational Linguistics.
Jun Chen, Xiaoming Zhang, Yu Wu, Zhao Yan, and Zhoujun Li. 2018. Keyphrase generation with correlation constraints. arXiv preprint arXiv:1808.07185.
Wang Chen, Hou Pong Chan, Piji Li, and Irwin King. 2020. Exclusive hierarchical decoding for deep keyphrase generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1095-1105, Online. Association for Computational Linguistics.
Wang Chen, Yifan Gao, Jiani Zhang, Irwin King, and Michael R Lyu. 2019. Title-guided encoding for keyphrase generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6268-6275.
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Wenying Duan, Xiaoxi He, Zimu Zhou, Hong Rao, and Lothar Thiele. 2021. Injecting Descriptive Meta-Information into Pre-Trained Language Models with Hypernetworks. In Proc. Interspeech 2021, pages 3216-3220.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics.
Carl Gutwin, Gordon Paynter, Ian Witten, Craig Nevill-Manning, and Eibe Frank. 1999. Improving browsing in digital libraries with keyphrase indexes. Decision Support Systems, 27(1-2):81-104.
Khaled M Hammouda, Diego N Matute, and Mohamed S Kamel. 2005. Corephrase: Keyphrase extraction for document clustering. In International workshop on machine learning and data mining in pattern recognition, pages 265-274. Springer.
Kazi Saidul Hasan and Vincent Ng. 2014. Automatic keyphrase extraction: A survey of the state of the art. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1262-1273.
Dan Hendrycks and Kevin Gimpel. 2016. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606.08415.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In NIPS.
Anette Hulth. 2003a. Improved automatic keyword extraction given more linguistic knowledge. In Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing, EMNLP '03, pages 216-223, Stroudsburg, PA, USA. Association for Computational Linguistics.
Anette Hulth. 2003b. Improved automatic keyword extraction given more linguistic knowledge. In Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing, EMNLP '03, page 216-223, USA. Association for Computational Linguistics.
Anette Hulth and Beáta Megyesi. 2006. A study on automatically extracted keywords in text categorization. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 537-544.
Steve Jones and Mark S Staveley. 1999. Phrasier: a system for interactive document retrieval using keyphrases. In Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, pages 160-167.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Su Nam Kim, Olena Medelyan, Min-Yen Kan, and Timothy Baldwin. 2010. SemEval-2010 task 5: Automatic keyphrase extraction from scientific articles. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 21-26, Uppsala, Sweden. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Mikalai Krapivin, Aliaksandr Autaeu, and Maurizio Marchese. 2009. Large dataset for keyphrases extraction. Technical report, University of Trento.
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North
American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 260-270, San Diego, California. Association for Computational Linguistics.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A lite BERT for self-supervised learning of language representations. CoRR, abs/1909.11942.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019. BART: denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. CoRR, abs/1910.13461.
Rui Liu, Zheng Lin, and Weiping Wang. 2020a. Keyphrase prediction with pre-trained language model. arXiv preprint arXiv:2004.10462.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Zhiyuan Liu, Wenyi Huang, Yabin Zheng, and Maosong Sun. 2010. Automatic keyphrase extraction via topic decomposition. In Proceedings of the 2010 conference on empirical methods in natural language processing, pages 366-376.
Zhiyuan Liu, Yankai Lin, and Maosong Sun. 2020b. Representation learning and nlp. In Representation Learning for Natural Language Processing, pages 1-11. Springer.
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel S. Weld. 2019. GORC: A large contextual citation graph of academic papers. CoRR, abs/1911.02782.
Matej Martinc, Blaž Škrlj, and Senja Pollak. 2020. Tntkid: Transformer-based neural tagger for keyword identification. arXiv preprint arXiv:2003.09166.
Rui Meng, Xingdi Yuan, Tong Wang, Sanqiang Zhao, Adam Trischler, and Daqing He. 2021. An empirical study on neural keyphrase generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4985-5007, Online. Association for Computational Linguistics.
Rui Meng, Sanqiang Zhao, Shuguang Han, Daqing He, Peter Brusilovsky, and Yu Chi. 2017. Deep keyphrase generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 582-592, Vancouver, Canada. Association for Computational Linguistics.