SlowGuess/NAACL_2022_OCR huggingface dataset:
Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing, pages 404-411.
Thuy Dung Nguyen and Min-Yen Kan. 2007. Keyphrase extraction in scientific publications. In Asian Digital Libraries. Looking Back 10 Years and Forging New Frontiers, pages 317-326, Berlin, Heidelberg. Springer Berlin Heidelberg.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 43-54, Hong Kong, China. Association for Computational Linguistics.
Vahed Qazvinian, Dragomir Radev, and Arzucan Özgür. 2010. Citation summarization through keyphrase extraction. In Proceedings of the 23rd international conference on computational linguistics (COLING 2010), pages 895-903.
Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, pages 1-26.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad:  $100,000+$  questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.
Dhruva Sahrawat, Debanjan Mahata, Mayank Kulkarni, Haimin Zhang, Rakesh Gosangi, Amanda Stent, Agniv Sharma, Yaman Kumar, Rajiv Ratn Shah, and Roger Zimmermann. 2019. Keyphrase extraction from scholarly articles as sequence labeling using contextualized embeddings. arXiv preprint arXiv:1910.08840.
Erik F Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. arXiv preprint cs/0306050.
Tyss Santosh, Debarshi Kumar Sanyal, Plaban Kumar Bhowmick, and Partha Pratim Das. 2020. Sasaki: Syntax and semantics aware keyphrase extraction from research papers. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5372-5383.
Debarshi Kumar Sanyal, Plaban Kumar Bhowmick, Partha Pratim Das, Samiran Chattopadhyay, and TYSS Santosh. 2019. Enhancing access to scholarly publications with surrogate resources. Scientometrics, 121(2):1129-1164.
Arnav Saxena, Mudit Mangal, and Goonjan Jain. 2020. Keygames: A game theoretic approach to automatic keyphrase extraction. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2037-2048.
II Yeol Song, Robert B Allen, Zoran Obradovic, and Min Song. 2006. Keyphrase extraction-based query expansion in digital libraries. In Proceedings of the 6th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL'06), pages 202-209. IEEE.
Sandeep Subramanian, Tong Wang, Xingdi Yuan, Saizheng Zhang, Adam Trischler, and Yoshua Bengio. 2018. Neural models for key phrase extraction and question generation. In Proceedings of the Workshop on Machine Reading for Question Answering, pages 78-88, Melbourne, Australia. Association for Computational Linguistics.
Avinash Swaminathan, Haimin Zhang, Debanjan Mahata, Rakesh Gosangi, Rajiv Shah, and Amanda Stent. 2020. A preliminary exploration of gans for keyphrase generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8021-8030.
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2021. K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1405-1418, Online. Association for Computational Linguistics.
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2019. KEPLER: A unified model for knowledge embedding and pre-trained language representation. CoRR, abs/1911.06136.
Thomas Wolf, Julien Chaumont, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pierric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45.
Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. 2019. Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model. CoRR, abs/1912.09637.
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. Luke: deep contextualized entity representations with entity-aware self-attention. arXiv preprint arXiv:2010.01057.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. CoRR, abs/1906.08237.
Hai Ye and Lu Wang. 2018. Semi-supervised learning for neural keyphrase generation. arXiv preprint arXiv:1808.06773.
Jiacheng Ye, Tao Gui, Yichao Luo, Yige Xu, and Qi Zhang. 2021. One2set: Generating diverse keyphrases as a set. arXiv preprint arXiv:2105.11134.
Xingdi Yuan, Tong Wang, Rui Meng, Khushboo Thaker, Peter Brusilovsky, Daqing He, and Adam Trischler. 2018. One size does not fit all: Generating and evaluating variable number of keyphrases. arXiv preprint arXiv:1810.05241.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pages 11328-11339. PMLR.
Yong Zhang, Yang Fang, and Xiao Weidong. 2017a. Deep keyphrase generation with a convolutional sequence to sequence model. In 2017 4th International Conference on Systems and Informatics (ICSAI), pages 1477-1485. IEEE.
Yongzheng Zhang, Nur Zincir-Heywood, and Evangelos Milios. 2004. World wide web site summarization. Web Intelligence and Agent Systems: An International Journal, 2(1):39-53.
Yuhao Zhang, Peng Qi, and Christopher D. Manning. 2018. Graph convolution over pruned dependency trees improves relation extraction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2205-2215, Brussels, Belgium. Association for Computational Linguistics.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017b. Position-aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 35-45, Copenhagen, Denmark. Association for Computational Linguistics.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced language representation with informative entities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1441-1451, Florence, Italy. Association for Computational Linguistics.
Wenxuan Zhou and Muhao Chen. 2021. An improved baseline for sentence-level relation extraction. CoRR, abs/2102.01373.
8.1 Limitations and Ethical Concerns
Experiments were conducted using a private infrastructure, which has a carbon efficiency of 0.432  $\mathrm{kgCO_2eq / kWh}$ . A cumulative of 6,144 hours of computation was performed on hardware of type Tesla V100-SXM2-32GB (TDP of 300W). We calculate that the combined cost of training all these models is 796.26 KGs of  $CO_{2}$  eq. Estimations were conducted using the Machine Learning Impact calculator presented in (Lacoste et al., 2019). Given the computational cost and environmental impact we restrict the number of experiment settings and ablation studies conducted in the pretraining stage - so there may be some hyperparameters that could be further optimized but have not been realized and also more robust results to further demonstrate the effectiveness our proposed approach.
The OAGKX dataset we train on is made publicly available with a Creative Commons License 4.0 and is primarily focused on scientific documents from the Open Academic Graph. This decreases the potential for the pre-trained model to imbibe offensive content and also in not generating the same.
We also acknowledge that the current setup uses a dataset that already contains pre-tagged keyphrases similar to how Wikipedia leverages entities and our attempt at using a basic unsupervised keyphrase tagging technique did not yield much success as seen in Section 6. We believe that further research in exploring more sophisticated techniques for unsupervised keyphrase tagging would help overcome this hurdle. Further our proposed approaches should work on entities from Wikipedia or even random spans from the BookCorpus, and we encourage exploration of the same.
8.2 Pre-training Strategies
Figure 2 provides a visual representation of the various masking strategies we deploy, a more detailed description of each stage is available in the subsections below.
We train LMs in different settings and hyperparameters as listed in Table 1.
Discriminative Setting - We pre-train three language models in the discriminative setting as described below. All of them use the pre-trained
weights of RoBERTa-large as the initial weights and are trained by continuing the learning of the parameters on the OAGKX dataset using our pretraining strategies.
- RoBERTa-extended - Previous work (Gururangan et al., 2020) has shown that adding more data to pre-training a language model typically results in better downstream performance. To verify that our performance gains stem from modeling improvements and the new pre-training objectives proposed by us rather than addition of data, we extend the training of RoBERTa-large on the OAGKX corpus. We call this model RoBERTa-extended. This also ensures fair comparison of the LMs trained by us using our pre-training objectives with that of RoBERTa.
- KBI - During the pre-training of the LM with the KBI objective, we employ both token masking and keyphrase masking strategies as shown in Figure 2 and explained in Section 2.1. We randomly mask  $15\%$  of the tokens that are not included in keyphrase spans. We additionally mask  $20\%$  of the keyphrase spans with a single [MASK] token. We restrict the maximum number of tokens for a keyphrase mask span to 10, based on the average keyphrase length reported in (Cano and Bojar, 2020).
- KBIR - While pre-training the LM with the KBIR objective we employ  $5\%$  token masking, in line with the findings reported in (Xiong et al., 2019) and  $20\%$  of keyphrases are masked through keyphrase masking, with a maximum possible span size of 10 as in the KBI LM. Additionally, we replace  $40\%$  of the non-masked keyphrases with randomly sampled keyphrases from the keyphrase universe as explained in Section 2.2. We restrict the maximum number of keyphrases to be replaced to no more than 20, restricted by the computational complexity of the problem. Figure 1 shows the final architecture, with a multi-task learning objective trained with a weighted combined loss.
Generative Setting - In the generative setting we pre-train two language models as described below. In both the models we continue the training of the
<sup>6</sup>https://huggingface.co/roberta-large
Text: Dystrophic epidermolysis bullosa is a rare disease characterized by widespread blistering of the skin and mucous membranes, which may ultimately prompt limb amputation. ... A large number of skin sarcomas were excised over the 15-year period of prosthesis use. Two falls have resulted in limb fractures. ...
Keyphrases: [Dystrophic epidermolysis bullosa, rare disease, mucous membranes, limb amputation, skin sarcomas, prosthesis use, limb fractures]
Figure 2: Pre-training Strategies, the keyphrase present in the text are highlighted in teal and every perturbation in the form of a random MASK is represented in red, keyphrase MASK is represented in blue and Keyphrase Replacement is represented in orange.
weights of BART-large<sup>7</sup> on our corpus using our pre-training strategies.
- KeyBART - We perform token masking, keyphrase masking and keyphrase replacement with same masking hyperparameters as KBIR on the input text and pre-train the model to predict the original keyphrases in Catseq format following the setup explained in Section 3.
- KeyBART-DOC - This setup uses the same input denoising settings as KeyBART, with the only difference in the output, where KeyBART generates the keyphrases associated with the document in Catseq format, whereas KeyBART-DOC similar to BART generates the original denoised input.
We use the exact same data in all the pre-training setups as explained above. We increase the number of steps while decreasing the batch size such that all the models see the data the same number of times (i.e., 2 epochs). The batch size is only reduced to accommodate increases in memory usage in the model pre-training.
<sup>7</sup>https://huggingface.co/facebook/bart-large
Input Text: On The Complexity of Combinatorial Auctions : Structured Item Graphs and Hypertree Decompositions. The winner determination problem in combinatorial auctions is the problem of determining the allocation of the items among the bidders that maximizes the sum of the accepted bid prices. While this problem is in general NPhard, it is known to be feasible in polynomial time on those instances whose associated item graphs have bounded treewidth called structured item graphs. Formally, an item graph is a graph whose nodes are in one-to-one correspondence with items, and edges are such that for any bid, the items occurring in it induce a connected subgraph. Note that many item graphs might be associated with a given combinatorial auction, depending on the edges selected for guaranteeing the connectedness. In fact, the tractability of determining whether a structured item graph of a fixed treewidth exists and if so, computing one was left as a crucial open problem. In this paper, we solve this problem by proving that the existence of a structured item graph is computationally intractable, even for treewidth 3. Motivated by this bad news, we investigate different kinds of structural requirements that can be used to isolate tractable classes of combinatorial auctions. We show that the notion of hypertree decomposition, a recently introduced measure of hypergraph cyclicity, turns out to be most useful here. Indeed, we show that the winner determination problem is solvable in polynomial time on instances whose bidder interactions can be represented with dual hypergraphs having bounded hypertree width. Even more surprisingly, we show that the class of tractable instances identified by means of our approach properly contains the class of instances having a structured item graph.
Extracted Keyphrases: [combinatorial auctions]; [structured item graphs]; [hypertree decompositions]; [item graphs]; [treewidth]; [bidders]; [hose nodes]; [hypertree cyclicity]; [polynomial time]
Generated Keyphrases: [combinatorial auctions]; [structured item graphs]; [treewidth]; [hypergraphs]; [hypertree decompositions]
Ground Truth: [hypergraph]; [structured item graph]; [polynomial time]; [combinatorial auction]; [fixed treewidth]; [accepted bid price]; [hypertree decomposition]; structured item graph complexity; simplification of the primal graph; hypertree based decomposition method; hypergraph hg; the primal graph simplification; well known mechanism for resource and task allocation; complexity of structured item graph;
Input Text: Interesting Nuggets and Their Impact on Definitional Question Answering. Current approaches to identifying definitional sentences in the context of Question Answering mainly involve the use of linguistic or syntactic patterns to identify informative nuggets. This is insufficient as they do not address the novelty factor that a definitional nugget must also possess. This paper proposes to address the deficiency by building a Human Interest Model from external knowledge. It is hoped that such a model will allow the computation of human interest in the sentence with respect to the topic. We compare and contrast our model with current definitional question answering models to show that interestingness plays an important factor in definitional question answering.
Extracted Keyphrases: [interesting nuggets]; [definitional sentences]; [question answering]; [nuggets]; [novelty]; [definitional nuggets]; [human interest model]; [human interest]
Generated Keyphrases: [definitional question answering]; natural language processing
Ground Truth: [human interest]; [use of linguistic]; [interesting nugget]; [definitional question answer]; [informative nugget]; [interest]; [computation of human interest]; sentence fragment; unique quality; manual labor; news corpus; baseline system; human interest computation; human reader; linguistic use; question topic; external knowledge; surprise factor; lexical pattern
Input Text: The Influence of Caption Features on Clickthrough Patterns in Web Search. Web search engines present lists of captions, comprising title, snippet, and URL, to help users decide which search results to visit. Understanding the influence of features of these captions on Web search behavior may help validate algorithms and guidelines for their improved generation. In this paper we develop a methodology to use clickthrough logs from a commercial search engine to study user behavior when interacting with search result captions. The findings of our study suggest that relatively simple caption features such as the presence of all terms query terms, the readability of the snippet, and the length of the URL shown in the caption, can significantly influence users' Web search behavior.
Extracted Keyphrases: [influence]; [caption features]; [clickthrough patterns]; [web search]; [snippet]; [methodology]; [clickthrough logs]
Generated Keyphrases: [web search]; [clickthrough]; [captions]; user study
Ground Truth: [clickthrough pattern]; [snippet]; [web search behavior]; [web search]; [caption feature]; summarization; extractive summarization; significant word; query log; human factor; clickthrough inversion; query term match; query re formulation
Table 10: Sample keyphrases extracted by KBI-REP-CRF and generated by KeyBART on the SemEval-2010 dataset. Present keyphrases are marked with square brackets.