SlowGuess/NAACL_2022_OCR huggingface dataset:
Anette Hulth and Beáta B. Megyesi. 2006. A study on automatically extracted keywords in text categorization. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 537-544, Sydney, Australia. Association for Computational Linguistics.
Jeff Johnson, Matthijs Douze, and Herve Jégou. 2021. Billion-scale similarity search with gpus. IEEE Trans. Big Data, 7(3):535-547.
Steve Jones and Mark S. Staveley. 1999. Phrasier: A system for interactive document retrieval using keyphrases. In SIGIR '99: Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, August 15-19, 1999, Berkeley, CA, USA, pages 160-167. ACM.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for opendomain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Jihyuk Kim, Myeongho Jeong, Seungtaek Choi, and Seung-won Hwang. 2021. Structure-augmented keyphrase generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2657-2667, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Dong-Hyun Lee. 2013. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086–6096, Florence, Italy. Association for Computational Linguistics.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459-9474. Curran Associates, Inc.
Xinnian Liang, Shuangzhi Wu, Mu Li, and Zhoujun Li. 2021. Unsupervised keyphrase extraction by jointly modeling local and global context. In Proceedings of
the 2021 Conference on Empirical Methods in Natural Language Processing, pages 155-164, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Rui Liu, Zheng Lin, Peng Fu, and Weiping Wang. 2020a. Reinforced keyphrase generation with bert-based sentence scorer. In 2020 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (ISPA/BDCloud/SocialCom/SustainCom), pages 1-8. IEEE.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020b. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726-742.
Zhiyuan Liu, Xinxiong Chen, Yabin Zheng, and Maosong Sun. 2011. Automatic keyphrase extraction by bridging vocabulary gap. the Fifteenth Conference on Computational Natural Language Learning.
Yichao Luo, Yige Xu, Jiacheng Ye, Xipeng Qiu, and Qi Zhang. 2021. Keyphrase generation with fine-grained evaluation-guided reinforcement learning. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 497-507, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Rui Meng, Xingdi Yuan, Tong Wang, Sanqiang Zhao, Adam Trischler, and Daqing He. 2021. An empirical study on neural keyphrase generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4985-5007, Online. Association for Computational Linguistics.
Rui Meng, Sanqiang Zhao, Shuguang Han, Daqing He, Peter Brusilovsky, and Yu Chi. 2017. Deep keyphrase generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 582-592, Vancouver, Canada. Association for Computational Linguistics.
Rada Mihalcea and Paul Tarau. 2004. TextRank: Bringing order into text. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 404-411, Barcelona, Spain. Association for Computational Linguistics.
Funan Mu, Zhenting Yu, LiFeng Wang, Yequan Wang, Qingyu Yin, Yibo Sun, Liquun Liu, Teng Ma, Jing Tang, and Xing Zhou. 2020. Keyphrase extraction with span-based feature representations. arXiv preprint arXiv:2002.05407.
Seoyeon Park and Cornelia Caragea. 2020. Scientific keyphrase identification and classification by pretrained language models intermediate task transfer
learning. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5409-5419, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocttäschel, and Sebastian Riedel. 2021. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2523-2544, Online. Association for Computational Linguistics.
Hieu Pham, Qizhe Xie, Zihang Dai, and Quoc V. Le. 2021. Meta pseudo labels. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11552-11563.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Jingbo Shang, Jialu Liu, Meng Jiang, Xiang Ren, Clare R Voss, and Jiawei Han. 2018. Automated phrase mining from massive text corpora. IEEE Transactions on Knowledge and Data Engineering, 30(10):1825-1837.
Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June Paul Hsu, and Kuansan Wang. 2015. An overview of microsoft academic service (MAS) and applications. In Proceedings of the 24th International Conference on World Wide Web Companion, WWW 2015, Florence, Italy, May 18-22, 2015 - Companion Volume, pages 243-246. ACM.
Lucas Sterckx, Thomas Demeester, Johannes Deleu, and Chris Develder. 2015. Topical word importance for fast keyphrase extraction. In Proceedings of the 24th International Conference on World Wide Web Companion, WWW 2015, Florence, Italy, May 18-22, 2015 - Companion Volume, pages 121-122. ACM.
Avinash Swaminathan, Haimin Zhang, Debanjan Mahata, Rakesh Gosangi, Rajiv Shah, and Amanda Stent. 2020. A preliminary exploration of gans for keyphrase generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8021-8030.
Minmei Wang, Bo Zhao, and Yihua Huang. 2016. *Ptr: Phrase-based topical ranking for automatic keyphrase extraction in scientific publications*. 23rd International Conference, ICONIP 2016.
Yue Wang, Jing Li, Hou Pong Chan, Irwin King, Michael R. Lyu, and Shuming Shi. 2019. Topi- aware neural keyphrase generation for social media
language. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2516-2526, Florence, Italy. Association for Computational Linguistics.
Jason Weston, Emily Dinan, and Alexander Miller. 2018. Retrieve and refine: Improved sequence generation models for dialogue. In Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI, pages 87-92, Brussels, Belgium. Association for Computational Linguistics.
Ian H. Witten, Gordon W. Paynter, Eibe Frank, Carl Gutwin, and Craig G. Nevill-Manning. 1999a. Kea: Practical automatic keyphrase extraction. In Proceedings of the Fourth ACM Conference on Digital Libraries, DL '99, pages 254-255, New York, NY, USA. ACM.
Ian H. Witten, Gordon W. Paynter, Eibe Frank, Carl Gutwin, and Craig G. Nevill-Manning. 1999b. KEA: practical automatic keyphrase extraction. In Proceedings of the Fourth ACM conference on Digital Libraries, August 11-14, 1999, Berkeley, CA, USA, pages 254-255. ACM.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Hai Ye and Lu Wang. 2018. Semi-supervised learning for neural keyphrase generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4142-4153, Brussels, Belgium. Association for Computational Linguistics.
Jiacheng Ye, Ruijian Cai, Tao Gui, and Qi Zhang. 2021a. Heterogeneous graph neural networks for keyphrase generation. arXiv preprint arXiv:2109.04703.
Jiacheng Ye, Tao Gui, Yichao Luo, Yige Xu, and Qi Zhang. 2021b. One2Set: Generating diverse keyphrases as a set. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4598-4608, Online. Association for Computational Linguistics.
Jing Zhao and Yuxiang Zhang. 2019. Incorporating linguistic constraints into keyphrase generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5224-5233.
In the cross-lingual dense passage retriever, we use "bert-base-multilingual-cased" model (Wolf et al., 2020) to initialize the query and passage encoders and fine-tune it for 15 epochs with a batch size of 32. We share the parameters between the query encoder  $E_{Q}(\cdot)$  and the passage encoder  $E_{P}(\cdot)$  and map English and non-English passages into the same embedding space. Empirical results show the encoder with parameter sharing can perform slightly better. The positive examples are the corresponding English passages while we randomly sample 100 passages as negative examples in training. For the retrieval-augmented keyphrase generator, we fine-tune "mbart-large-cc25" (Wolf et al., 2020) for 10 epochs with Adam (Kingma and Ba, 2015) optimizer with a learning rate of 1e-4, a batch size of 8, a warm-up rate of 50 training steps. Similar to most Seq2Seq models, we train the mBART-based generation module by optimizing the negative log-likelihood loss of the ground-truth keyphrase sequence, and use beam search decoding with a beam size of 5 during inference. The number of retrieved keyphrases  $m$  for retrieval-augmented generation is a hyperparameter and is tuned on the development set. We use keyphrases from  $m = 1$  English passages for AcademicMKP dataset and  $m = 5$  for EcommerceMKP dataset. During inference, we set the maximum target sequence length as 128 and set the beam decoding size as 5. For parallel passage mining via iterative training, we continue the iterative process until the retrieval recall does not improve. The total number of iterations ( $T$  in Algo. 1) are 6 and 3 on EcommerceMKP and AcademicMKP respectively. The threshold  $\tau$  in Line 23 for Algo. 1 is set as 5.
A.2 Variants of Retrieval Targets
There exists a misalignment between the retriever and the generator model. The retriever retrieves similar passages while the generator utilizes the associated keyphrases of these passages (not the retrieved passages) as external knowledge for generation. Therefore, a good retriever does not necessarily guarantee the good quality and usefulness of these keyphrases.
We tried two different retrieval targets which might close the misalignment. Given a non-English passage, we tried to either directly retrieve English keyphrases (RAMKG-P2K) or retrieve the
Model	Retrieval Results	Generation Results
Recall@1	Recall@2	Recall@5	P	R	F1
P2P	26.00%	36.78%	50.05%	48.51	44.71	44.50
P2K	2.86%	4.70%	9.09%	46.91	43.13	42.95
P2PK	25.25%	35.45%	49.51%	48.50	44.39	44.34
Table 6: Results of RAMKG variants with different retrieval targets.
τ	0	5	10	15
Recall	62.28%	64.01%	63.67%	62.97%
Table 7: Influence of the threshold  $\tau$  on the retriever-generator self-training algorithm.
concatenated sequences of passage-keyphrase pair (RAMKG-P2PK). We find that (1) RAMKG-P2K that directly retrieves keyphrases has poor retrieval performance. This is because it is hard to capture the similarity between non-English passages and English keyphrases; (2) RAMKG-P2PK has slightly worse results than only retrieval EN passages, which implies that additionally adding keyphrases in the retrieval targets does not bring any benefit.
Results are shown in Table 6. RAMKG (P2P) is our original model which retrieves English passages given a non-english passage. Results tell us that 1) directly retrieval of keyphrases have poor retrieval performance. This is because it is hard to capture the similarity between non-english passages and english keyphrases; 2) RAMKG (P2PK) has slightly worse results than the model, which implies that additionally adding keyphrases in the retrieval targets does not bring any benefit.
A.3 Discussion on Retriever-Generator Iterative training (RGIT) Algorithm
Difference between RGIT and Self-Training. Our approach shares some similarities with self-training (Lee, 2013; Pham et al., 2021) but there are some differences. In self-training, the teacher and student models are in the same architecture and focus on the same training objectives. In our proposed retriever-generator iterative training, the retriever and generator are two different models and optimized by different objectives.
Threshold Tuning. In this section, we investigate the impact of the chosen threshold  $\tau$  (line 24) in our proposed retriever-generator iterative training. We tune the threshold (tau) on AcademicMKP and results are shown in Table 7. Results show
that tau=5 receives the best retrieval performance. Tau=0 brings more pseudo parallel passage pairs but introduces more noise. Larger tau (10/15) reduces the number of pseudo pairs, making the iterative training less effective.