SlowGuess/NAACL_2022_OCR huggingface dataset:
After  $T$  iterations, we train the retriever on the pseudo data  $D_{\text{PSEUDO}}^T$  and fine-tune it on the parallel data  $D_{\text{PAR}}$ . Then we treat it as our final retriever and train the generation model in Sec. 4.2.
EcommerceMKP Dataset is collected from a popular E-commerce shopping platform. There are four languages we consider for building EcommerceMKP: German (DE), French (FR), Spanish (ES) and Italian (IT). The title, product description, and bullet description provided by manufacturers are concatenated and treated as source input. The keyphrases of each product are selected from search queries under the following protocol. First, given a product, we only keep search queries that lead to purchases and treat them as effective queries. Then phrases are chunked from these effective queries using AutoPhrase (Shang et al., 2018) and further ranked by their frequency. Our assumption is: the more times a phrase appears in effective search queries of a product, the more important a phrase is. Finally a threshold is set to filter out unimportant phrases. Under this protocol, we receive 73k examples over four languages. The statistics are shown in Table 1.
We collect the passages and keyphrases under the same protocol for the English (EN) dataset and name it as EcommerceMKP-EN. In total the English dataset contains 3 million passage-keyphrases pairs. To obtain the parallel passage pairs for training the cross-lingual dense passage retriever, we pair the product descriptions in different languages according to the product identification information. We select a total of 1,247 parallel passages from EcommerceMKP training set which include 480 passages for DE-EN, 244 for ES-EN, 340 for FR-EN, and 183 for IT-EN. Besides, we keep 1,000 parallel passage pairs in the DEV set of EcommerceMKP to evaluate the performance of retrieval and bi-text mining.
AcademicMKP Dataset is collected from the academic domain. We take the title and abstract of each paper as the source text and author-provided keywords as the target output. All papers are sampled from Microsoft Academic Graph (Sinha et al., 2015), a web-scale academic entity graph that contains multiple types of scholarly entities and relationships: field of study, author, institution, abstract, venue, and keywords. We use Spacy (https://spacy.io/) to detect the language of abstracts and keyphrases, and choose two languages Chinese (ZH) and Korean (KO) to construct the AcademicMKP dataset. Since Microsoft Academic Graph (MAG) is automatically crawled and
Model	German (DE)	Spanish (ES)	French (FR)	Italian (IT)	Average
P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Unsupervised Statistical Keyphrase Extraction
KP-Miner	9.13	2.35	3.34	14.56	4.59	6.22	7.78	2.76	3.62	22.51	7.71	10.31	11.80	3.69	5.01
YAKE	2.31	26.54	4.17	3.26	36.86	5.89	2.47	27.29	4.43	3.87	50.12	7.10	2.77	32.24	5.01
Unsupervised Graph-based Keyphrase Extraction
TextRank	5.77	9.00	6.39	7.15	11.59	7.97	5.45	8.33	5.94	8.18	15.27	9.85	6.31	10.25	7.09
TopicalPageRank	3.59	14.10	5.34	5.07	22.08	7.74	3.75	15.49	5.65	5.22	25.43	8.26	4.16	17.71	6.33
PositionRank	5.24	11.67	6.90	7.86	19.36	10.75	5.56	13.50	7.56	7.98	21.46	11.34	6.24	15.12	8.49
MultipartiteRank	5.09	8.76	6.15	7.56	14.25	9.46	5.31	9.86	6.59	7.72	15.93	10.12	6.02	11.19	7.50
Supervised Feature-based Keyphrase Extraction
Kea	9.64	17.73	11.92	12.69	24.64	16.11	8.81	16.76	11.05	14.16	30.18	18.81	10.68	20.65	13.52
Neural-based Supervised Keyphrase Generation
CopyRNN	13.48	7.59	9.08	21.11	12.40	14.78	14.79	8.48	10.04	34.66	20.07	24.28	18.45	10.61	12.69
Transformer	29.92	25.40	26.22	34.01	30.57	30.83	29.17	24.03	25.18	44.05	43.34	42.56	32.60	28.68	29.25
mBART (monolingual)	44.91	39.59	40.04	47.70	44.79	44.17	42.64	36.81	37.72	57.98	58.16	56.25	46.76	42.58	42.60
mBART (multilingual)	45.78	40.93	41.09	48.43	44.99	44.57	43.21	38.40	38.72	60.37	58.91	57.87	47.75	43.68	43.60
mBART + EN Joint Train	45.91	39.90	40.64	49.27	43.92	44.52	43.21	37.48	38.26	59.21	57.03	56.42	47.79	42.55	43.08
mBART + EN Pretrain	45.77	40.76	41.05	48.34	44.94	44.58	42.96	38.10	38.45	60.24	58.51	57.59	47.64	43.46	43.47
RAMKG (Ours)	46.88	41.90	42.14	49.35	45.89	45.49	43.79	39.48	39.61	60.89	59.51	58.43	48.59	44.61	44.50
RAMKG + RGIT (Ours)	48.11	43.05	43.30	50.54	47.04	46.64	45.07	40.77	40.86	62.35	60.75	59.87	49.86	45.81	45.73
Table 2: Main results on the EcommerceMKP dataset. The best results are in bold. (RGIT: Iterative-training)
Model	Chinese (ZH)	Korean (KO)	Average
P	R	F1	P	R	F1	P	R	F1
mBART (mono.)	32.52	31.50	31.50	24.57	26.46	24.96	29.26	29.43	28.81
mBART (multi.)	32.48	32.27	31.85	27.03	26.93	26.44	30.25	30.08	29.63
mBART + Joint	31.10	30.38	30.23	27.36	26.85	26.58	29.57	28.93	28.73
mBART + Pretrain	32.72	29.77	30.66	27.56	25.30	27.56	30.60	27.94	28.68
RAMKG (Ours)	33.40	32.66	32.45	28.30	28.17	27.68	31.31	30.82	30.49
RAMKG + RGIT (Ours)	34.38	33.05	33.15	29.33	27.87	28.00	32.31	30.92	31.04
Table 3: Results on AcademicMKP (mono: monolingual, multi: multilingual, Joint: EN Joint Train, Pretrain: EN Pretrain, RGIT: Iterative-training).
constructed, we find some of its data is extremely noisy. For example, keyphrases might be missing or contain incorrect information such as titles, author names, and publication venues. Some of the abstracts are incomplete. Therefore, we hire three annotators to manually examine the samples from MAG dataset. Data examples with incomplete abstracts are removed. We further manually verify the metadata of all examples and correct their keyphrase information if needed. Finally, 2,693 high-quality data examples of scientific papers in the computer science domain are collected to constitute AcademicMKP.
Besides the multilingual AcademicMKP dataset, we use KP20K (Meng et al., 2017) as the English data for retrieval-augmented generation. KP20K has 560k abstract-keyphrases pairs collected from various online digital libraries in computer science domain. The threshold is set as 1.03 for passage mining and we receive 841 parallel passage pairs from AcademicMKP training set, in which 433 ZH-EN passage pairs and 384 for KO-EN.
6.1 Evaluation Metrics
Keyphrase Generation. Let the ground truth keyphrases be  $Y: k_1, k_2, \ldots, k_n$  and the predicted keyphrases be  $\tilde{Y}: \tilde{k}_1, \tilde{k}_2, \ldots, \tilde{k}_M$ , we compute the precision  $(P@M)$ , recall  $(R@M)$  and F-score  $(F_1@M)$  between  $Y$  and  $\tilde{Y}$  as  $P@M = \frac{|Y \cap \tilde{Y}|}{|\tilde{Y}|}$ ,  $R@M = \frac{|Y \cap \tilde{Y}|}{|Y|}$ ,  $F_1@M = \frac{2 \times P \times R}{P + R}$ , where  $|Y|$  denotes the number of keyphrases in the gold set  $Y$ . We only consider exact match of two keyphrases (with some post-processing such as lowercase) for  $|Y \cap \tilde{Y}|$ . Then the average are computed for all languages in the test set.
Passage Retrieval. The quantity of retrieved English passages directly influences how much external knowledge could be utilized for keyphrase generation. Therefore, we evaluate the top-k recall (k=1,2,5,10,20) on the DEV set for evaluating retrieval performance.
6.2 Baselines and Ablations
We consider following baselines and ablations: 1) Unsupervised Statistical Keyphrase Extraction: KP-Miner (El-Beltagy and Rafea, 2010), YAKE (Campos et al., 2020); 2) Unsupervised Graph-based Keyphrase Extraction: TextRank (Mihalcea and Tarau, 2004), TopicalPageRank (Sterckx et al., 2015), PositionRank (Florescu and Caragea, 2017), MultipartiteRank (Boudin, 2018); 3) Supervised Feature-based Keyphrase Extraction: KEA (Witten et al., 1999b); 4) Neural
Supervised Keyphrase Generation: CopyRNN, Transformer; 5) mBART (monolingual): separately trained 6 mBART models on each language; 6) mBART (multilingual): a single mBART model on all languages; 7) mBART + EN Joint Train: a mBART model jointly trained on the multilingual data and English data (KP20K (Meng et al., 2017) for AcademicMKP; EcommerceMKPEN for EcommerceMKP). 8) mBART + EN Pretrain: a mBART firstly pre-trained on the English data and then fine-tuned on the multilingual data. 9) RAMKG (Ours): The Retrieval-Augmented Multilingual Keyphrase Generation model (Sec. 4.1 & 4.2). 10) RAMKG + RGIT (Ours): RAMKG improved with retriever-generator iterative training (RGIT) (Sec. 4.3).
7 Results and Analyses
Main results are shown in Table 2 & 3 for EcommerceMKP and AcademicMKP respectively, and we make the following observations:
- The unsupervised approaches, both statistical-based and graph-based, have robust results across all languages. PositionRank performs the best among all unsupervised approaches.
- The supervised approaches consistently outperform unsupervised approaches. The feature-based approach KEA receives a high recall by predicting more keyphrases while the CopyRNN receives a high precision. Different from the results on English keyphrase generation where Transformer and CopyRNN are comparable, the Transformer beats the CopyRNN by a large margin in the multilingual scenario.
- We observe that jointly training on all languages (mBART multilingual) receives better results than separately training on each language (mBART monolingual). This implies the ability of locating and summarizing key information is transferable across languages.
- Comparing different approaches using external large-scale English data, we find that our proposed RAMKG outperforms both "EN Joint Train" and "EN Pretrain". This is because the retrieval-augmented approach provides auxiliary knowledge information as part of the input to the generation module, while the other two variants have to "infuse" the knowledge learned from English data to model parameters. Moreover, "EN Joint Train" and "EN Pretrain" have
Recall @ Top K	1	2	5	10	20
RAMKG	26.4%	36.8%	50.1%	59.2%	67.3%
RAMKG + RGIT	45.8%	58.3%	72.4%	79.5%	85.2%
Table 4: Retrieval recall on EcommerceMKP DEV set.
Iteration	1	2	3	4	5	6
# Pseudo Passages	20,288	21,402	19,942	21,241	22,343	21,557
Label Accuracy %	28.0%	37.9%	40.5%	44.2%	45.1%	47.0%
Table 5: Number of pseudo parallel passages and their accuracy on EcommerceMKP DEV set in different iterations of parallel passages mining.
no positive effect on AcademicMKP dataset (Table 3). Compared with multilingual and English data are from the same website, there is still a domain gap between papers (multilingual) in AcademicMKP and papers (English) in KP20K.
- The retriever-generator self-training (RAMKG + Iter) alleviates the data scarcity issue with the help of stronger retriever: since the retriever can find more relevant English keyphrases, it leads to a general improvement on keyphrase performance across languages.
7.2 Effect of Iterative Training
Retrieval Results We investigate the effect of retriever-generator iterative training by comparing the retrieval recall for models trained w/o and w/the mined pseudo parallel passage pairs. Results on the DEV set of EcommerceMKP are shown in the Table 4. With additional mined pseudo parallel passage pairs, the retriever improves the Recall@5 from  $50.1\%$  to  $72.4\%$ . And therefore, the better retrieved English keyphrases lead to a better generation performance (44.50 vs. 45.73 in Table 2).
Quantity and Quality of Pseudo Parallel Passages We show the quantity and quality of mined pseudo parallel pairs in Table 5. After each iteration of passage mining, our algorithm can consistently find around 20k passage pairs from EcommerceMKP training set, which are nearly 20 times of the initial data. To assess the quality of mined passage pairs, we examine the label accuracy using the 1,000 parallel passage pairs from the DEV set of EcommerceMKP. Results in Table 5 show that while the passage mining finds a similar number of pseudo passage pairs, the labelling accuracy does increase from  $28.0\%$  to  $47.0\%$ . This is because the better pseudo parallel data improves the retriever, and the stronger retriever results in a better generator, which in turn leads to more relevant passages.
Figure 3: Performance of passage retrieval and keyphrase generation on EcommerceMKP, with different number of initial parallel data for iterative training.
Product Description (German): Steiff 113437 Soft Cuddly Friends Honey Teddybar, grau, 38 cm. Bereits der Name des Soft Cuddly Friends Honey Teddybar sagt es schon aus: der 38 cm große Freund mit seinem honigsüben Lächeln begestert alle Kinderherzen ...
(Translation in English): Steiff 113437 Soft Cuddly Friends Honey teddy bear, gray,  $38~\mathrm{cm}$ . The name of the Soft Cuddly Friends Honey Teddy bear already says it all: the  $38~\mathrm{cm}$  tall friend with his honey-sweet smile delights all children's hearts ...
Gold Keyphrases (German): steiff kuscheltier; steiff teddy; soft cuddly friend; steiff; baer; grau.
(Translation in English): steiff cuddly toy; steiff teddy; soft cuddly friend; steiff; bear; grey.
Retrieved English Keyphrases: steiff teddy bear; teddy bear; my first; grey; honey; sweetheart; steiff bear; pink; vintage; steiff stuffed animal; steiff; terry; soft; jammy.
Predicted Keyphrases (German): steiff kuscheltier; steiff teddy; soft cuddly friend; steiff; baer; grau; jammy.
(Translation in English): steiff cuddly toy; steiff teddy; soft cuddly friend; steiff; bear; grey; jammy.
Figure 4: Case study on the EcommerceMKP dataset. The present keyphrases (keyphrases shown in the description) are in bold while absent keyphrases are in italics. Correct predictions are in green while wrong predictions are in red.
Initializing with Different Amount of Parallel
Data To investigate the impact of initial parallel passage on the training, we conduct experiments by varying the number of parallel passage pairs on EcommerceMKP, from 1.2k (default setting) to 6k instances. We compare the single-round training (i.e., training with initial data) and iterative training after six rounds (in which round we generally obtain the best retrieval recall), on both passage retrieval  $(\mathbf{R}_1\& \mathbf{R}_6)$  and keyphrase generation  $(\mathbf{G}_1$  &  $\mathbf{G}_6)$ . Results are shown in Fig. 3. We observe that (1) the score of iterative training consistently increase when more annotated parallel data is available; (2) our iterative training demonstrates great robustness with limited parallel data (e.g. 1.2k pairs), while the benefit gradually diminishes while more parallel data becomes available.
7.3 Case Study
Fig. 4 exhibits an example of our model's prediction. Given a product description in German, the model retrieves several English keyphrases and generates keyphrases in German accurately (translations in English are also provided). Through this example, we find that the retrieved English keyphrases do provide certain useful information such as "steiff teddy bear", "grey" and "soft", while it also brings some noise such as "my first", "sweetheart" and "vintage". Although there is a wrong prediction "jimmy" caused by the retrieved English keyphrases, the improvement in results shows that the benefits of retrieved knowledge outweigh the noise it introduces. Moreover, the retrieved keyphrases are only regarded as a supplement to the original passage, and the generator can automatically focus on the informative parts from both inputs through self-attention. Our retrieval-augmented multilingual keyphrase generation can tolerate some noise from the retrieved English keyphrases and predict better keyphrases based on these external knowledge.
In this study, we investigate a novel task setting - multilingual keyphrase generation - and contribute two new multilingual keyphrase generation datasets covering multiple domains and languages. Furthermore, we propose a retrieval-augmented multilingual keyphrase generation framework with retriever-generator iterative training. Results show that the proposed approach outperforms a wide range of baselines.
Wasi Ahmad, Xiao Bai, Soomin Lee, and Kai-Wei Chang. 2021. Select, extract and generate: Neural keyphrase generation with layer-wise coverage attention. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1389–1404, Online. Association for Computational Linguistics.
Mikel Artetxe and Holger Schwenk. 2019. Margin-based parallel corpus mining with multilingual sentence embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3197-3203, Florence, Italy. Association for Computational Linguistics.
Florian Boudin. 2018. Unsupervised keyphrase extraction with multipartite graphs. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 667-672, New Orleans, Louisiana. Association for Computational Linguistics.
Deng Cai, Yan Wang, Huayang Li, Wai Lam, and Lemao Liu. 2021. Neural machine translation with monolingual translation memory. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7307-7318, Online. Association for Computational Linguistics.
Ricardo Campos, Vitor Mangaravite, Arian Pasquali, Alipio Jorge, Celia Nunes, and Adam Jatowt. 2020. *Yake!* keyword extraction from single documents using multiple local features. *Inf. Sci.*, 509:257-289.
Hou Pong Chan, Wang Chen, Lu Wang, and Irwin King. 2019. Neural keyphrase generation via reinforcement learning with adaptive rewards. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2163-2174, Florence, Italy. Association for Computational Linguistics.
Jun Chen, Xiaoming Zhang, Yu Wu, Zhao Yan, and Zhoujun Li. 2018. Keyphrase generation with correlation constraints. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4057-4066, Brussels, Belgium. Association for Computational Linguistics.
Wang Chen, Hou Pong Chan, Piji Li, Lidong Bing, and Irwin King. 2019a. An integrated approach for keyphrase generation via exploring the power of retrieval and extraction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2846-2856, Minneapolis, Minnesota. Association for Computational Linguistics.
Wang Chen, Hou Pong Chan, Piji Li, and Irwin King. 2020. Exclusive hierarchical decoding for deep keyphrase generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1095-1105, Online. Association for Computational Linguistics.
Wang Chen, Yifan Gao, Jiani Zhang, Irwin King, and Michael R. Lyu. 2019b. Title-guided encoding for keyphrase generation. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 6268-6275. AAAI Press.
Rajarshi Das, Manzil Zaheer, Dung Thai, Ameya Godbole, Ethan Perez, Jay Yoon Lee, Lizhen Tan, Lazaros
Polymenakos, and Andrew McCallum. 2021. Case-based reasoning for natural language queries over knowledge bases. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9594-9611, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Kushal Dave, Steve Lawrence, and David M. Pennock. 2003. Mining the peanut gallery: opinion extraction and semantic classification of product reviews. In Proceedings of the Twelfth International World Wide Web Conference, WWW 2003, Budapest, Hungary, May 20-24, 2003, pages 519-528. ACM.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Samhaa R. El-Beltagy and Ahmed Rafea. 2010. KPminer: Participation in SemEval-2. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 190-193, Uppsala, Sweden. Association for Computational Linguistics.
Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020. Language-agnostic BERT sentence embedding. CoRR, abs/2007.01852.
Corina Florescu and Cornelia Caragea. 2017. PositionRank: An unsupervised approach to keyphrase extraction from scholarly documents. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1105-1115, Vancouver, Canada. Association for Computational Linguistics.
Eibe Frank, Gordon W. Paynter, Ian H. Witten, Carl Gutwin, and Craig G. Nevill-Manning. 1999. Domain-specific keyphrase extraction. In Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence, IJCAI 99, Stockholm, Sweden, July 31 - August 6, 1999. 2 Volumes, 1450 pages, pages 668-673. Morgan Kaufmann.
Ygor Gallina, Florian Boudin, and Beatrice Daille. 2019. KPTimes: A large-scale dataset for keyphrase generation on news documents. In Proceedings of the 12th International Conference on Natural Language Generation, pages 130-135, Tokyo, Japan. Association for Computational Linguistics.
Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor O. K. Li. 2018. Search engine guided neural machine translation. In AAAI.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. REALM: Retrievalaugmented language model pre-training. International Conference on Machine Learning (ICML).