SlowGuess/NAACL_2022_OCR huggingface dataset:
Retrieval-Augmented Multilingual Keyphrase Generation with Retriever-Generator Iterative Training*
Yifan Gao $^{1}$ , Qingyu Yin $^{1\ddagger}$ , Zheng Li $^{1\ddagger}$ , Rui Meng $^{2}$ , Tong Zhao $^{1}$ , Bing Yin $^{1}$ , Irwin King $^{3}$ , Michael R. Lyu $^{3}$
$^{1}$ Amazon Inc.  $^{2}$ Salesforce Research  $^{3}$ Chinese University of Hong Kong  $^{1}$  {yifangao, qingyy, amzzhe}  $@$  amazon.com  $^{2}$ ruimeng@salesforce.com
Keyphrase generation is the task of automatically predicting keyphrases given a piece of long text. Despite its recent flourishing, keyphrase generation on non-English languages haven't been vastly investigated. In this paper, we call attention to a new setting named multilingual keyphrase generation and we contribute two new datasets, EcommerceMKP and AcademicMKP, covering six languages. Technically, we propose a retrieval-augmented method for multilingual keyphrase generation to mitigate the data shortage problem in non-English languages. The retrieval-augmented model leverages keyphrase annotations in English datasets to facilitate generating keyphrases in low-resource languages. Given a non-English passage, a cross-lingual dense passage retrieval module finds relevant English passages. Then the associated English keyphrases serve as external knowledge for keyphrase generation in the current language. Moreover, we develop a retriever-generator iterative training algorithm to mine pseudo parallel passage pairs to strengthen the cross-lingual passage retriever. Comprehensive experiments and ablations show that the proposed approach outperforms all baselines. $^{1}$
Keyphrases are single or multi-word lexical units that best summarize a piece of text. As such, they are of great importance for indexing, categorizing, and mining in many information retrieval and natural language processing tasks (Jones and Staveley, 1999; Frank et al., 1999; Hulth and Megyesi, 2006; Dave et al., 2003). Keyphrase generation is the task of automatically predicting keyphrases given
a piece of long text. Existing works on keyphrase generation mostly focus on English datasets (Gallina et al., 2019; Meng et al., 2017) while keyphrase generation for languages other than English is still under-explored. Since search engines usually provide services to customers using different languages, multilingual keyphrase generation becomes a significant problem while it is still unknown how well existing keyphrase generation approaches perform in non-English languages.
Nevertheless, there are two challenges we will face regarding multilingual keyphrase generation. First, to the best of our knowledge, there is no large-scale dataset publicly available for training and benchmarking multilingual keyphrase generation models. Building keyphrase datasets at a sufficient scale is difficult and costly. Second, compared with the existing datasets in English, which can contain millions of data examples and cover a wide diversity of topics, the data resources in non-English languages are inherently scarce. For example, in the domain of e-commerce, marketplaces using English have abundant customer queries to be used for keyphrase mining, while queries in some languages are relatively less than in English, which is probably because of a smaller size of user population or a shorter operation time.
We start tackling these challenges by contributing two new datasets for multilingual keyphrase, which cover six languages and two domains. The first dataset EcommerceMKP is collected from a real-world major e-commerce website. The product descriptions are used as the source text while the target keyphrases are collected from user search queries. This dataset contains a total of 73k data examples, covering four different languages (Spanish, German, Italian and French). The second multilingual keyphrase dataset AcademicMKP lies in the academic domain, in which titles and abstracts are used as the source text and the author-provided keyphrases are deemed targets. A total of 2,693 aca
* Qingyu Yin and Zheng Li are corresponding authors. This work was mainly done while Rui Meng was a Ph.D. student at University of Pittsburgh.
<sup>1</sup>The datasets are released at https://github.com/Yifan-Gao/multilingual_keyphrase_generation.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1233 - 1246 July 10-15, 2022 ©2022 Association for Computational Linguistics
demic papers in Chinese and Korean are included in AcademicMKP.
To overcome the resource scarcity challenge in training multilingual models, we propose a retrieval-based method to leverage the keyphrase knowledge in large-scale English datasets. By investigating multilingual keyphrase data, we observe that data in different languages may talk about similar topics. Therefore, we conjecture that passage-keyphrases pairs in English can be of help as an external knowledge base for multilingual keyphrase generation. To be specific, given a passage in low-resource language XX, we propose to use a retrieval model to find multiple top-related passages in English. These retrieved English passages provide high-quality English keyphrases that can be used as hints for generating keyphrases in other languages. After that, the generator takes the code-mixed inputs, including the passage in language XX and retrieved English keyphrases, and predicts keyphrases in language XX.
In the cross-lingual retrieval training, parallel passage-keyphrases pairs between English and other languages are extremely limited. For example, in the e-commerce domain, only a small fraction of products have both English and non-English descriptions (being sold in multiple countries). Such a data scarcity issue weakens the ability of cross-lingual knowledge acquisition from high-resource English keyphrases as intermediary, and finally hinders the potential of the retrieval-augmented keyphrase generation. To mitigate the problem, we propose a retriever-generator iterative training (RGIT) algorithm to automatically mine pseudo training parallel pairs from unlabeled data. Concretely, the retriever can dynamically adjust in terms of the current variations of generation performance between the proposed retrieval-augmented generator and the base one without the aid of retrieved English keyphrases. Starting from insufficient seed parallel pairs, if the retrieved pseudo passage-keyphrases pairs in the current iteration can bring in higher generation results as the generator's feedback, those pseudo parallel data will be regarded as high quality and incorporated into the seed ones to further boost the retriever. Such cycle providing positive effects can be repeated until the increasing generation performance stopped.
We conduct extensive experiments on EcommerceMKP and AcademicMKP and demonstrate that large-scale English datasets do provide use
ful knowledge for multilingual keyphrase generation. The proposed retrieval-augmented method outperforms traditional extraction-based models, sequence-to-sequence neural models, and its variants. Moreover, the RGIT algorithm boosts the retrieval performance significantly by mining over 20k pseudo-parallel passage pairs. We also conduct detailed analyses to investigate the effectiveness of retriever-generator iterative training.
Keyphrase Generation. The advance of neural language generation enables models to freely generate keyphrases according to the phrase importance and semantics, rather than extracting a list of sub-strings from the text (Witten et al., 1999a; Liu et al., 2011; Wang et al., 2016). Meng et al. (2017) propose the first keyphrase generation model CopyRNN, which not only generates words based on a vocabulary but also points to words in the source text — overcoming the barrier of predicting absent keyphrases. Following this idea, Chen et al. (2018); Zhao and Zhang (2019); Ahmad et al. (2021) leverage the attention mechanism to reduce duplication and improve coverage. Chen et al. (2019b); Ye and Wang (2018); Wang et al. (2019); Liang et al. (2021) propose to leverage extra structure information (e.g., title, topic) to guide the generation. Chan et al. (2019); Luo et al. (2021) propose a model using reinforcement learning, and Swaminathan et al. (2020) propose using GAN for KPG. Chen et al. (2020) introduce hierarchical decoding and exclusion mechanism to prevent models from generating duplicate phrases. Ye et al. (2021b) propose to dynamically align target phrases to eliminate the influence of order, as highlighted by Meng et al. (2021). Mu et al. (2020); Liu et al. (2020a); Park and Caragea (2020) use pre-trained language models for better representations of documents.
Retrieval Augmented Text Generation (RAG) recently shows great power in knowledge-intensive NLP tasks such as open-domain question answering, fact checking and entity linking (Lewis et al., 2020; Petroni et al., 2021; Guu et al., 2020). In RAG, a retriever (either sparse (Lee et al., 2019) or dense (Karpukhin et al., 2020)) searches for useful non-parametric knowledge from a knowledge base, then a generator combines the non-parametric retrieved knowledge with its parametric knowledge, learned during pre-training, for solving the task. Different from these tasks, keyphrase gen
Figure 1: Overview of our Retrieval-Augmented Multilingual Keyphrase Generation (RAMKG) framework.  $p^{\mathrm{XX}}$ ,  $k_{i}^{\mathrm{XX}}$  denote a passage and keyphrases in language XX (XX ∈ { DE, ES, FR, IT, KO, ZH}).  $p^{\mathrm{EN}}$ ,  $k_{i}^{\mathrm{EN}}$  denote relevant passages and keyphrases retrieved from the English dataset.
eration is not a knowledge-intensive task but we treat the English passage-keyphrase training data as our knowledge. Similar approaches have been investigated in neural machine translation (Gu et al., 2018; Cai et al., 2021), dialogue (Weston et al., 2018), and knowledge-base QA (Das et al., 2021). In keyphrase generation, Chen et al. (2019a); Ye et al. (2021a); Kim et al. (2021) retrieve similar documents from training data to produce more accurate keyphrases. However, their retrieval module is a non-parametric model and cannot be generalized in the multilingual setting due to the large vocabulary gap between languages.
In this paper, we aim to tackle the keyphrase generation task in a multilingual setting, which means one model of desire is capable of generating keyphrases in any language that it has been trained with. The benefits of having a single keyphrase generation model for multiple languages are threefold: (1) Collecting keyphrase annotation for individual language can be prohibitively expensive; (2) Training and deploying separate models for each language is laborious; (3) Joint training of multiple languages can alleviate the resource scarcity by utilizing rich monolingual data.
Formally, we define the multilingual keyphrase generation task as follows. Given a piece of text  $p^{\mathrm{XX}}$  in language XX, our goal is to predict its corresponding keyphrases  $k_1^{\mathrm{XX}}, k_2^{\mathrm{XX}}, \ldots, k_n^{\mathrm{XX}}$  in language XX, where  $n$  is the total number of target keyphrases for this text  $p^{\mathrm{XX}}$ . In this study, XX can be German (DE), Spanish (ES), Italian (IT), French (FR), Korean (KO) or Chinese (ZH).
Scarcity of resources is one of the topmost challenges for multilingual tasks, which is also the case for multilingual keyphrase generation. One may find it difficult to collect enough text data in languages other than English, much less the annotation of keyphrases in specific domains. To overcome this problem, we propose a retrieval-augmented approach to make use of the relatively rich resources in English. The motivation for our proposed retrieval-augmented approach comes from an observation from data: texts and keyphrases expressed in different languages usually share common topics or knowledge concepts. For example, in e-commerce websites, it is often the case that the same products are sold in different marketplaces/countries. Thus these products as well as their keyphrases, though exhibited in different languages, can share a high semantic similarity. In other words, given a text in language  $p^{\mathrm{AA}}$ , if we could find a similar text in language  $p^{\mathrm{BB}}$ , its associated keyphrases  $k^{\mathrm{BB}}$  may serve as a good hint for the to-be-generated keyphrases  $k^{\mathrm{AA}}$  in language AA. Since English has the most abundant text-keyphrases pairs in both e-commerce and academic papers domains, its resource can be treated as a non-parametric keyphrase knowledge base, which provides texts in English covering a wide range of topics and concepts, as well as the associated high-quality keyphrases.
As shown in Fig. 1, our framework consists of a retrieval step and a generation step:
1. Retrieval Step: given a source passage  $p^{\mathrm{XX}}$  in language XX, the cross-lingual retriever first finds  $m$  semantically relevant English passages  $p_1^{\mathrm{EN}}, p_2^{\mathrm{EN}}, \ldots, p_m^{\mathrm{EN}}$ . Each retrieved En
Figure 2: Retriever-Generator Iterative Training for Parallel Passage Mining
glish passage  $p_j^{\mathrm{EN}}$  has its associated  $n_j$  English keyphrases  $k_{j,1}^{\mathrm{EN}}, k_{j,2}^{\mathrm{EN}}, \ldots, k_{j,n_j}^{\mathrm{EN}}$ . These retrieved English keyphrases are taken as external knowledge for keyphrase generation in step 2.
2. Generation Step: taking the source text  $p^{\mathrm{XX}}$  in language XX and all retrieved English keyphrases  $\{k_{1,1}^{\mathrm{EN}}, \dots, k_{1,n_1}^{\mathrm{EN}}\}$ , ...,  $\{k_{m,1}^{\mathrm{EN}}, \dots, k_{m,n_m}^{\mathrm{EN}}\}$  as inputs, the generation module concatenates them as a sequence and generates keyphrases in target language XX.
4.1 Cross-Lingual Dense Passage Retrieval
The cross-lingual retriever includes a passage encoder  $E_P(\cdot)$  and a query encoder  $E_Q(\cdot)$ . The passage encoder  $E_P(\cdot)$  maps millions of English passages into  $d$ -dimensional vectors and builds indices for all English passages using FAISS (Johnson et al., 2021) offline. At inference time, the passage in language XX goes through the query encoder  $E_Q(\cdot)$  and is converted into a  $d$ -dimensional vector. Then the cross-lingual retriever performs a KNN search to retrieve  $m$  English passages whose vectors are closest to the query vector measured by the dot product similarity:  $\mathrm{sim}(p^{\mathrm{XX}}, p^{\mathrm{EN}_j}) = E_Q(p^{\mathrm{XX}})^\top E_P(p_j^{\mathrm{EN}})$ .
Passage Encoder. Since naive lexical similarity can hardly handle text matching across languages, we resort to a BERT-based dense passage retriever (Karpukhin et al., 2020), expecting the contextualized semantic matching can retrieve similar passages accurately and robustly. In order to meet the demand of multilingual representation, we utilize multilingual pre-trained model mBERT (Devlin et al., 2019) to encode passages into 768-dimensional vectors.
Training. Since the output vectors of mBERT are not aligned across languages, we need extra alignment training to ensure that similar passages in different languages can be mapped into near regions in the high-dimensional space. Given a passage  $p_i^{\mathrm{XX}}$  in language XX, we take its corresponding English passage  $p_i^{\mathrm{EN+}}$  as the positive example and randomly select  $n$  negative passages  $p_{i,1}^{\mathrm{EN-}},\dots,p_{i,n}^{\mathrm{EN-}}$  in the English corpus. The dense retriever is trained by optimizing the negative log likelihood loss of the positive English passage.
In the e-commerce domain, we select the positive passage according to product metadata. For a product sold in both EN and XX marketplaces, we regard its bilingual product descriptions  $(p_{i}^{\mathrm{XX}}, p_{i}^{\mathrm{EN+}})$  as a parallel passage pair, i.e., positive training example. For the domain of academic paper, we notice that papers with parallel text is very rare. Therefore, we develop an automatic approach to mine parallel abstract pairs of English and the target language. Specifically, we adopt an off-the-shelf bi-text mining tool named Sentence Transformers (Reimers and Gurevych, 2019) to mine pseudo parallel pairs. Given two datasets in different languages, we encode passages using LaBSE (Feng et al., 2020), the current best method for learning language-agnostic sentence embeddings for 109 languages, and then parallel passages can be extracted through nearest-neighbor retrieval and filtered by setting a fixed threshold over a margin-based similarity score, as proposed in (Artetxe and Schwenk, 2019).
4.2 Multilingual Keyphrase Generation with Code-Mixed Inputs
Given the top  $m$  retrieved English passages  $p_1^{\mathrm{EN}}$ , ...,  $p_m^{\mathrm{EN}}$ , we find their associated keyphrases in the
dataset:  $\{k_{1,1}^{\mathrm{EN}},\dots,k_{1,n_1}^{\mathrm{EN}}\}$  ...  $\{k_{m,1}^{\mathrm{EN}},\dots,k_{m,n_m}^{\mathrm{EN}}\}$  We utilize mBART (Liu et al., 2020b), a multilingual denoising pre-trained sequence-to-sequence language model, to integrate information from multiple languages. Different from machine translation which maps a sentence a the source language to a target language, our multilingual keyphrase generation model takes code-mixed inputs - a combination of retrieved English keyphrases  $\{k_{1,1}^{\mathrm{EN}},\dots,$ $k_{1,n_1}^{\mathrm{EN}}\}$  ...,  $\{k_{m,1}^{\mathrm{EN}},\dots,k_{m,n_m}^{\mathrm{EN}}\}$  from  $m$  retrieved English passages and the source passage  $p^{\mathrm{XX}}$  in the target language XX.
We concatenate retrieved English keyphrases with a delimiter token [SEP], and add special tokens to separate different inputs: [ENKPS] for retrieved keyphrases and [CTX] for the source passage. Besides, we follow the fine-tuning setup of mBART by adding the language identifier [XX] (e.g. [DE] for German) at the end of the input sequence to denote the current input language:
[ENKPS]  $k_{1,1}^{\mathrm{EN}}$  [SEP] ... [SEP]  $k_{m,n_m}^{\mathrm{EN}}$  [CTX]  $p^{\mathrm{XX}}$  [XX]. The training target is a sequence of concatenated keyphrases  $k_1^{\mathrm{XX}},\ldots ,k_n^{\mathrm{XX}}$  , separated by a special token [SEP]. [XX], the language identifier of the current language, is also added at the beginning of the target sequence to indicate the target language:
$$
[ \mathrm {X X} ] k _ {1} ^ {\mathrm {X X}} [ \mathrm {S E P} ] k _ {2} ^ {\mathrm {X X}} [ \mathrm {S E P} ] \dots [ \mathrm {S E P} ] k _ {n} ^ {\mathrm {X X}}.
$$
4.3 Retriever-Generator Iterative Training
In spite of having utilized parallel passage pairs to align the multilingual representations of the retrieval module, it remains a concern because the parallel passage pairs between English and non-English languages account for only a small portion of the whole multilingual dataset. For example, in a popular e-commerce platform, only a small percentage of products (less than  $10\%$ ) have both English and non-English descriptions. Without enough quality parallel pairs, the cross-lingual dense passage retriever may not work well to find relevant English passages. Consequently, associated English keyphrases may provide little help for multilingual keyphrase generation.
To make the multilingual keyphrase generation generalize better to any target languages or domains without reliance on numerous parallel passage pairs, we propose an iterative training method to mine parallel passages which requires only a small number of initial parallel pairs of bootstrap the process. Since our ultimate goal of the retriever model is to provide useful external knowledge for
multilingual keyphrase generation, we mine parallel passage pairs (English and a non-English language) according to whether the retrieved English passage-keyphrases pairs could help the keyphrase generation for the target non-English language XX. For example, let  $p_a^{\mathrm{EN}}$  and  $p_b^{\mathrm{EN}}$  be two retrieved English passages for a passage  $p^{\mathrm{XX}}$  in target language, if the associated keyphrases of  $p_a^{\mathrm{EN}}$  provide more useful information for generating the keyphrases of  $p^{\mathrm{XX}}$  than  $p_b^{\mathrm{EN}}$ , then  $(p_a^{\mathrm{EN}}, p^{\mathrm{XX}})$  would be considered as a better parallel passage pair. That said, we expect the mined pseudo parallel passage pairs to be of high quality according to the retrieval score, at the same time they can be directly helpful for training the generation module.
The proposed iterative training approach is sketched in Algo. 1 and Fig. 2. Given a Large-Scale keyphrase dataset in English  $D_{\mathrm{LS}} = \{(p_{\mathrm{LS}}^{\mathrm{EN}}, k_{\mathrm{LS}}^{\mathrm{EN}})\}$  and a smaller one  $D_{\mathrm{XX}} = \{(p_{\mathrm{XX}}^{\mathrm{X}}, k_{\mathrm{XX}}^{\mathrm{X}})\}$  in target language XX, we denote the set of annotated parallel examples (bilingual passages in English and other languages) as PARallel split  $D_{\mathrm{PAR}} = \{(p_{\mathrm{PAR}}^{\mathrm{EN}}, p_{\mathrm{PAR}}^{\mathrm{XX}}, k_{\mathrm{PAR}}^{\mathrm{EN}}, k_{\mathrm{PAR}}^{\mathrm{XX}})\}$ , in which  $\{(p_{\mathrm{PAR}}^{\mathrm{XX}}, k_{\mathrm{PAR}}^{\mathrm{XX}})\}$  comes from the XX dataset while  $\{(p_{\mathrm{PAR}}^{\mathrm{EN}}, k_{\mathrm{PAR}}^{\mathrm{EN}})\}$  comes from the English dataset. The remaining data examples in the target dataset
Language	Train Size	Dev Size	Test Size	Passage Length (Avg/Std/Mid)	#Keyphrases (Avg/Std/Mid)	Absent Kps%
AcademicMKP Dataset
Chinese (ZH)	1,110	158	319	217/48/207	5/1/5	27.2%
Korean (KO)	774	110	222	115/31/111	4/1/4	37.7%
Total	1,884	268	541	171/57/155	4/1/4	31.3%
EcommerceMKP Dataset
German (DE)	23,997	1,411	2,825	157/79/141	10/5/8	57.1%
Spanish (ES)	12,222	718	1,440	159/84/139	9/5/7	54.6%
French (FR)	16,986	998	2,000	163/84/144	9/5/8	63.0%
Italian (IT)	9,163	538	1,081	167/84/152	8/3/7	42.6%
Total	62,368	3,665	7,346	161/82/143	9/5/7	56.4%
Table 1: AcademicMKP & EcommerceMKP Dataset
$D_{\mathrm{XX}}$  have no annotated corresponding English examples in  $D_{\mathrm{LS}}$  (the pairs may exist but are not known yet), and we name this set as the Non-Parallel split  $D_{\mathrm{NP}} = \{(p_{\mathrm{NP}}^{\mathrm{XX}}, k_{\mathrm{NP}}^{\mathrm{XX}})\}$ . We firstly fine-tune a mBART using only keyphrases data of target language  $\{(p_{\mathrm{PAR}}^{\mathrm{XX}}, k_{\mathrm{PAR}}^{\mathrm{XX}})\}$  in  $D_{\mathrm{PAR}}$  (Line 4). Then we start a loop to mine pseudo parallel passage pairs for refining the passage retriever. Each iteration is expected to bring in a higher quality of pseudo passage pairs, resulting in a better performance of retriever, with three steps:
1. We train a retriever  $\mathbf{R}_t$  using existing available EN-XX passage pairs  $\{(p_{\mathrm{PAR}}^{\mathrm{EN}}, p_{\mathrm{PAR}}^{\mathrm{XX}})\}$  from both parallel passage data  $D_{\mathrm{PAR}}$  and most up-to-date pseudo passage data  $D_{\mathrm{PSEUDO}}^t$  (Line 8).
2. We train a retrieval-augmented model  $\mathbf{G}_t$  using multilingual passages  $\{(p_{\mathrm{PAR}}^{\mathrm{XX}}, k_{\mathrm{PAR}}^{\mathrm{XX}})\}$  from the parallel data  $D_{\mathrm{PAR}}$  and retrieved English keyphrases  $\{\hat{k}_{\mathrm{LS}}^{\mathrm{EN}}\}$  from the English dataset  $D_{\mathrm{LS}}$  (Line 14). To get the retrieved English keyphrases for each passage  $p_{\mathrm{PAR}}^{\mathrm{XX}}$ , we take the retriever  $\mathbf{R}_t$  trained in step (1) to do a KNN search for passages  $\hat{p}_{\mathrm{LS}}^{\mathrm{EN}}$  in EN  $D_{\mathrm{LS}}$  and find their associated keyphrases  $\{\hat{k}_{\mathrm{LS}}^{\mathrm{EN}}\}$  (Line 10-12).
3. For each passage  $p_{\mathrm{NP}}^{\mathrm{XX}}$  in the non-aligned dataset  $D_{\mathrm{NP}}$ , we also retrieve English passages  $\hat{p}_{\mathrm{LS}}^{\mathrm{EN}}$  and keyphrases  $\{\hat{k}_{\mathrm{LS}}^{\mathrm{EN}}\}$  from  $D_{\mathrm{LS}}$  (Line 18-19). Then the retrieved English passage  $\hat{p}_{\mathrm{LS}}^{\mathrm{EN}}$  will be taken as the parallel text to  $p_{\mathrm{NP}}^{\mathrm{XX}}$  if its associated keyphrases  $\{\hat{k}_{\mathrm{LS}}^{\mathrm{EN}}\}$  provide useful information. The usefulness is measured by the keyphrase generation performance (F-score) between the retrieval-augmented generation model  $\mathbf{G}_t$  and the base model  $\mathbf{G}_0$  that does not use EN keyphrases (Line 21-25).