Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper presents the first dataset focused on keyphrase generation for the French language. Previous works usually relied on unsupervised methods to perform extractive keyphrase generations. The authors also release several baseline models on related tasks (Papyrus f/e/m/a and a few previous datasets) that are useful for comparison.
strengths: (1) The methods used for data collection are simple and straightforward including tools used for language identification. It is also very easy to extend this to document libraries outside of Papyrus with little manual intervention.
(2) Papyrus-f appears to be very useful for evaluating French KPG models. 

weaknesses: The theme of the paper seems to be constantly switching between French and other languages. Other languages are mentioned quite casually with no details on how or why the language distribution is that way. Another detail missing is if the language distribution was used while training the multilingual models or while evaluating them (looks like it wasn’t). It is also a little hard for me to swallow that French itself is a minority language given the vast amount of literature in MT and unsupervised pretraining in the language. I would urge the authors to reconsider the wording in Section 6.

correctness: The dataset details and baseline metrics seem to be correct to me.
clarity: The paper is well structured and easy to read.
relation to prior work: Relation to prior work is discussed adequately. Maybe include using pretrained T5 models for comparison since they also tend to do very well on summarization/abstraction tasks?
additional feedback: L160: Citation style is incorrect, should be [38,28,26]
L264: imbalance* of data
Table 6: language codes need to be expanded somewhere in the paper
review_title: Good paper, useful dataset

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper proposes a new dataset for multilingual keyphrase generation, Papyrus, which includes 16,247 pairs of abstracts and key phrases. The multilingual corpus is collected from Papyrus, a repository of documents from the Université de Montréal. 
strengths: The task and dataset description of this paper is clear. The multilingual corpus comes from institutional reports or documents written by faculty members, therefore, the dataset quality can be relatively high.
weaknesses: 1. As a new dataset, data quality control or data quality evaluation is needed.
2. As a dataset for multilingual keyphrase generation, languages presented in the dataset are rather imbalanced. Except for English and French data, the amount of abstracts in other languages is too small.
correctness: Yes.
clarity: Yes, the paper is well written and easy to understand. 
relation to prior work: The discussions in the related work part are not comprehensive enough. The authors clarify the difference with Kp20K in terms of statistical information and baseline model performance.
additional feedback: None.
review_title: The dataset is important, but it needs refinements.

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: This paper introduces four versions of the Papyrus dataset which focus on keyphrase generation (KPG). The four datasets are Papyrus-e, which includes English examples, Papyrus-f, which includes French examples, Papyrus-m, which includes examples in any language, and Papyrus-a, which includes examples in several different languages at once. The authors also train several different versions of BART on their new datasets and compare results with state-of-the-art models. In general, their models outperform the state-of-the-art models such as YAKE on multilingual KPG.
strengths: •	Significance of contribution: Most of the previous work in the field of KPG is in English. In fact, the only other paper I know of in this specific field that isn’t English was published in 2022. Other work focusing on other languages in the field of keyphrase extraction has been done, but even then, it generally only focuses on a single language. People are just beginning to explore the concept of multilingual KPG, and this is one of the first multilingual KPG datasets.

•	Relevance to the broader research community: KPG can be relevant to many different fields. Augmenting the metadata for a dataset of texts to include keyphrases for each text can help make sorting through that text easier. For example, a dataset that includes the summaries or sneak peaks of books could be augmented to include keyphrases that might help readers choose which book to read.

•	Accessibility and accountability: The authors have clearly laid out how they will keep this dataset up and running and accessible going forward. 

•	Ethical and social implications: The ethical implications of this dataset are neutral, but in general I believe that having more data in languages other than English has a positive societal impact because as stated in the paper only around 25% of internet use is in English so that percentage should be reflected in the amount of available data. 

weaknesses: •	Significance of contribution: While two new datasets on this topic have been released this year by Gao et al. in their 2022 paper Retrieval-Augmented Multilingual Keyphrase Generation with Retriever-Generator Iterative Training, their datasets were only released a short time before this dataset would’ve been submitted for review and are significantly different enough that I don’t believe they affect the significance of this paper’s contribution. 
correctness: To the best of my knowledge the dataset is constructed in a sound way and the claims made in the submission are correct.
clarity: This paper is well written.
relation to prior work: Yes, the authors clearly state how this work relates to and differs from prior work in section 2.
additional feedback: Have you considered trying other multilingual encoders such as XLM-RoBERTa? Obviously mBART worked well for your purposes, but I’m just curious to know if you tried other methods as well before settling on BART based encoders.
review_title: This paper introducing four new well-formed multilingual datasets on keyphrase generation is very enjoyable.

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: This paper introduces a new multilingual keyphrase dataset and present a BART-based baseline with discussions. 
strengths: Keyphrase generation is valuable task, and having a suitable data set will be helpful to the community.
The method of task data construction sounds reasonable, and the use of data sources is also good
weaknesses: Lack of detailed explanation of data collection (i.e. the ``heuristic'' in Line 97)
Lack of a more complete introduction to manual evaluation, such as  the quality of evaluators and evaluation criteria
Lack of analysis on the shortcomings of current methods in multilingual scenarios
Although it is a multilingual data set, most of the data are only English and French, and the amount of data in other languages is too small.

correctness: Yes
clarity: Yes
relation to prior work: Yes
additional feedback: No
review_title: A multilingual keyphrase dataset with limited analysis

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper proposes a multilingual keyphrase generation dataset with four versions (Papyrus-e, Papyrus-f, Papyrus-m, Papyrus-a), reflecting multiple scenarios in multilingual keyphrase generation. The original corpus is extracted from a multilingual corpus (Papyrus) that contains various types of documents in different languages. To show the characteristics of the dataset, the authors utilize existing keyphrase extraction methods and BART and discuss some interesting patterns in keyphrase generation in multilingual settings.
strengths: The authors do a great job motivating the need for the multilingual version of keyphrase generation. 

Extracting keywords in different languages at the same time (Papyrus-a) is an interesting setup as some important quotes or terms can be written in other languages to preserve their original meaning.

weaknesses: According to the authors' reference on the percentage of language used on the internet, English and French comprise 29.2% (25.9% + 3.3%) of total languages used. To better reflect the authors' motivation of "multilinguality," other dominant languages such as Chinese and Spanish should be taken into account more in the dataset.

It would be good to mention existing works on multilingual keyphrase extraction as they are also related to the paper's main argument (why not paraphrase those keywords instead of constructing from scratch).

correctness: The work seems to be sound and correct.
clarity: The paper is easy-to-follow. Some concerns are noted above.


relation to prior work: Addressed in the section above. 
additional feedback: N/A
review_title: Review of "A new dataset for multilingual keyphrase generation"

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper introduces a new dataset for multilingual keyphrase generation called Papyrus. The dataset contains 16k pairs of abstracts and keyphrases essentially for English and French with a long tail of additional languages that, in practice, can only be used for evaluation. The authors release four versions of this dataset with different language configurations. Using multilingual Papyrus, they train the BART model, which shows better results for French, Spanish and Polish keyword extraction datasets. Training BART on the Kp20K and English version of the Papyrus dataset exhibits even better performance on a set of benchmarks from https://arxiv.org/abs/2201.05302. The percentage of absent keywords is lower than in Kp20K, but considering the dataset size can be helpful for training models.

strengths: - well-written paper
- very good expansion to English only keyword extraction datasets that could pave the way for using other universities' libraries to get even more linguistic diversity
- evaluation conducted on various languages

weaknesses: - "Multilinguality" is in practice limited to English and French, as the third most common language (Spanish) has only 172 items

- What is the purpose of introducing the long tail of languages, especially when there are only one or two occurrences? It can introduce a bias in evaluations 

- It would be nice to see some comparison of other models on Kp20K and other datasets in Table 7, especially if they are available in previous work: https://arxiv.org/abs/2201.05302

correctness: - **L146-148**: "It is to note that for mBART-large-50, the language has to be specified when tokenizing. For languages that are not available with mBART-large-50, or for the multilingual inputs of Papyrus-a, we default to English. "

---> I don't understand this part. Does it mean that for Papyrus-a (EN+FR+others), the only tokenizer used was an english tokenizer? What was the rationale for this choice when half of the dataset contains french keywords and abstracts?



- **L127-128**: "One is, as far as we can judge, Inuktitut. The other was a mistaken entry where, in addition to the French and English abstracts, the term ”2002-10” had been marked as an abstract."

---> Maybe it makes sense to add a threshold on the minimum text size as too short texts can generate noise?


- Having introduced the Papyrus-m version of the dataset, what are the gains of also having Papyrus-a? Wouldn't it only introduce noise and misclassification about their usage?
clarity: The paper is fairly clear and easy to understand.

One suggestion that could make the flow even better would be to consider moving some of the footnotes into the main text.
relation to prior work: Related work is clearly discussed.

Just pointing as an additional reference very recent paper proposing a similar dataset for six languages (although I was not able to access the dataset itself, so it can be hard to compare): 
Retrieval-Augmented Multilingual Keyphrase Generation with Retriever-Generator Iterative Training
Yifan Gao, Qingyu Yin, Zheng Li, Rui Meng, Tong Zhao, Bing Yin, Irwin King, Michael R. Lyu
https://arxiv.org/abs/2205.10471
additional feedback: It would be nice to see the distribution of labels by the field of study to check if some disciplines are not too much over/underrepresented, so the model will not learn the artefacts.
review_title: Review of "A new dataset for multilingual keyphrase generation"

Meta-review: metareview: This paper presents the dataset Papyrus, consisting of abstracts and keyphrases extracted from institutional reports from the Université de Montréal. The dataset is mostly in two languages and was generated using straightforward, yet easily extensible methods for extraction. This is one of the first datasets that expands KPG beyond English. Although limited in coverage, it sets the tone for generalization of techniques beyond English.


**pros** 
* Expands the field of KPG to other languages beyond English.
* The methods for extraction are straightforward.
* Given its origin, the quality of the data is expected to be high.
* A portion of the data has been validated through human evaluation.

**cons**
* The coverage of the dataset is mostly limited to English and French. Other languages aren't well represented. 


confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept