KPEVAL: Towards Fine-Grained Semantic-Based Keyphrase Evaluation (2024)

Abstract: Despite the significant advancements in keyphrase extraction and keyphrase generation methods, the predominant approach for evaluation mainly relies on exact matching with human references. This scheme fails to recognize systems that generate keyphrases semantically equivalent to the references or diverse keyphrases that carry practical utility. To better assess the capability of keyphrase systems, we propose KPEVAL, a comprehensive evaluation framework consisting of four critical aspects: reference agreement, faithfulness, diversity, and utility. For each aspect, we design semantic-based metrics to reflect the evaluation objectives. Meta-evaluation studies demonstrate that our evaluation strategy correlates better with human preferences compared to a range of previously proposed metrics. Using KPEVAL, we re-evaluate 23 keyphrase systems and discover that (1) established model comparison results have blind-spots especially when considering reference-free evaluation; (2) large language models are underestimated by prior evaluation works; and (3) there is no single best model that can excel in all the aspects.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 1959–1981 August 11-16, 2024 ©2024 Association for Computational Linguistics
Despite the significant advancements in keyphrase extraction and keyphrase generation methods, the predominant approach for evaluation mainly relies on exact matching with human references. This scheme fails to recognize systems that generate keyphrases semantically equivalent to the references or diverse keyphrases that carry practical utility. To better assess the capability of keyphrase systems, we propose KPEVAL, a comprehensive evaluation framework consisting of four critical aspects: reference agreement, faithfulness, diversity, and utility. For each aspect, we design semantic-based metrics to reflect the evaluation objectives. Meta-evaluation studies demonstrate that our evaluation strategy correlates better with human preferences compared to a range of previously proposed metrics. Using KPEVAL, we re-evaluate 23 keyphrase systems and discover that (1) established model comparison results have blind-spots especially when considering reference-free evaluation; (2) large language models are underestimated by prior evaluation works; and (3) there is no single best model that can excel in all the aspects.

heading: 1 Introduction
text: Building automated keyphrase prediction systems has been a long-lasting research interest of Information Retrieval (IR) and Natural Language Processing (NLP) (Witten et al., 1999; Hulth, 2003; Meng et al., 2017). While a large number of keyphrase prediction systems have been proposed, the majority of them are assessed using a simplistic method: comparing the stemmed predictions with human references for exact matches. An extensive review of 76 recent keyphrase extraction and generation papers published in major conferences reveals a predominant reliance on exact matching, with 75 of 76 papers employing it and 51 treating it as the sole evaluation criterion (Appendix A).
This over-reliance brings two major concerns. First, it has been established that the evaluation accuracy of exact matching is inadequate (Zesch and Gurevych, 2009). Although a number of heuristics are proposed to relax the matching criteria (Zesch and Gurevych, 2009; Kim et al., 2010; Luo et al., 2021; Koto et al., 2022) or enrich the label set (Chan et al., 2019), they still struggle to accurately capture phrase semantics and have not been validated by systematic meta-evaluation. Second, solely relying on reference-based evaluation is an incomplete strategy that overlooks critical aspects such as diversity of the predicted keyphrases (Bahuleyan and El Asri, 2020) or their utility in practical applications of keyphrase systems such as indexing for IR applications (Boudin and Gallina, 2021).
In this paper, we undertake a systematic approach to advance keyphrase evaluation. For reference-based evaluation, we propose a phraselevel semantic matching metric with a high quality embedding trained on large-scale keyphrase data. Based on the human evaluation corpus annotated on KP20k (Meng et al., 2017) and KPTimes (Gallina et al., 2019), the meta-evaluation on five keyphrase systems shows that our metric significantly outperforms existing metrics by more than 0.15 absolute points in Kendall’s Tau. By contrast, many proposed improvements to exact matching surprisingly fail to improve its human agreement (§5.3). Further analyses reveal that the proposed metric exhibits enhanced stability under the label variations commonly present in the keyphrase annotations.
Next, we move beyond reference agreement and holistically consider the desiderata for evaluating keyphrase systems. Three crucial aspects are introduced: (1) faithfulness, whether the predictions are grounded to the document (§6.1); (2) diversity, whether the predictions represent distinct concepts (§6.2); and (3) utility for downstream IR applications (§6.3). To accurately assess each aspect, we propose semantically-oriented metric designs
1 1959
including embedding similarity, model-based consistency evaluation, and dense retrieval.
Together, these aspects and metrics form KPEVAL, a fine-grained semantic-based keyphrase evaluation framework (Figure 1). In §7, we employ KPEVAL to evaluate 23 keyphrase systems, producing intriguing insights:
1. KPEVAL uncovers blind-spots in established model comparisons, such as the actual superiority of ExHiRD-h (Chen et al., 2020) in many aspects as well as a common difficulty to outperform a baseline in all the aspects. 2. We find that large language models (LLMs), particularly GPT-3.5 (Ouyang et al., 2022), exhibit remarkable performance compared to current state-of-the-art keyphrase generation and extraction models. Our results challenge existing conclusions and lead to a reconsideration of using LLMs as keyphrase systems. 3. Finally, KPEVAL’s four aspects test distinct abilities at which different models excel, suggesting the importance of aligning evaluation with the diverse needs of real applications.
In summary, KPEVAL establishes a new standard for keyphrase evaluation by advancing reference-based evaluation accuracy and aligning model development with application values via holistic reference-free evaluation. To facilitate future studies, the implementation is released as a toolkit along with the meta-evaluation annotations at https://github.com/uclanlp/KPEval.

heading: 2 Related Work
text: In this section, we review the relevant literature on evaluating keyphrase systems.
Reference-based evaluation The major metrics for evaluating keyphrase systems are precision, recall, and F1 based on exact-match between the stemmed predictions and references (Mihalcea and Tarau, 2004; Meng et al., 2017; Yuan et al., 2020). This method indiscriminately penalizes unmatched predictions, including synonyms or parent/child concepts of the reference. Later works attempt to improve the metric by relaxing the matching criterion. Zesch and Gurevych (2009) propose to use R-precision with approximate matching, tolerating a prediction to be a substring of a reference and vice versa. Kim et al. (2010) employ n-gram matching metrics such as BLEU (Papineni et al., 2002) and Rouge (Lin, 2004). Chan et al. (2019) expand the references with name variations. Luo et al. (2021) propose a fine-grained score that combines token-level matching, edit distance, and duplication penalty. Koto et al. (2022) and Glazkova and Morozov (2022) use the semantic-based BertScore (Zhang et al., 2020) with predictions and references concatenated into two strings.
Meanwhile, ranking-based metrics such as Mean Reciprocal Rank, mean Averaged Precision, and Normalized Discounted Cumulative Gain are introduced to evaluate the ranking provided by keyphrase extraction models (Florescu and
2
Caragea, 2017; Boudin, 2018; Kim et al., 2021). These metrics also compute exact matching to the references during their evaluation.
Reference-free evaluation Directly evaluating keyphrase predictions without references is less common. Early studies conduct human evaluation (Barker and Cornacchia, 2000; Matsuo and Ishizuka, 2004). Later work evaluates the predictions’ utility in applications such as retrieval (Bracewell et al., 2005; Boudin and Gallina, 2021) or summarization (Litvak and Last, 2008). Bahuleyan and El Asri (2020) conduct reference-free evaluation of the predictions’ diversity.
Meta-evaluation Meta-evaluation studies that compare keyphrase metrics with human evaluations have been limited in scope, with a focus on reference-based evaluation. Kim et al. (2010) compare five lexical matching metrics and concluded that R-precision has the highest Spearman correlation with human judgments. Bougouin et al. (2016) annotated a meta-evaluation corpus with 400 documents in French, evaluating 3 keyphrase models on "appropriateness" and "silence", approximately corresponding to precision and false negative rate.
Discussion Building upon existing literature, this work systematically rethinks the goals of keyphrase evaluation and advances the evaluation methodology. We introduce KPEVAL, a holistic evaluation framework encompassing four key aspects (§4). KPEVAL incorporates semantic-based metrics validated via rigorous meta-evaluation (§5.3 and §6.1). Finally, we conduct a large-scale evaluation of 21 keyphrase systems and offer novel insights into existing model comparisons and LLMs (§7).

heading: 3 Background
text: This section formulates the keyphrase prediction and evaluation tasks and outlines the scope of study.

heading: 3.1 Keyphrase Prediction
text: We denote an instance of keyphrase prediction as a tuple (X ,Y), where X represents an input document and Y = {y1, ..., yn} is a set of n reference keyphrases provided by humans. Each yi is categorized as a present keyphrase if it corresponds to contiguous word sequences in X after stemming, or an absent keyphrase if it does not. Keyphrase generation (KPG) assumes Y to include both present and absent keyphrases, whereas keyphrase extraction (KPE) only allows present keyphrases in Y .

heading: 3.2 Keyphrase Evaluation
text: The keyphrase evaluation process can be viewed as mapping a 4-element tuple (X ,Y,P, C) to a real number via a function f . P = {p1, ..., pm} is a set of m predictions made by a model M on X . Different from the commonly followed works (Meng et al., 2017; Yuan et al., 2020), we do not distinguish between present and absent keyphrases. This enables matching a predicted keyphrase to any semantically relevant reference, and vice versa. C is a corpus that represents the domain of interest, which is an important factor in assessing keyphrase quality. For example, "sports games" may be informative in a general news domain but less so in the specialized domain of basketball news1. In this paper, C will play a crucial role in evaluating the utility of keyphrases for facilitating ad-hoc in-domain document retrieval (§6.3).

heading: 3.3 Evaluation Scope
text: Models This paper covers 21 representative, strong, and diverse keyphrase prediction models spanning three categories: (1) KPE models, (2) KPG models, and (3) large language models (LLMs) and APIs. We aim to include highly cited (up to February 2024) models such as MultipartiteRank (Boudin (2018), 219 citations), CatSeq (Yuan et al. (2020), 92 citations), and SetTrans (Ye et al. (2021), 65 citations). We provide introductions and implementation details in Appendix B.
Datasets We test on two datasets throughout the paper: (1) KP20k (Meng et al., 2017) that features 20k Computer Science papers with keyphrases extracted from the paper metadata and (2) KPTimes (Gallina et al., 2019) that provides 10k news documents paired with keyphrases assigned by expert editors. The two datasets are selected due to their large training sets (500k for KP20k and 250k for KPTimes) and their wide usage in keyphrase research. As such, the models’ performance is easier for the community to relate to and the reproduction correctness can be verified more easily.

heading: 4 KPEVAL: Evaluation Aspects
text: We introduce KPEVAL, a fine-grained framework for keyphrase evaluation. KPEVAL posits to evaluate P’s quality across four crucial aspects:
1. Reference Agreement: Evaluates the extent to which P aligns with human-annotated Y .
1Check Tomokiyo and Hurst (2003) for more examples.
3
2. Faithfulness: Determines whether each pi in P is semantically grounded in X . 3. Diversity: Assesses whether P includes diverse keyphrases with minimal repetitions. 4. Utility: Measures the potential of P to enhance downstream applications, such as document indexing for improved IR performance.
Table 1 outlines the assumptions of the evaluated aspects: whether they are calculated on a set of phrases and whether X , Y , or C is needed for evaluation. By design, these aspects have deep groundings in the previous literature. Faithfulness and reference agreement can be seen as different definitions of informativeness: the former enforces the information of pi to be contained in X , while the latter measures P’s coverage of X ’s salient information with respect to a background domain (Tomokiyo and Hurst, 2003). Diversity (Bahuleyan and El Asri, 2020) and IR-based utility (Boudin and Gallina, 2021) reflect major efforts to move beyond reference-based evaluation. Building upon these works, KPEVAL aims to provide a unified perspective and to advance the evaluation methodology. Figure 1 illustrates the evaluation design for each aspect, which we will introduce next.

heading: 5 KPEVAL: Reference-Based Evaluation with Semantic Matching
text: To begin with, we focus on reference agreement, the most extensively investigated aspect. Recognizing the limitations of previous approaches, we introduce a semantic matching formulation and conduct meta-evaluation to confirm its effectiveness.

heading: 5.1 Reference Agreement: Metric Design
text: Desiderata: a prediction should be credited if it is semantically similar to a human-written keyphrase; matching should be at phrase-level.
Despite the prevalent use of existing referencebased metrics, their designs harbor intrinsic limitations. On one hand, F1@5 (Meng et al., 2017) and F1@M (Yuan et al., 2020) fail to credit many semantically correct predictions. On the other hand,
BertScore with concatenated predictions and references (Koto et al., 2022) reflects semantic similarity, but its token-level matching strategy obscures the semantics of individual keyphrases. Recognizing these limitations, we propose a phrase-level semantic matching strategy in KPEVAL and define semantic precision (SemP ), recall (SemR), and F1 (SemF1) as follows2:
SemP (P,Y) = 1|P| ∑ p∈P maxy∈Y sim(p, y),
SemR(P,Y) = 1|Y| ∑ y∈Y maxp∈P sim(p, y),
SemF1(P,Y) = 2·SemP (P,Y)·SemR(P,Y)SemP (P,Y)+SemR(P,Y) ,
where sim is the similarity between the representation of two phrases. To enable the use of any existing dense embedding model, in this paper, we operationalize sim with the cosine similarity:
sim(p, q) = cos_sim(hp, hq) = hTp hq
||hp||·||hq || ,
where hp is the representation of phrase p obtained by aggregating the representation of all tokens in the phrase. To obtain a high quality embedding that captures phrase-level semantics well, we fine-tune a paraphrase model from Reimers and Gurevych (2019)3 using unsupervised SimCSE (Gao et al., 2021) on 1.04 million keyphrases from the training sets of KP20k, KPTimes, StackEx (Yuan et al., 2020), and OpenKP (Xiong et al., 2019). The data covers a wide range of domains including science, news, forum, and web documents. At inference time, a single phrase p is provided as the input to the model, and the last hidden states are meanpooled to obtain hp. We further document the training details of this model in Appendix C.1.

heading: 5.2 Meta-Evaluation Setup
text: We conduct rigorous meta-evaluation to compare SemF1 with existing metrics. We sample 50 documents from the test sets of KP20k and KPTimes each. For each document, we obtain predictions from five representative models: MultipartiteRank, CatSeq, SetTrans, in-domain BART models from Wu et al. (2023a), as well as five-shot prompting GPT-3.54. This variety encompasses both KPE
2Out metric should be distinguished from Bansal et al. (2022). We keep the name choice as the tasks are different.
3We use the checkpoint at https://huggingface.co/ sentence-transformers/all-mpnet-base-v2.
4We use SciBART+OAGKX for KP20k and KeyBART for KPTimes to represent in-domain BART models. Detail regarding the evaluated models are provided in Appendix B.
4
and KPG model, and includes unsupervised, supervised, and few-shot prompting methods. Then, three crowd-source annotators are asked to rate on Likert-scale the semantic similarity between (1) each prediction keyphrase pi and the most semantically similar keyphrase in Y , and (2) each reference keyphrase yi and the most semantically similar keyphrase in P . We report the details of the annotator recruitment process, the annotation instructions, and the interface in Appendix D.
A total of 1500 document-level annotations with 13401 phrase-level evaluations are collected. As presented in Table 2, we observe 0.75 Krippendorff’s alpha for both datasets and both matching directions, indicating a high inter-annotator agreement. The annotations are aggregated to obtain (1) phrase-level scores for matching a single phrase to a set of phrases (pi → Y and yi → P) and (2) document-level precision, recall, and F1 scores, calculated after normalizing the scores to a 0-1 scale.

heading: 5.3 Meta-Evaluation Results
text: Using the document-level F1 score annotations, we compare SemF1 with six baseline metrics:
1. Exact Matching F1@M (Yuan et al., 2020). 2. F1@M with Substring Matching. We con-
clude a match between two phrases if either one is a substring of the other. This corresponds to the INCLUDES and PARTOF strategy in Zesch and Gurevych (2009). 3. R-precision (Zesch and Gurevych, 2009). 4. FG (Luo et al., 2021). 5. Rouge-L F1 (Lin, 2004). 6. BertScore FScore (Zhang et al., 2020)5. We
concatenate all the phrases in P with commas to form a single prediction string, and do the same for Y to form the reference string6. We apply Porter Stemmer (Porter, 1980) on P and Y before calculating baseline 1, 2, and 3.
5We use the RoBERTa-large model and the representation at the 17th layer, as recommended by the official implementation at https://github.com/Tiiiger/bert_score.
6We find that BertScore is insensitive to the order of labels and predictions. Details are discussed in Appendix C.4.
In Figure 2, we report the 95% confidence interval of Kendall’s Tau via input-level bootstrap resampling with 1000 samples, following Deutsch et al. (2021). Surprisingly, although exact matching produces many false negatives, existing proposals to relax exact matching do not provide much overall performance gains either: while substring matching consistently outperforms exact matching by a small amount, R-precision and FG have a lower correlation with human compared to exact matching. BertScore’s performance is highly domain-dependent: it achieves the second-best performance on KPTimes while performs poorly on KP20k. By contrast, SemF1 greatly outperforms other metrics on both datasets, with a much higher mean score and a smaller variation.
Is the observed high performance of SemF1 consistent with any embedding model? Our ablation studies suggest a negative answer. We evaluate with fine-grained phrase-level annotations for both directions of matching (i.e., pi → Y and yi → P). Table 3 presents the Pearson correlation (r), Spearman correlation (ρ), and Kendall’s Tau (τ ) of exact matching and semantic matching with various embedding models: FastText (Joulin
5
et al., 2016)7, Phrase-BERT (Wang et al., 2021), SpanBERT (Joshi et al., 2020), SimCSE (Gao et al., 2021)8, our phrase embedding, and the model before the proposed fine-tuning. Despite being a strong strategy by design, semantic matching fails to outperform exact matching in Kendall’s Tau with SpanBERT embedding. With the proposed model, semantic matching outperforms exact matching by 0.1 absolute points in Kendall’s Tau and more than 0.15 absolute points in Pearson and Spearman correlation. It is worth-noting that although the base SBERT model already achieves strong performance, our phrase-level contrastive fine-tuning provides further performance gains.
Remark We have confirmed that the semantic matching strategy better accommodates semantically correct predictions. Additionally, our preliminary study indicates that human references often exhibit lexical variability. When faced with such variability, SemF1 demonstrates lower variance than F1@M (detailed in Appendix C.5).

heading: 6 KPEVAL: Reference-Free Evaluation
text: For a range of text generation tasks, the optimal output is often highly aspect-specific (Wen et al., 2015; Mehri and Eskenazi, 2020; Fabbri et al., 2021). As such, reference-based evaluation is incomplete as it does not always align with the evaluation goals. To address this gap, KPEVAL introduces three novel evaluation aspects, along with corresponding reference-free metrics, aimed at aligning closer with real-world application requirements.

heading: 6.1 Faithfulness
text: Desiderata: keyphrase predictions should always be grounded in the document.
In practical scenarios, it is vital for keyphrase systems to refrain from producing concepts not covered in the document, which we term as unfaithful keyphrases. Determining whether a keyphrase is faithful is non-trivial: an absent keyphrase could be faithful by being synonyms or parent/child concepts of the concepts in the document, while a present keyphrase could be deemed unfaithful if it has a wrong boundary. For instance, the keyphrase "hard problem" is unfaithful to a document discussing "NP-hard problem". This example also
7We use the crawl-300d-2M.vec model. 8We use the simcse-roberta-large models distributed
by the original authors on huggingface.
illustrates the inadequacy of reference-based evaluation, as "hard problem" may achieve a high score when matched against "NP-hard problem".
Are existing keyphrase models faithful? We conduct a human evaluation of the same set of five models in §5.2 on 100 documents from KP20k and KPTimes each. For each (document, keyphrase prediction) pair, three annotators are asked to make a binary judgement between faithful and unfaithful (details in Appendix D). Table 2 presents the interannotator agreement. We find a moderate agreement for present keyphrases and a lower agreement for absent keyphrases. We aggregate the present keyphrase annotations by majority voting. For the absent keyphrases, two of the authors manually resolve the instances where the crowd-source annotators do not agree. Table 4 presents the faithfulness scores for the evaluated models. Surprisingly, M3’s outputs are not as faithful as the neural KPG models, supporting the hypothesis that extractive models can suffer from the boundary mistakes that harm their faithfulness. In addition, models make more unfaithful predictions in KP20k compared to KPTimes, indicating the possible difficulty of accurately generating concepts grounded in scientific papers compared to news documents.
Automatic Faithfulness Evaluation Using the human annotations, we evaluate three automatic metrics for judging a keyphrase’s faithfulness:
1. The precision metric of BertScore(X , pi). We use the RoBERTa-large model as in §5.3. 2. The faithfulness metric of BartScore (Yuan et al., 2021). pi is embedded into "In summary, this is a document about pi" for calculating its probability given X . BART-large trained on CNN-DM (See et al., 2017) is used. 3. The consistency metric of UniEval (Zhong et al., 2022), which scores text generation as boolean QA. We embed X and pi with a
6
template for summarization evaluation: "question: Is this claim consistent with the document? 
 summary: the document discusses about pi. 
 document: X ". Then, we use the UniEval model for summarization evaluation provided by the original authors to obtain a score expressed as the probability of the model generating "Yes" normalized by the probability for "Yes" and "No".
All of these metrics output a real number score. To compare their performance, we report their AUROC in Table 5. On both datasets, UniEval outperforms BertScore and BartScore, achieving the highest agreement with human raters. Currently, KPEVAL adopts UniEval as the default faithfulness metric. We encourage future work to continue developing stronger metrics for this aspect.

heading: 6.2 Diversity
text: Desiderata: reward more semantically distinct concepts and penalize repetitions.
Generating keyphrases with minimal repetition is a desirable property of keyphrase applications. To assess the diversity of P , KPEVAL includes one lexical and one semantic metric based on Bahuleyan and El Asri (2020). The lexical metric dup_token_ratio is the percentage of duplicate tokens after stemming. The semantic metric dup_emb_sim is the average of pairwise cosine similarity, using the phrase embedding in §5.1:
emb_sim(P) = ∑m i=1 ∑m j=1 1(i ̸=j)sim(pi,pj) m(m−1) .
We note that by design, we do not penalize overgenerating uninformative keyphrases, as it intuitively implies a high diversity9. Judging the quality of the keyphrases is instead delegated to the metrics for reference agreement and faithfulness.
9As a result, metrics such as the orthogonal regularization term used by CatSeqD (Yuan et al., 2020) are not suitable for our purposes, as the term naturally increases with |P|.

heading: 6.3 Utility
text: Desiderata: reward predictions that facilitate effective ad-hoc retrieval of the document.
Information Retrieval (IR) is an important downstream application for keyphrases (Jones and Staveley, 1999; Song et al., 2006; Kim et al., 2013). To directly evaluate whether M can generate useful keyphrases for IR-related tasks, KPEVAL tests P on facilitating ad-hoc retrieval of X from an indomain corpus C (Boudin and Gallina, 2021).
Concretely, we leverage an in-domain corpus C that has documents and human-annotated keyphrases10. We first index C’s documents into the form (title, keyphrases) → document. To evaluate P , we add a single entry (X ’s title, P) → X to the aforementioned database. Then, a set of queries Q = {q1, q2, ..., q|Q|} specifically written for X are used to attempt to retrieve X from this pool. The utility of P is measured by two metrics for retrieval effectiveness: Recall at k (Recall@k) and Reciprocal Rank at k (RR@k), averaged across all queries in Q. To simulate the queries written by real users, we use GPT-4 (OpenAI, 2023) to annotate three adhoc queries based on each document. For KP20k, the queries are in the style of in-text citations similar to Boudin and Gallina (2021). For KPTimes, we generate short phrases that mimic queries on web search engines. We present the prompting details in Appendix C.2. For metric calculation, we consider BM25 (Robertson and Walker, 1994) and a dense embedding model11 as the retriever and report the averaged scores.

heading: 7 Fine-grained benchmarking of keyphrase systems
text: Finally, we benchmark 23 keyphrase systems with KPEVAL, with the full evaluation results on KP20k and KPTimes presented in Table 7. The implementation details are presented in Appendix B. This section presents the insights on two questions:
1. Do our finding align with the conclusions drawn from previous model comparisons? 2. How do large language models compare with existing keyphrase prediction methods?
Uncovering blind-spots of established results We revisit a set of models compared in Ye et al.
10We use the respective training sets as C for KP20k and KPTimes. In practice, one can run any keyphrase prediction method if human-written keyphrases are not available for C.
11We use cross-encoder/ms-marco-MiniLM-L-6-v2 model via huggingface.
7
(2021): CatSeq, CatSeqTG-2RF1, ExHiRD-h, and SetTrans. As shown in Table 6, a nuanced pattern emerges when evaluating beyond F1@M , the main metric reported in the original paper. SemF1 is consistent with F1@M in recognizing SetTrans as the best model for reference agreement. However, SetTrans does not outperform all the three baselines in reference-free evaluation. Specifically, the best faithfulness and diversity scores are achieved by ExHiRD-h, and the difference between SetTrans and CatSeqTG-2RF1 in utility is insignificant. Moreover, contradicting with F1@M , KPEVAL’s metrics show a superiority of ExHiRD-h over CatSeqTG-2RF1 for reference agreement, faithfulness, and diversity. We provide several supporting examples in Appendix E. By revealing these blind-spots in previous results, KPEVAL enables a holistic view in model comparison and a stricter criterion in establishing the state-of-the-art.
LLM vs traditional keyphrase models With the ascent of LLMs as foundational elements in NLP, their efficacy in keyphrase prediction warrants examination. Prior research reported significant performance gaps when evaluating LLMs with exact matching F1@M (Song et al., 2023; MartínezCruz et al., 2023). With KPEVAL, we conduct a more comprehensive investigation. In Figure 3, we compare GPT-3.5 with state-of-the-art KPE and KPG methods. For supervised methods, perfor-
mance of five-shot prompting GPT-3.5 is comparable or better than HyperMatch along every dimension, and comparable to SciBART+OAGKX in diversity, utility, and faithfulness. In addition, zeroshot prompting outperforms the unsupervised TextRank, PromptRank, and the Azure API in diversity while being competitive across other dimensions. These results suggest that the potential of LLMs for keyphrase prediction may be underappreciated under traditional evaluation paradigms.
Discussion What have we learned in this reevaluation? First, the refined evaluation facilitated by KPEVAL challenges some of the existing model comparisons and emphasizes the difficulty of outperforming baselines across all aspects. In fact, these aspects test unique abilities, exhibiting weak cross-aspect correlations (Appendix C.3) and distinct preferences for keyphrase systems (Table 6, 7). Our findings advocate for a tailored approach to metric weighting, allowing users to customize evaluations based on their evaluation desiderata. Finally, our results reveal strong performance of GPT-3.5, encouraging future work to further understand and improve LLMs for keyphrase prediction.

heading: 8 Conclusion
text: We introduce KPEVAL, a fine-grained evaluation framework that conducts semantic-based evaluation on reference agreement, faithfulness, diversity,
8
and utility of keyphrase systems. We show the advantage of our metrics via rigorous human evaluation, and exhibit the usability of KPEVAL through a large-scale evaluation of keyphrase systems including LLM-based methods and keyphrase APIs. Our framework marks the first step towards systematically evaluating keyphrase systems in the era of LLMs. We hope KPEVAL can motivate future works to adopt more accurate evaluation metrics and further advance the evaluation methodology. Future studies might also explore the development of utility metrics tailored to the specific requirements of applications in niche domains.
9
Limitations
While our study sheds light on enhancing the keyphrase evaluation methodology, several limitations exist for KPEVAL.
1. Multilingual Evaluation. We encourage future work to extend the evaluations in this paper to multilingual setting. By design, the aspect and metric formulations in KPEVAL are language-agnostic. For instance, SemF1 can be implemented with multilingual embeddings. Such embeddings need not to keyphrase-specific. For instance, Table 3 suggests that off-the-shelf embeddings cam already have reasonable performance.
2. Alternative Scoring Schemes. KPEVAL’s evaluation and meta-evaluation strategies always target at producing fine-grained numeric scores. This is different from tasks like machine translation where direct assessment scores are annotated (Graham et al., 2013) or LLM competitions that report Elo scores (Wu et al., 2023b). Exploring whether these schemes may provide better evaluation quality is an important question for future work on keyphrase evaluation.
3. LLM-Based Evaluation. Recent works have shown the viability of using LLMs for humanaligning aspect-specific evaluation (Liu et al., 2023). By comprehensively establishing the possible evaluation aspects and curating metaevaluation data for reference agreement and faithfulness, our work sets up the necessary preparations for evaluating LLM-based metrics. We encourage future work to formally define and investigate the performance of keyphrase evaluation metrics based on LLMs.

heading: Acknowledgments
text: The research is supported in part by Taboola, NSF CCF-2200274, and an Amazon AWS credit award. We thank the Taboola team for helpful discussion. We thank Amita Kamath, Wasi Ahmad, as well as other members of the UCLA-NLP group for providing their valuable feedback. We also thank Jingnong Qu, Xiaoxian Shen and Xueer Li for their help in a meta-evaluation pilot study.

heading: A Literature survey: evaluation methods used in recent keyphrase papers
text: We survey all the papers published from 2017 to 2023 in major conferences for AI, NLP, and IR (ACL, NAACL, EMNLP, AAAI, and SIGIR) about keyphrase extraction or keyphrase generation. We choose year 2017 as it marks the start of deep keyphrase generation methods (Meng et al., 2017). We manually check each paper’s experiment sections and note down which of the six major categories do the reported evaluation metrics belong to: (1) precision, recall, and F1 based on exact-matching; (2) diversity metrics, such as duplication ratio; (3) ranking-based metrics such as mAP, α-NDCG, and MRR; (4) approximate versions of exact matching such as n-gram matching; (5) retrieval-based utility metrics; (6) human evaluation. We make sure each of metrics used in the surveyed papers can fall under one and only one category under this ontology.
The survey results are presented in Figure 4. Overall, despite its limitation, exact matching has been de facto the method for assessing the performance of newly proposed keyphrase systems, and there has been limited progress in adopting alternative metrics. The majority of papers report exact matching precision, recall, and F1. Two thirds of all papers use exact matching as the only metric, including 10 out of 11 papers published in 2023. Human evaluation is only conducted in one paper surveyed (Bennani-Smires et al., 2018).

heading: B Keyphrase Systems: Implementation Details and Full Evaluation Results
text: In this section, we describe in detail the considered keyphrase systems as well as how we obtain their outputs for evaluation.
B.1 Keyphrase Systems
We consider three types of keyphrase systems: keyphrase extraction models, keyphrase generation models, and APIs including large language models.
Keyphrase Extraction Systems KPE has traditionally been approached through unsupervised methods, where noun phrase candidates are ranked using heuristics (Hulth, 2003; Mihalcea and Tarau, 2004). Supervised approaches include featurebased ranking (Witten et al., 1999), sequence label-
ing (Zhang et al., 2016), and the use of pre-trained language models (PLMs) for task-specific objectives (Song et al., 2021, 2022). We consider the following nine KPE models: M1 TF-IDF (Jones, 1972) selects the phrases containing words with highest TF-IDF weight. M2 TextRank (Mihalcea and Tarau, 2004) runs
PageRank (Brin and Page, 1998) on an undirected word cooccurrence graph.
M3 RAKE (Rose et al., 2010) is an efficient single-document unsupervised KPE algorithm that uses the cooccurrence graph to score keyphrase candidates. M4 MultipartiteRank (Boudin, 2018) represents the document as a multipartite graph to encode topical diversity and improve intra-topic keyphrase selection preferences. M5 YAKE! (Campos et al., 2020) is an unsupervised KPE method relying on local features such as term co-occurrence and frequencies. M6 Kea (Witten et al., 1999) builds a supervised keyphrase classifier using statistical features including TF-IDF and position information. M7 BERT+CRF (Wu et al., 2022) fine-tunes a pre-trained BERT (Devlin et al., 2019) on sequence labeling with conditional random fields (Lafferty et al., 2001). M8 HyperMatch (Song et al., 2022) trains a supervised model to rank phrase-document relevance in a hyperbolic space. M9 PromptRank (Kong et al., 2023) ranks the
15
phrases by their probability given a prompt prefix using a sequence-to-sequence pretrained language models.
Keyphrase Generation Systems KPG models are often trained with various supervised objectives, including One2One, One2Seq, and One2Set (Meng et al., 2017; Yuan et al., 2020; Ye et al., 2021). A range of strategies have been proposed, including hierarchical modeling of phrases and words (Chen et al., 2020), reinforcement learning (Chan et al., 2019), unifying KPE and KPG (Chen et al., 2019; Ahmad et al., 2021), and using PLMs (Kulkarni et al., 2022). We consider ten KPG models: M10 CatSeq (Yuan et al., 2020) is an RNN trained
with copy mechanism (Gu et al., 2016) on generating keyphrases as a sequence.
M11 CatSeqTG-2RF1 (Chan et al., 2019) introduces an RL-based approach using recall and F1 as the rewards. M12 ExHiRD-h (Chen et al., 2020) extends CatSeq with a hierarchical decoding and an exclusion mechanism to avoid duplications. M13 SEG-Net (Ahmad et al., 2021) unifies keyphrase extraction and keyphrase generation training and introduces layer-wise coverage attention mechanism M14 Transformer (Ye et al., 2021) is a Transformer model (Vaswani et al., 2017) trained with copy mechanism on generating keyphrases as a sequence. M15 SetTrans (Ye et al., 2021) generates keyphrases in parallel based on control codes trained via a k-step target assignment process. M16 SciBERT-G (Wu et al., 2022) fine-tunes SciBERT (Beltagy et al., 2019) for seq2seq keyphrase generation using a prefix-LM objective (Dong et al., 2019). M17 BART-large (Wu et al., 2022) is a BART model (Lewis et al., 2020) fine-tuned on generating keyphrases as a sequence. M18 KeyBART (Kulkarni et al., 2022) is a BART model adapted to scientific keyphrase generation before fine-tuning. M19 SciBART-large+OAGKX (Wu et al., 2022) is pre-trained on scientific corpus and scientific keyphrase generation before fine-tuning.
Large language models and APIs. Recent advancements highlight the capability of large language models (LLMs) to perform in-context learning (Brown et al., 2020). We explore GPT-3.5 (Ouyang et al., 2022) for KPG in the zero-shot
and few-shot prompting setting12. We also assess two commercial keyphrase extraction APIs. M20 Zero-shot prompting GPT-3.5. M21 Five-shot prompting GPT-3.5. M22 Amazon Comprehend API M23 Azure Cognitive Services API
B.2 Implementation Details We obtain the output from the original authors for M8, M10, M11 for KP20k, and M7, M16, M17, M18 for both KP20k and KPTimes. For the other KPE and KPG models, we reproduce the results on our own. For M1, M2, M4, M6, we obtain the outputs using the pke library. For M3, M5, M8, M10, M11, M12 (KPTimes only) and M9, M13, M14, M15, we use the original implementations provided by the authors. For M18, we use the DeepKPG toolkit. For the commercial APIs, we implement the API call following the instructions. We obtained results on 3/5/2023 for M20 and 3/11/2023 for M21. Following existing KPE literature, we consider the top 10 predictions from M1, 2, 3, 4, 5, 6, 8, 9, 22, and 23. For all the systems, we truncate the input to 512 tokens. We perform hyperparameter tuning on the validation sets and ensure that the models match the performance reported by original paper or existing works such as Wu et al. (2022). For M11 and M13 on KPTimes, we failed to obtain reasonable performance and thus choose to omit the results.
For GPT-3.5, we always start the prompt with a task definition: "Keyphrases are the phrases that summarize the most important and salient information in a document. Given a document’s title and body, generate the keyphrases."
In the zero-shot setting, we provide the title and body in two separate lines, and start a new line with "Keyphrases (separated by comma):". In the 5-shot setting, we randomly sample 5 examples from the train set for each test document, and provide their title, body, and keyphrases in the same format in the prompt before the document tested.

heading: C KPEVAL Metrics: Implementation Details and Further Analyses
text: C.1 Phrase Embedding Training Details We fine-tune the paraphrase model provided by Reimers and Gurevych (2019) distributed at https://huggingface.co/ sentence-transformers/all-mpnet-base-v2.
12We use OpenAI’s text-davinci-003 via API.
16
Unsupervised SimCSE (Gao et al., 2021) is used as the training loss. Specifically, given a batch of B phrases, the loss can be expressed as:
Lsimcse = 1B ∑B i=1− log e sim(hi,h ′ i)/τ
∑B j=1 e sim(hi,h ′ j )/τ
,
where hi and h′i are the representations of phrase i obtained using two separate forward passes with dropout enabled. This objective discourages the clustering of unrelated phrases in the representation space and retains a high similarity between semantically related phrase pairs. τ is a scaling factor which we empirically set to 0.05.
We fine-tune the model on Lsimcse using 1.04 million keyphrases from the training set of KP20k, KPTimes, StackEx (Yuan et al., 2020), and OpenKP (Xiong et al., 2019), covering a wide range of domains including science, news, forum, and web documents. We use the AdamW optimizer with maximum sequence length 12, batch size 512, dropout 0.1, and learning rate 1e-6 to fine-tune for 1 epoch. The hyperparameters are determined using a grid search on the following search space: batch size {128, 512, 1024, 2048}, learning rate {1e-6, 5e-6, 1e-5, 5e-5}. We randomly hold out 0.5% from the training data for validation and model selection. The final training takes 30 minutes on a single Nvidia GeForce RTX 2080 Ti GPU.
Remark on embedding quality In Table 8, we provide an additional study on the trained embedding. Specifically, following Gao et al. (2021); Wang and Isola (2020), we evaluate alignment, the average similarity between keyphrases of similar meanings, and uniformity, the average similarity between unrelated keyphrase pairs. For alignment, we utilize the name-variation pairs constructed by Chan et al. (2019). We find that our model achieves the best uniformity, which means that it assigns close to 0 similarity for unrelated pairs. For phrases with similar meanings, it achieves 0.58 alignment, which is also close to human perceptions. Finally, the separation between uniformity and alignment is also the largest for our embedding model.
C.2 Ad-hoc Query Construction for Utility We use GPT-4 (OpenAI, 2023) to annotate three ad-hoc queries per document from KP20k and KPTimes test sets. For both datasets, we sample with temperature set to 0.9 to balance quality and diversity. Due to budget constraints, we sample 1000 documents per dataset to construct the evaluation set. The prompts are presented in Figure 9.
C.3 Inter-metric Correlations
Using the document-level evaluation scores of the 21 keyphrase systems, we calculate the pair-wise Kendall’s Tau for all the metrics in KPEVAL. The results are shown in Figure 5. Overall, we find that only the metrics for the same dimension show a moderate or strong correlation with each other, and the metrics for different aspects hardly correlate. This results suggest that KPEVAL’s aspects measure distinct abilities and that optimizing a single metric does not automatically transfer to a superior performance on the other aspects.
C.4 Order Sensitivity of BertScore
As BertScore is evaluated on two sequences instead of two sets of phrases, previous works concatenate all the predicted phrases as a single string and evaluate against all the references concatenated together (Koto et al., 2022; Glazkova and Morozov, 2022). However, it is unclear how to order the prediction and reference keyphrases within these two strings and whether BertScore’s performance is sensitive to this ordering or not. We conduct phrase-level meta-evaluation of BertScore with phrase-level permutation applied to the matching target. Specifically, we shuffle the labels and the references before calculating meta-evaluation metrics. We repeat this process for 100 times and report the mean and standard deviation of human correlation in Table 9. Overall, we find that the metric-human correlation of BertScore is relatively insensitive to permuta-
17
Se mP Se mR Se mF 1
du p_t
oke n_r
ati o
em be
d_s im
Fai thf
uln ess RR @5
Re cal
l@ 5
SemP
SemR
SemF1
dup_token_ratio
embed_sim
Faithfulness
RR@5
Recall@5
1 0.39 0.72 0.15 -0.017 0.18 0.038 0.061
0.39 1 0.67 0.066 0.028 -0.057 0.034 0.052
0.72 0.67 1 0.12 0 0.072 0.043 0.067
0.15 0.066 0.12 1 0.092 -0.039 0.018 0.038
-0.017 0.028 0 0.092 1 0.46 -0.027 -0.025
0.18 -0.057 0.072 -0.039 0.46 1 0.021 0.016
0.038 0.034 0.043 0.018 -0.027 0.021 1 0.73
0.061 0.052 0.067 0.038 -0.025 0.016 0.73 1
KP20k
Se mP Se mR Se mF 1
du p_t
oke n_r
ati o
em be
d_s im
Fai thf
uln ess RR @5
Re cal
l@ 5
SemP
SemR
SemF1
dup_token_ratio
embed_sim
Faithfulness
RR@5
Recall@5
1 0.48 0.79 0.11 0.13 0.23 -0.018 -0.014
0.48 1 0.69 0.021 0.11 0.06 0 0.006
0.79 0.69 1 0.072 0.12 0.17 -0.017 -0.011
0.11 0.021 0.072 1 0.088 0.041 0.066 0.056
0.13 0.11 0.12 0.088 1 0.21 0.004 -0.002
0.23 0.06 0.17 0.041 0.21 1 -0.093 -0.1
-0.018 0 -0.017 0.066 0.004 -0.093 1 0.82
-0.014 0.006 -0.011 0.056 -0.002 -0.1 0.82 1
KPTimes
1.00
0.75
0.50
0.25
0.00
0.25
0.50
0.75
1.00
Figure 5: Correlation between KPEVAL’s metrics, measured by Kendall’s Tau. The diversity scores are negated to provide a more intuitive view. We find that only the metrics for the same dimension correlate with each other.
tions: when given reference phrases or prediction phrases concatenated in different orders, BertScore maintains a similar evaluation quality. One notable pattern is that when the references and the predictions are not permuted, BertScore obtains slightly higher performance. We hypothesize that in this case many phrases may present in the same order in the reference and the prediction, making the exactly matched instances easier to distinguish.
C.5 Variation in Keyphrase Annotations Motivates Semantic Matching
A major motivation for semantic matching is that valid predictions vary in many ways. But at the same time, do human references also exhibit lexical variations? We investigate with a pilot study of model-in-the-loop keyphrase annotation.
Setup We sample 100 documents each from the test sets of KP20k and KPTimes and combine each document’s Y with P from four systems: M8, M10, M15, and M18 (KPTimes only)/M19 (KP20k only). Three MTurk annotators are presented with the document and the phrases re-ordered alphabetically. They are then asked to write keyphrases that best capture the salient information. We state that they may select from the provided keyphrases or write new keyphrases. Figure 12 presents the annotation interface. We use the same set of annotators in appendix D and collect 3226 keyphrase annotations, which approximately cost $700.
Keyphrase annotations exhibit lexical variations. Figure 6 presents the distribution of phrase se-
lected by the annotators from each source. The reported counts are averaged over three annotators. Surprisingly, we find that the keyphrases in the original labels are not preferred over the outputs from other models. First, nearly 70% keyphrases in the original labels are not selected. Second, the annotators select several keyphrases from each model’s outputs that do not appear in the label set. For KP20k, the annotators even select more such keyphrases compared to the phrases from the labels. This suggests that label variations can be common in keyphrase annotations, even if candidate keyphrases are given as a guidance.
Are the observed label variations caused by annotators writing entirely different concepts? We find that the average character-level edit distance between the selected phrases and the closest phrase in label keyphrases is 11.0 for KP20k and 7.0 for KPTimes, much smaller than the metric for phrases that are not selected (17.5 for KP20k and 14.6 for KPTimes). In other words, keyphrases written by different humans are lexically similar to the origi-
18
nal references, with slightly different phrasings.
Semantic matching is more robust to label variations. A desired property of keyphrase metrics is outputting consistent scores with similar sets of labels. Using the 200 annotated documents, we compare F1@M and SemF1 on the outputs from M20 with four sets of labels13: the original labels and three sets of newly annotated labels. As shown in Table 10, the output of SemF1 is more stable using different label sets, indicating a higher robustness and reliability compared to F1@M .

heading: D Annotation for Meta-Evaluation
text: Annotator Hiring For all the annotation experiments, we use Amazon Mechanical Turk (MTurk) and designed a qualification task to hire and train annotators. We require the annotators to be located in United States or United Kingdom and have finished more than 1000 HITs with 97% pass rate. In the qualification task, the annotators are presented with the definition of semantic matching with examples, and then asked to annotate three documents. 46 annotators that have average ≤ 1.5 wrong annotations per document are selected. We ensure that the purpose of the main tasks and how we use the annotations are clearly explained in the qualification task to the potential annotators.
Cost For reference agreement, a total of 1500 document-level annotations with 13401 phraselevel evaluations are collected from the qualified annotators, costing approximately $1400. For faithfulness, we collect 6450 phrase-level annotations for KP20k and 6486 annotations for KPTimes, costing approximately $800. We adjust the unit pay to ensure $15 hourly pay.
13We choose M20 as many supervised models’ outputs largely overlap with those included in the annotation process.
Interface The annotation instructions and interface for reference agreement are presented in Figure 10 and Figure 11. The interface for faithfulness is presented in Figure 13.

heading: E Qualitative Study
text: E.1 SemF1 vs. existing metrics In Figure 7, we present a qualitative example where the model (M21) predicts two keyphrases exactly matching to some reference keyphrases and two keyphrases semantically similar to the reference. Correspondingly, human annotators assign partial credits to both of the "near-misses". However, for these phrases, exact matching gives a score 0 and substring matching gives full credit. As the reference and the prediction contain many similar tokens, BertScore is also near 1.0. Semantic matching’s scoring is the most similar to humans.
E.2 Rethinking model comparisons In Figure 8, we compare ExHiRD-h and CatSeqTG2RF1 on two instances from KP20k. When evaluated with exact matching, CatSeqTG-2RF1 is preferred. However, since a lot of correct keyphrases are not recognized by exact matching, the irrelevant concepts and duplications are under-penalized by F1@M . By contrast, these issues are identified by the metrics from KPEVAL dedicated to faithfulness and diversity. With semantic-based evaluation, KPEVAL suggests that ExHiRD-h outperforms CatSeqTG-2RF1 on these two instances.
19
20
21
22
23

heading: Future Work
text: We hope KPEVAL can motivate future works to adopt more accurate evaluation metrics and further advance the evaluation methodology. Future studies might also explore the development of utility metrics tailored to the specific requirements of applications in niche domains. 9
Limitations
While our study sheds light on enhancing the keyphrase evaluation methodology, several limitations exist for KPEVAL. 1. Multilingual Evaluation. We encourage future work to extend the evaluations in this paper to multilingual setting. By design, the aspect and metric formulations in KPEVAL are language-agnostic. For instance, SemF1 can be implemented with multilingual embeddings. Such embeddings need not to keyphrase-specific. For instance, Table 3 suggests that off-the-shelf embeddings cam already have reasonable performance. 2. Alternative Scoring Schemes. KPEVAL’s evaluation and meta-evaluation strategies always target at producing fine-grained numeric scores. This is different from tasks like machine translation where direct assessment scores are annotated (Graham et al. , 2013) or LLM competitions that report Elo scores (Wu et al. , 2023b). Exploring whether these schemes may provide better evaluation quality is an important question for future work on keyphrase evaluation. 3. LLM-Based Evaluation. Recent works have shown the viability of using LLMs for humanaligning aspect-specific evaluation (Liu et al. , 2023). By comprehensively establishing the possible evaluation aspects and curating metaevaluation data for reference agreement and faithfulness, our work sets up the necessary preparations for evaluating LLM-based metrics. We encourage future work to formally define and investigate the performance of keyphrase evaluation metrics based on LLMs