Hybrid Matching Model for Open Domain Web Keyphrase Extraction (2024)

Abstract: Keyphrase extraction aims to automatically extract salient phrases representing the critical information in the source document. Identifying salient phrases is challenging because there is a lot of noisy information in the document, leading to wrong extraction. To address this issue, in this paper, we propose a hybrid matching model for keyphrase extraction, which combines representation-focused and interactionbased matching modules into a unified framework for improving the performance of the keyphrase extraction task. Specifically, HybridMatch comprises (1) a PLM-based Siamese encoder component that represents both candidate phrases and documents, (2) an interactionfocused matching (IM) component that estimates word matches between candidate phrases and the corresponding document at the word level, and (3) a representation-focused matching (RM) component captures context-aware semantic relatedness of each candidate keyphrase at the phrase level. Extensive experimental results on the OpenKP dataset demonstrate that the performance of the proposed model HybridMatch outperforms the recent state-of-the-art keyphrase extraction baselines. Furthermore, we discuss the performance of large language models in keyphrase extraction based on recent studies and our experiments.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 17–27 August 11-16, 2024 ©2024 Association for Computational Linguistics
Keyphrase extraction aims to automatically extract salient phrases representing the critical information in the source document. Identifying salient phrases is challenging because there is a lot of noisy information in the document, leading to wrong extraction. To address this issue, in this paper, we propose a hybrid matching model for keyphrase extraction, which combines representation-focused and interactionbased matching modules into a unified framework for improving the performance of the keyphrase extraction task. Specifically, HybridMatch comprises (1) a PLM-based Siamese encoder component that represents both candidate phrases and documents, (2) an interactionfocused matching (IM) component that estimates word matches between candidate phrases and the corresponding document at the word level, and (3) a representation-focused matching (RM) component captures context-aware semantic relatedness of each candidate keyphrase at the phrase level. Extensive experimental results on the OpenKP dataset demonstrate that the performance of the proposed model HybridMatch outperforms the recent state-of-the-art keyphrase extraction baselines. Furthermore, we discuss the performance of large language models in keyphrase extraction based on recent studies and our experiments.

heading: 1 Introduction
text: Keyphrase Extraction (KE) is a fundamental task in natural language processing that aims to identify a series of salient and representative phrases relevant to the primary information discussed in a source document. Specifically, as shown in Table 1, keyphrases are concise content that encapsulates the central semantic meaning of a longer text (Hasan and Ng, 2014; Song et al., 2023b). Keyphrases play a crucial role in various natural
* Corresponding Author
language processing and information retrieval applications (Song et al., 2021a, 2022b, 2023h,a). Keyphrases can generally be categorized into two types: present keyphrases, which appear in the document, and absent keyphrases, which do not.
Keyphrase extraction models usually consist of two components: candidate keyphrase extraction, which selects candidate keyphrases from the given document by some heuristics (Liu et al., 2009; Grineva et al., 2009; Medelyan et al., 2009; Liang et al., 2021), and keyphrase importance estimation, which determines whether a candidate keyphrase is a true keyphrase via unsupervised (Mihalcea and Tarau, 2004; Liang et al., 2021) or supervised approaches (Mu et al., 2020; Sun et al., 2020; Song et al., 2021b). Most existing supervised keyphrase extraction approaches have focused on estimating the importance of keyphrases effectively and adequately, such as the recent studies (Sun et al., 2020; Wang et al., 2020; Song et al., 2021b).
17
Although recent state-of-the-art keyphrase extraction approaches have made significant progress, they ignore explicitly modeling the relevance between candidate phrases and their corresponding document. Consequently, Song et al. (2022a) explicitly models the phrase-document relevance by using the Poincaré distance in the hyperbolic space, further improving the performance of keyphrase extraction, as presented in Figure 1 (a). However, a document often contains redundant information and typically has more complex semantics than each candidate phrase. Therefore, directly calculating the relevance between the entire candidate keyphrases and their corresponding document may lead to wrong keyphrase extraction.
Inspired by the issue above, we propose a new hybrid matching model, HybridMatch, to enhance the performance of the keyphrase extraction task. HybridMatch first evaluates the importance scores of candidate phrases from multiple granularities and then aggregates these scores into a final importance score for ranking and extracting keyphrases. Specifically, HybridMatch comprises a PLM-based Siamese encoder, an interaction-focused matching module(IM), and a representation-focused matching module (RM). The PLM-based Siamese encoder is employed to encode candidate phrases and their corresponding documents. The IM module estimates the phrase-document relevance at the word level, providing a surface-level importance score for each candidate keyphrase. Meanwhile, the RM module assesses the phrase-document similarity to capture the semantic relatedness at the phrase level, yielding high-level importance scores for the candidate keyphrases. Finally, we combine these two matching scores into a total score and utilize
a ranking model to train the entire system. Extensive experimental results on the OpenKP dataset demonstrate the effectiveness of our model by comparing with recent state-of-the-art baselines. To summarize, our contributions are fourfold:
• A hybrid matching model is proposed to estimate the importance of candidate keyphrases from different perspectives by taking advantage of both interaction- and representationfocused matching models.
• An interaction-focused matching module estimates the relevance scores between candidate phrases and their corresponding document at the word level.
• A representation-focused matching module captures context-aware semantic relatedness between candidate keyphrases and their corresponding document.
• Extensive experiments on the OpenKP dataset show that HybridMatch outperforms the recent state-of-the-art keyphrase extraction baselines. An ablation test is performed to demonstrate the superiority of each component.
The rest of this paper is organized into six Sections. Among them, Section 2 describes the proposed HybridMatch model. The experimental setting and results are discussed in Section 3 and Section 4. Section 5 discusses the related work. A brief conclusion is given in Section 6.

heading: 2 Methodology
text: In this section, we describe the architecture of HybridMatch. Figure 2 shows the proposed model HybridMatch, consisting of three components: a PLMbased Siamese encoder, an interaction-focused matching module, and a representation-focused matching module. In the following sections, the details of HybridMatch are given.

heading: 2.1 Problem Definition
text: Traditional keyphrase extraction typically can be formulated as follows: given the source document D, the keyphrase extraction model aims at obtaining a keyphrase set P . In this paper, however, we treat the keyphrase extraction task as a text matching problem, which can be reformulated as follows: given a candidate keyphrase p and its corresponding document D, the text matching model aims to compute and learn a matching score f(D, p) using the candidate phrase terms {xp1, ..., xpn} and document terms {xD1 , ..., xDM}, where n and M are the number of terms in p and D respectively. To be clear, "term" refers to either a single word or aggregating all words in phrase p or document D.

heading: 2.2 Siamese Encoder
text: In this section, we describe the contextualized text encoder. To minimize information loss, we represent phrases and documents at varying levels of granularity.

heading: 2.2.1 Document Representation
text: To embed the given document D, we adopt the pre-trained language model (RoBERTa (Liu et al., 2019)) as the backbone to obtain word representations H = {hcls,h1,h2, ...,hm, ...,hM},
H = RoBERTa(D), (1)
where hm ∈ Rd is the word representation of the m-th word xm in the document D. Here, hcls is the representation of the [CLS] token of the pretrained language model RoBERTa, which indicates the whole document representation.

heading: 2.2.2 Candidate Keyphrase Representation
text: As previously mentioned, we treat keyphrase extraction as a matching task involving creating a set of candidate keyphrases from the given document, similar to the two-stage keyphrase extraction methods (Sun et al., 2020; Song et al., 2021b). In this section, we provide details on how we construct the candidate keyphrase set for the given document.
Based on the RoBERTa, we can obtain the output representation H. Concretely, to allow for a better collection of candidate keyphrase representations, we only retain tokens corresponding to each word in the source document for word-level document representation H, thus removing some special symbols (i.e., [CLS] and paddings). After the above operations, we can obtain each word representation Ĥ = {h1,h2, ...,hM} ∈ RM×d of the given document, and M indicates the actual length of the document D. Next, our model composes words to n-grams, where hnm:m+n indicates the m-th ngram candidate phrase representation (contains n words). Specifically, we use hnm:m+n as the wordlevel phrase representation. Furthermore, we also aggregate the word-level phrase representation to a whole phrase representation as follows,
hnm = CNN n(hnm:m+n)Wc, (2)
where Wc is the learnable matrix and CNNn indicates the Convolutional Neural Network with the kernel size of n ∈ [1, N ]. Here, N denotes the maximum length of the extracted candidate keyphrases.

heading: 2.3 Interaction-focused Matching Module
text: Since keyphrases are frequently used to retrieve documents in various real-world applications, a robust keyphrase extraction system should comprehensively model and leverage the relevance of candidate keyphrases to their respective documents. However, documents often contain redundant information. To address this, we propose an interaction-focused matching module that captures matching signals at the word level, as illustrated in Figure 3. Specifically, we calculate the relevance score between each candidate keyphrase and the corresponding document to obtain the word-level phrase-document relevance scores.
To obtain the word-level phrase-document relevance scores, we calculate the textual similarity between candidate phrases and their corresponding document by multiplying the word-level candidate
phrase representation hnm ∈ Rn×d and the wordlevel document representation Ĥ ∈ RM×d,
Spp = hnm:m+nĤ ⊤,Spp ∈ Rn×M , (3)
Sppi,j = softmax(S pp i,j) =
eS pp i,j ∑m k=1 e Sppi,k , (4)
where Sppi,j can be considered the word-level similarity score by matching the i-th word representation of the candidate keyphrase with the j-th word representation of the document. Specifically, since the candidate keyphrase and the document are embedded in the same pre-trained language model (RoBERTa), similar words will be placed closer in the semantic space, and their product will produce larger scores.
Then, we apply max, mean pooling to the similarity matrix to obtain discriminative feature vectors:
Max(Spp) = [Max(Spp1,:), ...,Max(S pp n,:)], (5) Mean(Spp) = [Mean(Spp1,:), ...,Mean(S pp n,:)], (6)
where Max(Spp) ∈ Rn, Mean(Spp) ∈ Rn indicate the matching signals of each candidate keyphrases. Each score generated from pooling can be viewed as one piece of matching evidence for a specific query term or phrase to the document, and its value denotes the importance of the relevance signal. Compared to max pooling, mean pooling is beneficial for cases where a phrase is matched to multiple relevant terms in the document as follows,
rpp = Mean([Max(Spp);Mean(Spp)]), (7)
where [; ] indicates the concatenation of two vectors and rpp ∈ R1×2 indicates the word-level phrasedocument relevance matching signals. In addition, when two keyphrases have similar semantics, the keyphrase with more words often gets a higher score. Therefore, the word-level matching module may alleviate such problems.

heading: 2.4 Representation-focused Matching Module
text: The word-level phrase-document relevance matching typically focuses on the local view of salient information in each candidate keyphrase, often overlooking valuable content that could be used for extracting keyphrases from the document. In contrast, representation-focused matching emphasizes "meaning" correspondences by leveraging linguistic information (such as words, phrases, and entities) and compositional structures (like dependency
trees). Therefore, in this paper, we first introduce a representation-focused matching (RM) approach to address these limitations, which captures semantic similarities through attention mechanisms applied to candidate phrases and their corresponding document representations.
To obtain the phrase-level phrase-document relevance matching scores, we calculate the textual similarity between the candidate phrase and the document by multiplying the phrase-level candidate phrase representation hnm ∈ Rn×d and the word-level document representation Ĥ ∈ RM×d,
Spw = hnmĤ ⊤,Spw ∈ R1×M , (8) rpw = softmax([Max(Spw);Mean(Spw)]), (9)
where Spw represents the phrase-level relevance of the whole candidate keyphrase to all words in the document and rpw ∈ R1×2 denotes the phraselevel phrase-document relevance matching scores.

heading: 2.5 Matching Scores Aggregation
text: Usually, the importance of words in a sentence depends on the importance of the sentence as a whole (Hsu et al., 2018). Similarly, the importance of words in a phrase depends on whether the phrase as a whole is essential. Therefore, we further influence the word-level phrase-document relevance by using the phrase-level phrase-document relevance as a weighting factor as the relevance matching score as follows,
rnm = Mean(r pp ⊙ rpw), (10)
where rnm ∈ R1×1 denotes the final score and ⊙ indicates an element-wise product. This multiplica-
tion ensures that the updated word-level relevance score can be high only when both word-level and phrase-level relevance scores are high. Furthermore, this weighting method reduces the impact of high matching scores from common words like stopwords. Our relevance matching has two essential properties. First, all the operations, including matching, softmax, pooling, and weights, have no learnable parameters. Second, the parameter-free nature makes our model more interpretable and robust from over-fitting. Finally, the importance score of each candidate keyphrase can be calculated as follows,
f(D, p) = λ1FNNh(hnm) + λ2r n m, (11)
where λ1 and λ2 are used to balance the influence of two matching modules, FNNh(·) is a 2-layer feed-forward network, and f(D, p) indicates the relevance matching score and will be used for ranking in the training procedure.

heading: 2.6 Training and Inference
text: To train the whole model, the pairwise hinge loss is adopted for training and optimizing the model parameters, which is widely used in the field of NLP and IR, formulated as,
Lrank = max(0, δ−f(D, p+)+f(D, p−)), (12) where Lrank is the pairwise loss based on a triplet of the document D, a relevant (positive) keyphrase sample p+, and an irrelevant (negative) keyphrase sample p−. Following the previous studies (Sun et al., 2020; Song et al., 2021b, 2022a, 2023h), we use the same approach to select positive and negative samples from all candidates.

heading: 3 Experimental Settings
text: In this section, we first introduce the used dataset. Then, we present the evaluation metric, baselines, and implementation details.

heading: 3.1 Datasets
text: The OpenKP1 dataset is an open-domain keyphrase extraction dataset encompassing a wide range of domains and topics. It includes over 150,000 realworld online pages, each annotated with the most relevant key terms by experts. In this paper, we utilize the OpenKP dataset as our primary benchmark, adhering to its official splits: 134,000 documents for training, 6,600 documents for development, and 6,600 documents for testing.
1https://github.com/microsoft/OpenKP

heading: 3.2 Evaluation Metrics
text: Following previous studies (Xiong et al., 2019; Sun et al., 2020; Song et al., 2021b, 2023h), Precision (P), Recall (R), and F-measure (F1) of the top K keyphrase predictions are used for evaluating the performance of keyphrase extraction approaches. Specifically, we adopt K = 1, 3, 5 on the OpenKP dataset.
Furthermore, we use Porter Stemmer2 (Porter, 2006) to determine the match of two phrases, which is consistent with prior studies (Meng et al., 2017; Xiong et al., 2019).

heading: 3.3 Implementation and Training Details
text: We adopt the pre-trained language model RoBERTa as the backbone of our model, initialized from their pre-trained weights. Our models are optimized using AdamW (Li and Li, 2018) with a 5e-5 learning rate and 10% warm-up proportion. Here, we set the batch size to 64 and the maximum sequence length to 512. Following the previous work (Sun et al., 2020; Song et al., 2021b), the maximum phrase length is set to 5 (N = 5). The training process of our model uses eight RTX A4000 GPUs. In addition, our code is implemented based on PyTorch 1.83. Table 2 shows the details.

heading: 3.4 Baselines
text: We compare HybridMatch with two types of models to comprehensively prove the effectiveness of our models. We first compare traditional unsupervised methods TF-IDF (Jones, 2004) and TextRank (Mihalcea and Tarau, 2004). Then, we compare three kinds of keyphrase extraction methods that utilize variants of the pre-trained language models as the backbone to extract keyphrases by spanning (Sun et al., 2020), chunking (Sun et al., 2020), ranking (Sun et al., 2020), multi-task learning(Song et al., 2021b), and hyperbolic deep learning (Song et al., 2022a). There are also no additional features available in many real applications, such as visual features. Meanwhile, neither the above selection baselines nor the proposed model utilizes additional feature information to assist in estimating the importance of candidate phrases, so BLING-KPE (Xiong et al., 2019) and SMART-KPE+R2J (Wang et al., 2020) are not selected as baselines.
2https://tartarus.org/martin/PorterStemmer/ 3https://pytorch.org/

heading: 4 Results and Analysis
text: In this section, we show the results of the baselines and our proposed models. We also show the analyses of HybridMatch in different scenarios, including an ablation test and a case study. In addition, we also discuss the performance of large language models in keyphrase extraction based on recent studies and our experiments.

heading: 4.1 Overall Performance
text: Our primary experimental results on the OpenKP dataset are presented in Table 3, and the best results are underlined. Specifically, for the OpenKP dataset, F1@3 is the primary evaluation metric4. As shown in Table 3, we can see that our model HybridMatch achieves better performance on all evaluation metrics than the previous state-of-the-art baselines. This proves that considering both interactionfocused and representation-focused matching can extract keyphrases more accurately. Meanwhile, the proposed models beat other ranking-based competitive baselines (RankKPE-BERT and RankKPERoBERTa) by a large margin and are still superior to those baselines (SpanKPE-BERT, SpanKPERoBERTa, ChunkKPE-BERT, and ChunkKPERoBERTa). Furthermore, this finding further suggests that filtering the redundant information and selecting the saliency from multiple views is beneficial for keyphrase extraction.
4https://microsoft.github.io/msmarco/

heading: 4.2 Ablation Test
text: To better understand the contribution of each module in our proposed model, we conducted an ablation study on the base HybridMatch model by systematically removing each component. We aimed to examine how the interaction-focused matching and representation-focused matching modules contribute to the model’s effectiveness. The results on the OpenKP dataset are presented in Table 4, with each row indicating the removal of a specific module. For instance, the row labeled "HybridMatchIM" represents the model with the representationfocused matching module removed.
The results demonstrate that each component of our model (HybridMatch-IM and HybridMatchRM) outperforms the compared baselines. This is because, unlike previous baselines, our model considers multiple perspectives to select important keyphrases from the document.
Moreover, combining the two kinds of matching modules significantly enhances the performance of keyphrase extraction. This improvement is attributed to integrating different matching signals, which leads to a more accurate estimation of the importance scores of each candidate keyphrase.

heading: 4.3 Case Study
text: In this section, we show an example from the OpenKP dataset in Figure 5. We compare the extraction results of our three models, including HybridMatch, HybridMatch-IM, and HybridMatchRM. Concretely, the output keyphrases extracted by the HybridMatch-IM model are more inclined to extract the content that appears several times in the given document or the content that appears more frequently. For the output keyphrases extracted by
the HybridMatch-RM model, it is more inclined to extract keyphrases that match the semantics of the given document. Overall, the HybridMatch model combines the advantages of both IM and RM modules to focus on both semantically important and frequently occurring keyphrases and ignore the extraction of interfering items such as stop words. This example shows that the joint modeling of semantic and relevance matching can improve the performance of keyphrase extraction.

heading: 4.4 LLMs for Keyphrase Extraction
text: Large Language Models (LLMs) have achieved good results in many natural language processing downstream tasks. However, in previous studies (Song et al., 2023c,e), getting good results in keyphrase extraction or generation tasks is not easy
for LLMs, which may be inferior to small models trained on specific domain data. Therefore, in this paper, we also test the performance of LLMs on the keyphrase extraction task, i.e., GPT-4-Turbo. Unlike previous work (Song et al., 2023c), we enrich and improve the prompt for extracting keyphrases, as shown in Table 6. The results show that even GPT-4-Turbo does not meet expectations in the keyphrase extraction task.
Based on the issue mentioned earlier, we suppose that the most likely reason is that the annotation results in the existing keyphrase extraction or generation datasets are unreliable or of low quality. Therefore, referring to Zheng et al. (2023), we adopt GPT-4-Turbo to evaluate the annotation information in the validation set of the OpenKP dataset via the prompt shown in Table 7. The evaluation results are shown in Figure 4. Indeed, the results align with our hypothesis. The scores evaluated by GPT-4-Turbo mostly fall between 2 and 3, indicating potential issues with the annotations
of the OpenKP dataset. We also selected three unsupervised keyphrase extraction datasets (e.g., DUC2001 (Wan and Xiao, 2008b), Inspec (Hulth, 2003), SemEval2010 (Kim et al., 2010)) for testing the performance of GPT-4-Turbo and compared them with a SOTA prompt-based keyphrase extraction method (i.e., PromptRank (Kong et al., 2023)). From the results in Table 9, we can see that there are still many differences in the effects of GPT-4-Turbo on different keyphrase extraction benchmarks. Therefore, in the future, unlocking the potential of LLMs to improve the performance of the keyphrase extraction task, utilizing LLMs to build a new benchmark for the keyphrase extraction and generation tasks, or correcting the annotations of existing benchmarks using some leading LLMs may be valuable research directions.

heading: 5 Related Work
text: Typically, keyphrase extraction aims to automatically extract a set of phrases related to the primary information discussed in the source document (Tomokiyo and Hurst, 2003; Hasan and Ng, 2014; Meng et al., 2017; Song et al., 2023b). Specifically, recent keyphrase extraction baselines can be mainly
divided into two components: candidate keyphrase extraction and keyphrase importance estimation. The former uses heuristics to select a candidate keyphrase set (Grineva et al., 2009; Mihalcea and Tarau, 2004; Song et al., 2022c; Witten et al., 1999; Wan and Xiao, 2008a; Medelyan et al., 2009; Song et al., 2023f,d,g,i) for the given document, and the latter is to measure the importance of candidates for extracting keyphrases (Sun et al., 2019; Zhang et al., 2020; Sun et al., 2020; Wang et al., 2020; Song et al., 2021b, 2022a, 2023h).
Unlike the existing methods, in this paper, we combine interaction-focused and representationfocused matching into a unified matching model for extracting keyphrases.

heading: 6 Conclusion
text: In this paper, we propose a new hybrid matching model (HybridMatch) to combine representationfocused and interaction-focused matching modules into a unified model. To filter the noisy information, we first design an interaction-focused matching module and a representation-focused matching module to estimate the important scores of candidate phrases from different granularities. Then, we aggregate the relevance matching scores from IM and RM as the importance scores to rank and extract keyphrases. Extensive experiments demonstrate the effectiveness of our model, and ablation studies verify the performance of each component. In addition, we discuss the performance of large language models in keyphrase extraction based on recent studies and our experiments. In the future, it will be interesting to introduce a graph-based network to capture the interactions from multiple granularities to calculate the phrase-document relevance as the importance of each candidate for improving the performance of keyphrase extraction.

heading: 7 Limitations
text: There are still some limitations to our work. In this paper, we follow previous research and generate candidate keyphrases using the n-gram rule, which may introduce many noisy or irrelevant keyphrases. Therefore, improving the procedure for extracting candidate keyphrases may be interesting and valuable in improving the upper bound of keyphrase extraction performance. One possible approach is to prune candidate keyphrases from the source document by utilizing surface-level information of candidate keyphrases.

heading: 8 Acknowledgments
text: We thank the three anonymous reviewers for carefully reading our paper and their insightful comments and suggestions.
This work was partly supported by the National Natural Science Foundation of China under Grant 62176020; the Joint Foundation of the Ministry of Education for Innovation team (8091B042235); the Beijing Natural Science Foundation under Grant L211016; the Fundamental Research Funds for the Central Universities (2019JBZ110); and the State Key Laboratory of Rail Traffic Control and Safety (Contract No. RCS2023K006), Beijing Jiaotong University.

heading: Future Work
text: In the future, it will be interesting to introduce a graph-based network to capture the interactions from multiple granularities to calculate the phrase-document relevance as the importance of each candidate for improving the performance of keyphrase extraction