Improving Embedding-based Unsupervised Keyphrase Extraction by Incorporating Structural Information
Year: 2023

Abstract:
Keyphrase extraction aims to extract a set of keyphrases with the central idea of the document. In a structured document, there are certain locations (e.g., the title or the first sentence) where a keyphrase is most likely to appear. However, when extracting keyphrases from the document, most existing embedding-based unsupervised keyphrase extraction models ignore the indicative role of the highlights in certain locations, leading to wrong keyphrases extraction. In this paper, we propose a new HighlightGuided Unsupervised Keyphrase Extraction model (HGUKE) to address the above issue. Specifically, HGUKE first models the phrasedocument relevance via the highlights of the documents. Next, HGUKE calculates the crossphrase relevance between all candidate phrases. Finally, HGUKE aggregates the above two relevance as the importance score of each candidate to rank and extract keyphrases. The experimental results on three benchmarks demonstrate that HGUKE outperforms the state-of-the-art unsupervised keyphrase extraction baselines.

heading: 1 Introduction
text: Keyphrase extraction is the fundamental task of automatically extracting a set of salient phrases from a document that concisely describes its primary content (Hasan and Ng, 2014; Song et al., 2023a). Figure 1 shows an example of the source document and its corresponding keyphrases.
Recent developments in pre-trained language models (Devlin et al., 2019) have heightened the need for utilizing pre-trained embeddings on natural language processing tasks, which significantly improves the performance of embedding-based unsupervised keyphrase extraction models (Sun et al., 2020; Liang et al., 2021; Zhang et al., 2022). Existing embedding-based models mainly consist of two components: candidate keyphrase extraction and keyphrase importance estimation (Hasan and
∗Corresponding Author
Ng, 2014; Song et al., 2021, 2022a). The former extracts continuous words from the document as candidate keyphrases through heuristic rules, and the latter estimates the importance of candidate phrases by matching similarity with their corresponding document.
Generally, the source document has both salient information and noises (redundant content). Hence, there may be a deviation when directly using the phrase-document relevance as the importance score of each candidate to select keyphrases. For many specific-domain documents (e.g., news or scientific articles), the highlights (the title or the first sentence) typically contains the central information of the source document (as shown in Figure 1), which has more significant guidance for extracting keyphrases. However, the recent embedding-based unsupervised keyphrase extraction models ignore the effect of the highlight information, leading to extract wrong keyphrases.
Motivated by the above issues, we propose a new Highlight-Guided Unsupervised Keyphrase Extraction model (HGUKE), which estimates the impor-
1041
tance score of each candidate phrase by jointly considering the global and local relevance between candidate phrases and their corresponding document. Concretely, HGUKE first calculates the global relevance by leveraging the highlights rather than the whole document and then locally computes the cross-phrase relevance between all candidate keyphrases, as illustrated in Figure 2. Finally, HGUKE aggregates the global and local relevance as the importance score of each candidate keyphrase to rank and extract keyphrases. Experimental results demonstrate that the proposed model HGUKE outperforms the recent state-ofthe-art embedding-based unsupervised keyphrase extraction baselines on three benchmark keyphrase extraction datasets.

heading: 2 Methodology

heading: 2.1 Candidate Keyphrase Extraction
text: To extract candidate keyphrases from the source document, we follow the previous studies (Liang et al., 2021; Song et al., 2022b; Ding and Luo, 2021) and leverage Stanford CoreNLP Tools1 for tokenizing, part-of-speech tagging and noun phrase chunking. Concretely, in our model, the regular expression {< NN. ∗ |JJ > ∗ < NN.∗ >} is designed to extract noun phrases as the candidate keyphrases via the python package NLTK2.

heading: 2.2 Phrase and Document Encoding
text: After constructing a set of candidate keyphrases P = {p1, ..., pi, ..., p|P|} for the source document via the above method, we adopt the pre-trained language model BERT (Devlin et al., 2019) as the embedding layer to obtain pre-trained word embeddings H = {h1, ..., hm, ..., h|D|} for the source document D = {w1, ..., wm, ..., w|D|} where hm indicates the m-th word in the document.
1https://stanfordnlp.github.io/CoreNLP/ 2https://github.com/nltk
Next, we leverage word embeddings to obtain candidate keyphrase representations. To capture the central semantics of the candidate keyphrases, we obtain candidate keyphrase representations by leveraging the max pooling operation, which is a simple and effective parameter-free approach and can be calculated as follows,
hpi = Max-Pooling({h1, ..., hk, ..., h|pi|}), (1)
where hpi is the representation of the i-th candidate keyphrase and |pi| indicates the length of pi. Specifically, hk represents the word in the document associated with the candidate keyphrase pi. At the same time, we use the mean pooling operation to obtain the highlight representation hs of the document.

heading: 2.3 Phrase-Document Relevance
text: To obtain more relevant candidates, we model the similarity between candidate phrases and the corresponding document as follows,
phi = 1
||hs − hpi ||1 , (2)
where phi denotes the phrase-document relevance of i-th candidate keyphrases and ||·||1 indicates the Manhattan Distance.
For news and scientific articles, keyphrases often appear at the beginning or front position (Florescu and Caragea, 2017a,b), which means that the position information is important and indicative for extracting keyphrases. For example, the word appearing at 2-th, 5-th and 10-th, has a weight ρi = 1/2+1/5+1/10 = 0.8. Inspired by the previous work (Florescu and Caragea, 2017b; Liang et al., 2021), we adopt a position regularization as follows, ρi = softmax(e1/i), where ρi is the position regularization factor of the i-th candidate phrase. Then, the weighted phrase-document relevance p̂hi can be re-calculated as follows,
p̂hi = p h i · ρi, (3)
Here, we finally employ p̂hi to estimate the phrasedocument relevance of the i-th candidate phrase.

heading: 2.4 Cross-Phrase Relevance
text: Generally, the phrase-document relevance is calculated between the highlight information and each candidate independently, and consequently, it cannot determine which candidates are better than the
others. To determine which candidate phrases are more salient than the others, we sum the semantic relatedness between the i-th candidate phrases and all candidates as the cross-phrase relevance. Thus, it calculates the local relevance as follows,
ppi = ∑
j=1,j 6=i (hpih
> pj − λδi). (4)
where δi = Mean( ∑ j=1,j 6=i hpih > pj ). Here, we treat δi as a de-noisy factor to filter the noises, which is far different from the i-th candidate keyphrase in the document.

heading: 2.5 Relevance Aggregation
text: We aggregate the phrase-document relevance and the cross-phrase relevance into a whole score as the importance score of each candidate via a simple multiplication,
ri = p̂ h i · ppi (5)
where ri indicates the importance score of the i-th candidate phrase. Then, we rank all candidates with their importance score ri and extract top-ranked k phrases as keyphrases of the source document.

heading: 3 Experiments and Results

heading: 3.1 Experimental Settings
text: This paper conducts experiments on three benchmark and popular used keyphrase datasets, which includes DUC2001 (Wan and Xiao, 2008), Inspec
(Hulth, 2003), and SemEval2010 (Kim et al., 2010). Due to page limits, please refer to the corresponding articles for the details of the three datasets.
Following the previous work (Liang et al., 2021; Ding and Luo, 2021; Song et al., 2023b), we use the standard practice and evaluate the performance of our model in terms of f-measure at the top-K keyphrases (F1@K) and adopt stemming to both extracted keyphrases and gold truth. Concretely, we report F1@5, F1@10, and F1@15 of each model on three benchmark datasets.
We adopt the pre-trained language model BERT (Devlin et al., 2019) as the backbone of our model, initialized from their pre-trained weights. In our experiments, λ is set to 0.9 for three benchmark datasets.

heading: 3.2 Overall Performance
text: Table 1 shows the performance of baselines and our model on three benchmark datasets (DUC2001, In-
spec, and SemEval2010). The results show that our method significantly improves over state-of-the-art unsupervised keyphrase extraction baselines. Compared with the current state-of-the-art models, our model achieves significantly better performance on F1@5, F1@10, and F1@15 evaluation metrics, demonstrating the effectiveness of estimating the importance of candidate phrases by leveraging the highlights to calculate the relevance.
Compared with EmbedRank (Bennani-Smires et al., 2018), KeyGames (Saxena et al., 2020), and SIFRank (Sun et al., 2020), HGUKE achieves significant improvement, which benefits from using the highlights to calculate the importance score of each candidate keyphrase. Compared with the best baseline JointGL, our model achieves better performance on several benchmark keyphrase extraction datasets in all evaluation metrics. The main reason for this improvement is that we use the highlights as the guidance information instead of the whole document when estimating the importance of keyphrases.

heading: 3.3 Ablation Test
text: The ablation experiments on three benchmark keyphrase extraction datasets are shown in Figure 3. It can be seen from the results that using the highlight information can significantly improve the performance of keyphrase extraction, which benefits from estimating the importance score of each candidate by using its corresponding highlight information rather than the whole document. We consider the main reason is that the title or the first sentence of the document usually has a strong guidance for extracting keyphrases.

heading: 3.4 Impact of Pooling Methods
text: In this section, we study different pooling methods, including mean- and max-pooling operations. For all pooling methods, HGUKE using the last BERT layer achieves the best results, demonstrating that HGUKE benefits from stronger contextualized semantic representations. We can see the results in Table 2 that the document encoded via the meanpooling operation obtains the best performance.

heading: 3.5 Impact of Different Similarity Measures
text: Our model adopts Manhattan Distance to measure the textual similarity between candidate phrases and the highlight information. Furthermore, we attempt to employ different measures to estimate the phrase-document relevance. The results of different similarity measures are shown in Table 3, and we can see that the advantage of Manhattan Distance is obvious.

heading: 4 Related Work
text: Most existing unsupervised keyphrase extraction methods can be mainly divided into four categories: statistics-based, topic-based, graph-based, and embedding-based models. Specifically, statisticsbased models (Salton and Buckley, 1988; Witten et al., 1999) usually extract keyphrases by estimating the importance of candidate phrases with different statistic features, such as word frequency feature, phrase position feature, linguistic features of natural language, etc. Topic-based models (Liu et al., 2009, 2010) typically utilize topic information to determine whether a candidate phrase is a keyphrase. Graph-based models (Mihalcea and Tarau, 2004; Grineva et al., 2009) represent the
document as a graph and rank candidate phrases by graph-based similarities.
Embedding-based models usually adopt the pretrained embeddings to obtain document and candidate phrase representations and calculate the importance score of each candidate depending on the obtained representations. Benefiting from the development of transformer-based pre-trained language models (Devlin et al., 2019) in the natural language processing field, embedding-based models (Bennani-Smires et al., 2018; Sun et al., 2020; Liang et al., 2021) have achieved outstanding performance. Concretely, embedding-based models mainly consist of two procedures: candidate keyphrase representation and keyphrase importance estimation (Hasan and Ng, 2014; Song et al., 2023a). The first procedure utilizes natural language linguistic features to construct candidate keyphrases and represents them by pre-trained embedding approaches (e.g., BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019)). The second procedure estimates the importance of candidate phrases from different perspectives to determine whether a candidate phrase is a keyphrase.
Unlike the existing unsupervised keyphrase extraction models, we use the highlight information of the document to calculate the phrase-document relevance instead the whole document.

heading: 5 Conclusion and Future Work
text: In this paper, we incorporate structural information to improve the performance of embedding-based unsupervised keyphrase extraction. Specifically, in this paper, we propose a new Highlight-Guided Unsupervised Keyphrase Extraction model (HGUKE), which calculates the phrase-document relevance via the highlight information instead of the whole document to select relevant candidate phrases. Extensive experiments demonstrate that HGUKE outperforms the state-of-the-art unsupervised baselines. Future research may investigate adopting different structural information of the source document to improve the performance of unsupervised keyphrase extraction.

heading: 6 Acknowledgments
text: We thank the three anonymous reviewers for carefully reading our paper and their insightful comments and suggestions. This work was partly supported by the Fundamental Research Funds for the Central Universities (2019JBZ110); the National
Natural Science Foundation of China under Grant 62176020; the National Key Research and Development Program (2020AAA0106800); the Beijing Natural Science Foundation under Grant L211016; CAAI-Huawei MindSpore Open Fund; and Chinese Academy of Sciences (OEIP-O-202004).

heading: 7 Limitations
text: There are still some limitations of our work. In the future, we plan to enhance the procedure of extracting candidate keyphrase, to improve the upper bound of the performance of keyphrase extraction. One possible way is to generate candidate phrases of the document by utilizing the high-level semantic relatedness (e.g., attention weights) instead of using the surface-or syntactic-level information.

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
7
3 A2. Did you discuss any potential risks of your work? 7
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 3
3 B1. Did you cite the creators of artifacts you used? 3
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? 3
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 3
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? 3
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? 3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 3
C 3 Did you run computational experiments? 3
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 3
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 3
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 3
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, we plan to enhance the procedure of extracting candidate keyphrase, to improve the upper bound of the performance of keyphrase extraction. One possible way is to generate candidate phrases of the document by utilizing the high-level semantic relatedness (e. g. , attention weights) instead of using the surface-or syntactic-level information