Unsupervised Open-domain Keyphrase Generation
Year: 2023

Abstract:
In this work, we study the problem of unsupervised open-domain keyphrase generation, where the objective is a keyphrase generation model that can be built without using human-labeled data and can perform consistently across domains. To solve this problem, we propose a seq2seq model that consists of two modules, namely phraseness and informativeness module, both of which can be built in an unsupervised and open-domain fashion. The phraseness module generates phrases, while the informativeness module guides the generation towards those that represent the core concepts of the text. We thoroughly evaluate our proposed method using eight benchmark datasets from different domains. Results on in-domain datasets show that our approach achieves stateof-the-art results compared with existing unsupervised models, and overall narrows the gap between supervised and unsupervised methods down to about 16%. Furthermore, we demonstrate that our model performs consistently across domains, as it overall surpasses the baselines on out-of-domain datasets.1.

heading: 1 Introduction
text: Keyphrases are short word sequences that describe the core concepts of the text. The prediction of keyphrases for a text is a task that has received much attention recently. It is a crucial problem as its outputs can be useful for a variety of downstream tasks such as building digital libraries (Gutwin et al., 1999; Witten et al., 2009), document summarization (Litvak and Last, 2008), document visualization (Chuang et al., 2012) and so on.
There are mainly two approaches to keyphrase prediction, namely keyphrase extraction (Mihalcea and Tarau, 2004; Florescu and Caragea, 2017a; Bennani-Smires et al., 2018) and keyphrase generation (Meng et al., 2017; Chen et al., 2019; Yuan
∗Work done while visiting Cazoodle Inc. 1Code and data will be available at https://github.
com/ForwardDataLab/UOKG.
et al., 2020; Shen et al., 2022). Keyphrase extraction highlights keyphrases that appear within the text. On the other hand, keyphrase generation generates keyphrases based on the understanding of the given text and therefore allows predicting absent keyphrases alongside present ones (Meng et al., 2017). This ability has made keyphrase generation receive more attention than keyphrase extraction in recent years, as human also tend to use keyphrases that are absent from the text.
Most of the existing keyphrase generation models use manually labeled data for training (Meng et al., 2017; Chen et al., 2018, 2019; Yuan et al., 2020; Ahmad et al., 2021). However, obtaining labeled data is often the most expensive component of any machine learning model, and this is the same for keyphrase generation. Compared to labeled data, access to unlabeled data is easier and mostly available. For example, the arXiv dataset (Clement et al., 2019) containing metadata (e.g., title, abstract) of 1.7 million research articles is readily available on Kaggle. Therefore, it is more desirable to construct a keyphrase generation model in an unsupervised fashion. Furthermore, in practice, the model may have to encounter texts that come from various domains or even unseen ones. Therefore, another attractive property of a keyphrase generation model is the ability to handle open-domain documents.
Considering the above scenario, we propose a new problem called Unsupervised Opendomain Keyphrase Generation. Similar to every keyphrase generation methods, the model of our objective is given a text x as input, and as output, it generates a set of keyphrases {y}. Both x and y are word sequences. Furthermore, the model of our objective should satisfy two requirements: 1) it can be built using only an unlabeled corpus, denoted as D; 2) it can effectively handle inputs from across domains.
This is a challenging task because we do not
10614
have access to labeled data from which to learn the patterns for keyphrases. Additionally, we also need our model to work across domains. This is difficult because there might exist different patterns for keyphrases for different domains. None of the existing work addresses these challenges. For instance, supervised keyphrase generation models (Meng et al., 2017; Chen et al., 2018, 2019; Yuan et al., 2020; Ahmad et al., 2021) not only require manually labeled data for training but are also known to perform poorly when being moved out-of-domain. On the other hand, (Shen et al., 2022) propose AutoKeyGen, which uses pseudo-labeled data to train a seq2seq model in a weakly-supervised fashion, thereby removing the need for human annotation effort. However, similar to supervised models, the weakly-supervised approach taken by AutoKeyGen does not enable it to maintain performance in unseen domains.
Therefore, to solve our problem, we propose an unsupervised keyphrase generation model that can work across domains. The key idea is to modularize a seq2seq model into two modules. The motivation for modularizing is to decompose keyphrase generation into two simpler problems where each of which can be addressed in an unsupervised and open-domain setting. The first module, named the phraseness module, is responsible for generating phrases, while the second module, named the informativeness module, guides the generation toward the phrases that represent the most crucial concepts of the text.
The phraseness module is a retrieval-augmented seq2seq model, where the retriever assists the seq2seq component in generating absent phrases alongside present ones. This module can be built in an unsupervised fashion because it leverages noun phrases to index the retriever and to train the seq2seq model, which can easily be obtained using open-sourced software libraries such as NLTK (Bird et al., 2009), and therefore does not require human annotation effort. Furthermore, the phraseness module can also be built in an open-domain fashion, thanks to 1) the part-of-speech information incorporated into the seq2seq model, which allows copying words to form grammatically correct noun phrases regardless of domains; 2) the fact that the retriever can be further indexed with domain-relevant information, to provide reliable references.
The informativeness module is another seq2seq model, where a phrase is likely to be generated if it contains words that are informative to the given text. Inspired by embedding-based unsupervised keyphrase extraction (UKE) methods, we quantify informativeness of a word and a text based on their closeness in meaning, which is measured via the similarity between their embeddings. We choose this method of evaluating informativeness over other UKE methods (e.g. graph-based, statistics based) since it supports not only present phrases, but also absent ones. Similar to the phraseness module, the informativeness module can also be built in an unsupervised and open-domain fashion.
This is obtained by using a domain-general, unsupervised text embedding model (e.g. Sent2Vec (Pagliardini et al., 2018)).
We summarize the contributions of our paper. Firstly, we propose a new problem called unsupervised open-domain keyphrase generation. Secondly, we design a model for solving the problem. Our proposed model is a seq2seq model that consists of two modules, one is responsible for generating phrases and the other guides the generation towards the phrases that represent the core concepts of the text. Finally, we conduct extensive experiments on multiple datasets across domains to demonstrate the effectiveness of our model as we contrast it against multiple strong baselines.

heading: 2 Proposed method
text: Figure 1 illustrates our proposed framework. We propose a seq2seq model that consists of two modules, namely phraseness and informativeness module. We adopt the two terms phraseness and informativeness from (Tomokiyo and Hurst, 2003), to describe the desirable criteria a keyphrase should satisfy. Phraseness refers to the degree to which a word sequence is considered a phrase, and informativeness refers to how well the phrase illustrates the core concepts of the text. Each of the two modules guarantees a criterion mentioned above. In particular, the phraseness module generates (present and absent) phrases, while the informativeness module guides the generation toward phrases that describe the core concepts of the text. In the following sections, we will describe in detail the two modules, as well as how they are combined to generate keyphrases.

heading: 2.1 Phraseness module
text: In order to generate keyphrases, it is crucial to know how to first generate phrases. We emphasize the difference between a keyphrase and a phrase
- the former needs to be informative to the given text, while the latter does not. It has been shown that keyphrases mostly take the form of noun phrases (Chuang et al., 2012). Also, recent work on keyphrase generation has shown that absent keyphrases can often be retrieved from other texts (Ye et al., 2021), suggesting that absent phrases can be found similarly. Therefore, a simple solution to obtaining phrases is to extract noun phrases as present phrases and retrieve related noun phrases as absent ones.
However, this simple solution may not be optimal. Since the retrieved phrases are originally used in other texts, they may not be suitable to describe the concepts of the given text. We demonstrate this limitation using the example in Figure 3a. In this example, the absent phrases obtained via retrieval describe concepts related to “topic modeling”. However, our desired outputs need to also describe concepts related to “author modeling”.
The above problem could be mitigated if we also consider the given text alongside the retrieved noun phrases. In the example above, relevant phrases such as “author topic distributions” can be generated by combining “author”, which is from the given text, and “topic distributions”, which is one of the retrieved phrases. With this in mind, we employ a retrieval-augmented seq2seq model as the phraseness module. First, a set of related but absent noun phrases is retrieved, which we will now refer to as references. Then, a seq2seq model generates noun phrases based on both the text and the references.

heading: 2.1.1 Retriever
text: Figure 2 describes the retrieval of references given a text. To obtain references for the input, we leverage existing noun phrases observed in other documents. We assume that a noun phrase is related to a text if it occurs in contexts similar to that text. With this in mind, we collect noun phrases from documents in the unlabeled corpus D to form a phrase bank B. We index each noun phrase z ∈ B with a context embedding, denoted as cz, which is obtained by averaging the embeddings of the documents in which z appears in. We obtain the embeddings of texts by using Sent2Vec (Pagliardini et al., 2018), an unsupervised sentence embedding model. To retrieve references for a text x, we first use Sent2Vec to compute its embedding, denoted as vx, and then retrieve the top-k phrases z based on the following retrieval score Rx(z) = cos(cz, vx).
Input: We introduce the author-topic model, a generative model for documents that extends Latent Dirichlet Allocation (LDA; Blei, Ng, & Jordan, 2003) to include authorship information. Each author is associated with a multinomial distribution over topics and each topic is associated with a multinomial distribution over words. A document with multiple authors is modeled as a distribution over topics that is a mixture of the distributions associated with the authors. We apply the model to a collection of 1,700 NIPS conference papers and 160,000 CiteSeer abstracts. Exact inference is intractable for these datasets and we use Gibbs sampling to estimate the topic and author distributions. We compare the performance with two other generative models for documents, which are special cases of the author-topic model: LDA (a topic model) and a simple author model in which each author is associated with a distribution over words rather than a distribution over topics. We show topics recovered by the author-topic model, and demonstrate applications to computing similarity between authors and entropy of author output.
Retrieved noun phrases: topic distributions, word distribution, novel topic model, topic models, topic distribution, pitman-yor process, latent topics, hdp, statistical topic models, multinomial distributions, lda model, latent dirichlet allocation model, dirichlet distribution, hierarchical dirichlet process, collapsed gibbs
(a) Example of using retrieval to predict absent phrases
Input: A framework for the analysis of error in global illumination algorithms. In this paper we identify sources of error in global illumination algorithms and derive bounds for each distinct category. Errors arise from three sources: inaccuracies in the boundary data, discretization, and computation. Boundary data consists of surface geometry, reflectance functions, and emission functions, all of which may be perturbed by errors in measurement or simulation, or by simplifications made for computational efficiency. Discretization error is introduced by replacing the continuous radiative transfer equation with a finite-dimensional linear system, usually by means of boundary elements and a corresponding projection method. Finally, computational errors perturb the finite-dimensional linear system through imprecise form factors, inner products, visibility, etc., as well as by halting iterative solvers after a finite number of steps. Using the error taxonomy introduced in the paper we examine existing global illumination algorithms and suggest new avenues of research.
EmbedRank: discretization error, discretization, global illumination algorithms, computational errors, finite-dimensional linear system, computation, iterative solvers, continuous radiative transfer equation, error taxonomy, error
TextRank: error, computational errors, form, projection, linear, functions, boundary, illumination, radiative transfer, computation
Groundtruth: discretization, boundary elements, global illumination, reflectance functions
(b) Example of EmbedRank and TextRank’s prediction, along with the groundtruth keyphrases
Figure 3: Examples
Furthermore, in order to prevent retrieving unreliable references, we filter those whose retrieval scores are smaller than a threshold τ . We denote the set of references for x as Zx.
As mentioned above, we can further index the retriever with other corpora, denoted as D′, from different domains. To do this, all we need to do is to update the phrase bank B with new phrases from D′ and update the context embeddings of every phrase that occur in both D and D′.

heading: 2.1.2 Seq2Seq model
text: Input representation. The seq2seq model takes as inputs not only the source text x but also its references Zx, to generate phrases. The text and its references are combined into a single input x̃, defined as
x̃ = [BOS] x [BOR] Zx [EOR] [EOS] (1) where [BOS] and [EOS] are respectively the beginning and end of sentence token. The two tokens [BOR] and [EOR] signals the start and end of the reference block. In addition, the references are separated by a [SEP] token.
Model architecture. We employ Transformer (Vaswani et al., 2017) with copy mechanism (Gu et al., 2016; See et al., 2017) as the architecture of our seq2seq model. First, the encoder receives the word embeddings of the tokens in x̃, producing a sequence of encoder hidden states h = {hi}|x̃|i=1. The decoder takes the embeddings of the previously generated words y<t and the encoder hidden states, outputting the decoder hidden state st. For each input, we build an extended vocabulary Vx̃, which is the union of the decoder’s vocabulary V and the words in the augmented input x̃. Finally, we compute the phraseness probability of predicting a word from Vx̃ as Ppn(yt|y<t, x̃) = pgenP Vpn(yt|y<t, x̃)+
(1− pgen)PCpn(yt|y<t, x̃). Here, P Vpn(yt|y<t, x̃) = softmax(W V st) is the distribution over the word vocabulary V , pgen = sigmoid(W g s st+W g y yt−1) is the soft switch between generating and copy. All the W terms are trainable parameters, and we omit the bias terms for less cluttered notation.
We incorporate part-of-speech information to copy words from x̃. More formally, PCpn(yt = w|y<t, x̃) = ∑ x̃i=w ati, where a t i = softmax(eti), and e t i = FFh(h̃i T )FFs(s̃t). Here, h̃i = concat(hi, lx̃i) is the encoder hidden state of x̃i enhanced with its part-of-speech embedding lx̃i . Similarly, s̃t = concat(st, lyt) is the decoder hidden state enhanced with the part-of-speech embedding of the previously generated word. FFh and FFs denotes the feedforward neural networks, whose purposes are to help project h̃i and s̃t into the same semantic space.
Model training. For every document xi ∈ D, we maximize logPpn(y = z|x̃), where
Ppn(y = z|x̃) = T∏
t=1
Ppn(yt = zt|y<t, x̃) (2)
for the phrases z = {zt}, which include the present noun phrases and the references. To encourage the model to generate absent phrases instead of just copying it from the references, we randomly mask some references and train the model to generate them.

heading: 2.2 Informativeness module
text: Knowing to generate phrases is not sufficient to obtain keyphrases. It is also important to guide the generation towards the phrases that are informative to the input. Previous work on unsupervised keyphrase extraction offer multiple classes of methods, namely graph-based, statistics-based and embedding-based, for evaluating informativeness
of phrases (for more details, see Section 5). Graphbased and statistics-based methods are not suitable in our setting. These methods utilize only in-text information and therefore cannot determine the informativeness of absent phrases. On the other hand, embedding-based methods evaluate informativeness of a phrase based on its closeness in meaning with the input text. As a result, these methods can support both present and absent phrases. We therefore adopt the idea of embedding-based methods in building our informativeness module.
Let us define S(a, b) = max(0, vTa vb) as the similarity score between two pieces of text, where va, vb are embeddings obtained using Sent2Vec. Using this score, we define the informativeness distribution Pin(y|x), by decomposing it into conditional distributions of each word given the previous context. More formally, Pin(y|x) =∏T
t=1 Pin(yt|y<t,x), where
Pin(yt = w|y<t,x) ∝ { S(w,x), if w ̸= [EOS] S(y<t,x), otherwise
(3) The probability Pin(yt = w|y<t,x) is normalized over the extended word vocabulary Vx̃, which is the same one used by the phraseness module. Intuitively, a word has high probability of being generated if that word has close meaning to the text. The [EOS] token is likely to be generated if the currently generated phrase y<t already form an informative phrase.

heading: 2.3 Combining phraseness and informativeness
text: Generating keyphrases require us to enforce both phraseness and informativeness on the output sequence. A simple solution is to adopt the approaches taken by existing unsupervised keyphrase extraction methods, which enforce the two criteria sequentially. In particular, they either 1) form phrases first, then choose those that are most informative as keyphrases; or 2) choose informative words first, then form keyphrases using these words. However, both approaches may not be optimal. The first approach may include uninformative words in the prediction, while the second rigidly assume that a keyphrase should only contain keywords. We illustrate the limitation of these approaches using an example, shown in Figure 3b. Here, we show the predictions of EmbedRank (Bennani-Smires et al., 2018), which takes approach 1) and TextRank (Mihalcea and Tarau, 2004), which takes approach 2).
Both of them fail to predict the golden keyphrase “global illumination”. EmbedRank redundantly include the word “algorithms”, while TextRank only outputs “illumination”, as “global” is not predicted as a keyword.
This problem could be alleviated if both phraseness and informativeness is considered when forming the keyphrase. In the example above, the word “algorithms” should be excluded, since it neither contributes to the informativeness of the phrase, nor it is required to make the phrase understandable. On the other hand, the word “global” may not be among the most informative words to the text, however, this word is essential as excluding it results in a phrase with a different concept.
In light of this, we propose to generate keyphrases, one word at a time, where each word is generated if it is predicted by both the phraseness and informativeness module. To this end, we propose to combine the two modules in a productof-experts fashion (Hinton, 2002). In particular, the conditional distribution of a keyphrase given a text is defined as follows
Pkp(y|x) ∝ Ppn(y|x̃)λ · Pin(y|x)
∝ T∏
t=1
Ppn(yt|y<t, x̃)λ · Pin(yt|y<t,x)
(4) where λ is a hyperparameter for balancing the two modules. The idea of combining two language models using the product-of-experts has previously been studied for the task of unsupervised abstractive summarization (Zhou and Rush, 2019). To the best of our knowledge, we are the first to use this idea in unsupervised keyphrase generation. In the above paragraphs, we also discussed why it is a suitable choice.

heading: 2.4 Keyphrase decoding
text: To decode keyphrases, we employ beam search to search for keyphrases based on s(y) = − logPkp(y|x). As beam search tend to favor shorter keyphrases, we employ the length normalization strategy similarly to that described in (Sun et al., 2019), which is to divide s(y) by |y| + α, where α is a length penalty factor.
It has been shown in previous work that positional information is useful for the prediction of present keyphrases (Florescu and Caragea, 2017b; Gallina et al., 2020). Therefore, it is desirable to incorporate this feature into our model. Furthermore,
we found that the model tends to generate absent keyphrases that are entirely new. This behavior may not be desirable for downstream tasks such as document retrieval, where we need to associate documents with common keyphrases. Based on the above discussion, we propose to rerank the beam search results using the following score
ŝ(y) = s(y)
|y|+ α × b(y) (5)
b(y) =    β, if y is absent and y ∈ B 1, if y is absent and y ̸∈ B log2(1+Px(y))
log2(1+Px(y))+1 , if y is present
(6) where b(y) is an adjustment weight, β is a hyperparameter for adjusting the scores of absent phrases that exist in the phrase bank B (β < 1 indicates that we favor y ∈ B), and Px(y) is the word offset position of the phrase y in the text x. Intuitively, b(y) favors present keyphrases that appear earlier in the text, and absent keyphrases that exist in the phrase bank B.

heading: 3 Experiments

heading: 3.1 Datasets
text: We use the documents from the training set of KP20K (Meng et al., 2017) to train our model and to index the retriever in the training phase. It contains the abstracts and titles of 514k scientific articles. In the testing phase, we utilize 8 datasets, namely SemEval (Kim et al., 2013), Inspec (Hulth, 2003), NUS (Nguyen and Kan, 2007), Krapivin (Krapivin et al., 2009), DUC-2001 (Wan and Xiao, 2008), OpenKP (Xiong et al., 2019), StackExchange (Yuan et al., 2020) and KPTimes (Gallina et al., 2019). The title and abstract of an article are concatenated to form a testing document.
The testing datasets are categorized into indomain and out-of-domain, by measuring the percentage of keyphrase overlap with the training corpus, i.e. the percentage of golden keyphrases in the testing dataset that also appear in some documents in KP20K. We choose the mean value of ∼ 33 as a threshold to classify the testing datasets. As a
result, the in-domain datasets include SemEval, Inspec, NUS, Krapivin and StackExchange, while the other three are out-of-domain.
In the testing phase, besides using KP20K, we also use the training set of StackExchange (300k documents) and KPTimes (260k documents) to further index the phrase bank and the retriever. The purpose of adding these additional sources in the testing phase is to test whether or not our model can easily integrate additional information to work in domains unseen during training, without having it re-trained.

heading: 3.2 Baselines & evaluation metrics
text: Baselines. We adopt five unsupervised keyphrase extraction (UKE) algorithms, namely TF-IDF, TextRank2 (Mihalcea and Tarau, 2004), MultiPartiteRank3 (Boudin, 2018), EmbedRank (BennaniSmires et al., 2018) and Global-Local Rank4 (Liang et al., 2021) as baselines.
We also compare our model with AutoKeyGen (Shen et al., 2022), which is the only previous work on unsupervised keyphrase generation. With the permission from the authors, we implemented and report the AutoKeyGen-Copy version. Furthermore, we present CopyRNN (Meng et al., 2017) as a supervised baseline. We employ the Transformerbased pointer-generator network for both AutoKeyGen and CopyRNN, with the same settings as described in A.1. Both AutoKeyGen and CopyRNN are trained using KP20K.
Evaluation metrics. We follow the widely-used strategy and separate the evaluation of present and absent keyphrase generation. We employ macroaverage F1 and macro-average Recall for evaluating present and absent keyphrase generation, respectively. We evaluate present keyphrases at top 3 and 5 predictions; and absent keyphrases at top 5 and 10. The predictions as well as the groundtruths are stemmed using Porter Stemmer5 (Porter, 1980) and duplicates are removed before evaluation.

heading: 3.3 Results

heading: 3.3.1 Keyphrase generation for in-domain cases
text: Table 2 illustrates the performance of our proposed model and the baselines for the five in-domain
2https://github.com/boudinfl/pke 3See footnote 1 4https://github.com/xnliang98/uke_ccrank 5https://github.com/nltk/nltk/blob/develop/
nltk/stem/porter.py
datasets. We also display the average performance across datasets.
Present keyphrase generation. For predicting present keyphrases, our model is best or secondbest on most datasets. On SemEval, our model is slightly inferior to TF-IDF and Global-Local Rank. The results on Inspec are worth noting, as our proposed model is significantly outperformed by UKE methods. This inferior performance may be due to this dataset not favoring generative methods, as even CopyRNN, the supervised baseline, failed to compete with UKE methods on Inspec. This behavior has also been observed in a recent work (Gallina et al., 2020). Although not being able to outperform existing methods on all datasets, our proposed model still achieves the best weightedaverage results, outperforming the second-best by about 14% for top 3 predictions and 10% for top 5.
Absent keyphrase generation. For predicting
absent keyphrases, our proposed model outperforms existing work on all datasets. UKE methods cannot be compared with our model, as they only extract present keyphrases. When comparing with AutoKeyGen, we observe that our proposed model have significantly better performance, except for the Inspec dataset where the results are on par. On average, we outperform AutoKeyGen by nearly twice for both top 5 and top 10 predictions.

heading: 3.3.2 Keyphrase generation for out-of-domain cases
text: One important objective of this work is the proposed model’s capability to perform in outof-domain settings. We show present and absent keyphrase generation performance for out-ofdomain datasets in table 3. For absent keyphrase generation, we only report results on KPTimes, as DUC-2001 and OpenKP mainly contain present keyphrases.
Present keyphrase generation. Our model achieves the best or second-best results on all outof-domain datasets. Similar to the in-domain cases, our model achieves the best weighted-average results despite not being able to outperform all baselines on all datasets. Of the two unsupervised keyphrase generation methods, our proposed model achieves significantly better results than AutoKeyGen in the out-of-domain setting.
Absent keyphrase generation. In the out-ofdomain setting. It can be seen that AutoKeyGen fails to generate absent keyphrases, with the re-
call of only 0.3% for top 10 predictions. On the other hand, our model can recall 3.6% of absent keyphrases. This improvement is significant considering that absent keyphrase generation has been pointed out to be a “very challenging task” (Meng et al., 2017).

heading: 3.3.3 Comparison to supervised baseline
text: Although not being able to compete with the supervised baseline on the in-domain datasets, our model has narrowed the gap between supervised and unsupervised keyphrase generation methods. In addition, our model shows remarkable performance on out-of-domain datasets, while the supervised baseline shows poor generalization. It can be seen from Table 2 and 3 that the performance of the supervised baseline plummets on out-of-domain datasets. On the other hand, our model is able to retain performance across domains.

heading: 3.4 Ablation study
text: We perform an ablation study to further understand the role of the components of our proposed model. In particular, we test our model with some components removed, namely the adjustment weight defined in Equation 6, the references and the partof-speech information. We report the results in Table 4. For KPTimes and OpenKP, we sample 200 and 500 documents from their original validation and test set to perform the ablation study.
We observe that no model in the ablation study
achieve the best performance in all cases. However, the full version of our model shows to be more well-rounded compared to its ablations.
Firstly, the adjustment weight b(y) proves to be crucial, as removing it cause our model’s performance to drop in most cases. This confirms that the positional information is useful in predicting present keyphrases, as has been pointed out by previous work (Florescu and Caragea, 2017b; Gallina et al., 2020). Moreover, prioritizing phrases that exist in the phrase bank also proves to be effective for predicting absent keyphrases. Next, removing the references shows to heavily affect absent keyphrase generation, especially on the out-of-domain dataset KPTimes. On the other hand, present keyphrase generation seems not to be affected without using references. Finally, the version of our model without part-of-speech information is able to maintain present keyphrase generation performance for the in-domain dataset (Krapivin), but slightly worsens when being moved out-of-domain. For absent keyphrase generation, it seems that part-of-speech information does not help for KPTimes. A possible explanation is that KPTimes mostly contains single-word keyphrases and therefore grammatical information can offer little help in this case.

heading: 4 Case study
text: We display two examples of generated keyphrases from AutoKeyGen and our proposed model in Fig-
ure 4. The first example is from Krapivin, an indomain dataset, while the second one is from KPTimes, an out-of-domain dataset. For the first example, we observe that both the proposed model and AutoKeyGen correctly predict the groundtruth (present and absent) keyphrases. However, it can be seen that, for generating absent keyphrases, AutoKeyGen only reorders words that are present in the given text. On the other hand, our model can generate keyphrases whose component words are absent, such as “relational models”, “categorical models” and “kripke models”.
In the second example, it is clear that our model predicts more correct keyphrases. We observe that the absent keyphrases generated by AutoKeyGen are highly irrelevant. On the other hand, our model successfully predicts “storms” and also outputs other absent keyphrases that are relevant, although not being within the ground truth keyphrases. This example help shows that our model is better at handling documents from different domains.

heading: 5 Related work

heading: 5.1 Unsupervised keyphrase extraction
text: Unsupervised keyphrase extraction (UKE) aims at identifying keyphrases within the text. Currently, there are three main classes of UKE methods, namely statistics-based, graph-based and embedding-based. Statistics-based methods (Campos et al., 2018) employ features such as TF-IDF, word position and casing aspect, to determine the relevance of a candidate phrase.
Graph-based methods typically build a graph from the source text, where a node could be a word or a phrase. Then, different graph-theoretic measures are used to estimate the importance of nodes, and finally phrases are formed based on the top ranked nodes. TextRank (Mihalcea and Tarau,
2004) builds a word graph where a link between two words exists if they co-occur within a window. SingleRank (Wan and Xiao, 2008), CiteTextRank (Gollapalli and Caragea, 2014) employs related documents to better measure similarity between word nodes. TopicRank (Bougouin et al., 2013), Topical PageRank (Liu et al., 2010) incorporate topical information in the graph ranking algorithm. Positional information is used in PositionRank (Florescu and Caragea, 2017a) to favor keyphrases that appear earlier in the text. (Boudin, 2018) utilizes the structure of multi-partite graphs to extract diverse keyphrases.
Embedding-based methods utilize embedding spaces to measure informativeness of candidates. EmbedRank (Bennani-Smires et al., 2018) rank candidates by measuring their distance to the source text in a pretrained sentence embedding space, then an optional diversification step is performed using maximal-marginal relevance to ensure diversity of extracted keyphrases. (Liang et al., 2021) jointly models local and global context of the document when ranking candidates.

heading: 5.2 Unsupervised keyphrase generation
text: Keyphrase generation aims at predicting both present and absent keyphrases for the source text. To our best knowledge, AutoKeyGen (Shen et al., 2022) is currently the only unsupervised keyphrase generation method. AutoKeyGen trains a seq2seq model on automatically generated silver labeled document-keyphrase pairs. The silver keyphrases are both present and absent, where present ones are extracted, and the absent ones are constructed from the words present in the text.

heading: 6 Conclusions
text: In this paper, we propose a new problem called unsupervised open-domain keyphrase generation. We propose a seq2seq model that consists of two modules, one is responsible for generating phrases while the other guides the generation towards phrases that reflect the core concepts of the given text. Our experiments on eight benchmark datasets from multiple domains demonstrate that our model outperforms existing unsupervised methods and narrows the gap between unsupervised and supervised keyphrase generation models. Furthermore, we demonstrate that the proposed model can perform consistently across domains.
Limitations
One limitation of the proposed method is that it does not consider domain-specific information to evaluate informativeness. The phraseness module has access to domain-specific knowledge, which are the phrases that occur in similar contexts, i.e. the references. On the other hand, the informativeness module only employs a domain-general sentence embedding model to measure informativeness of phrases. Therefore, the integration of both domain-specific and domain-general information for the evaluation of informativeness may be worth further investigation.
Another limitation of this work is that we only tested the proposed method on short texts. Therefore, it is uncertain of the proposed framework’s performance on long text documents. Handling long texts could be significantly more difficult than short text, as long texts contain much more information (can discuss a variety of topics).
The final limitation of this work is the absence of experiments on using different sentence embedding models to construct the informativeness module. Therefore, it might be useful to explore the impact of different sentence embedding models on keyphrase generation performance. We leave this for future work.

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Yes, it’s in the "Limitation" section after section 6.
7 A2. Did you discuss any potential risks of your work? In our paper, we aim to address the keyphrase generation task. We cannot think of any potential risks of our work.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Section 1 Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Left blank.
3 B1. Did you cite the creators of artifacts you used? Section 3.1
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Yes, we cite the source to the software and dataset that we used for reproducing previous work’s results in the footnote in Section 3.2 and the footnote in appendix A.1.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No, because the licenses for software and datasets are well known, standard licenses, which allow use of the artifacts in work like ours.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No, since the dataset we applied is a commonly used open-source benchmark datasets in the field of keyphrase generation.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Yes, in section 3.1, table 1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Yes, in section 3.1, table 1
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Left blank.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? In appendix A.1
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? In appendix A.3
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? In section 3.3, Table 2.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Yes, in Section 3.2 (mention using Porter Stemmer for preprocessing).
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We leave this for future work