SimCKP: Simple Contrastive Learning of Keyphrase Representations
Authors: Minseok Choi
Chaeheon Gwak
Seho Kim
Si hyeong Kim
Jaegul Choo
Venue: EMNLP 2023 Findings (2023)
Keywords: keyphrase prediction
contrastive learning
reranking
TL;DR: This paper proposes a two-stage contrastive framework for keyphrase prediction.

Abstract: Keyphrase generation (KG) aims to generate a set of summarizing words or phrases given a source document, while keyphrase extraction (KE) aims to identify them from the text. Because the search space is much smaller in KE, it is often combined with KG to predict keyphrases that may or may not exist in the corresponding document. However, current unified approaches adopt sequence labeling and maximization-based generation that primarily operate at a token level, falling short in observing and scoring keyphrases as a whole. In this work, we propose SimCKP, a simple contrastive learning framework that consists of two stages: 1) An extractor-generator that extracts keyphrases by learning context-aware phrase-level representations in a contrastive manner while also generating keyphrases that do not appear in the document; 2) A reranker that adapts scores for each generated phrase by likewise aligning their representations with the corresponding document. Experimental results on multiple benchmark datasets demonstrate the effectiveness of our proposed approach, which outperforms the state-of-the-art models by a significant margin.

SimCKP: Simple Contrastive Learning of Keyphrase Representations
Minseok Choi"
Chaeheon Gwak
Seho Kim $ Si Hyeong Kim² Jaegul Choo'
KAIST Al *Naver Webtoon AI
(minseok. choi, jchoo) @ekaist.ac.o {ch. gwak, seho. kim, typekim) @webttonnscorppcco | com
Abstract
Tite: Nortinear magnitostali: BEM formuation using one unknown double layer charge
Document: Purpose The purpose this paper is to solve generic mag hoggoiisiaticc problems BEM by stucying how use bouncary integral equation (BIE) with doxbi layer charge so urknown deried from the scalar potertial basggmmmeoooogyysppeeoo
Keyphrase generation (KG) aims generate a set of summarizing words or phrases given a source document, while keyphrase extrac- tion (KE) aims identify them from the text. Because the search space is much smaller in KF, it is often combined with KG to predict keyphrases that may may not exist in the corresponding document. However, current uni- fied approaches adopt sequence labeling and maximization-buset generation that primarily operate at a token level, falling short in ob- serving and scoring keyphrases whole. In this work, we propose SIMCKP, simple con- trastive learning framework that consists of two stages: 1) An sttractor-generatte that extracts keyphrases by learning context-aware phrase- level representations in a contrastive manner while also generating keyphrases that do not ap- pear in the document; 2) A reranker that adapts scores for each generated phrase by likewise aligning their representations with the corre- sponding document, Experimental results on multiple benchmark datasets demonstrate the effectiveness of our proposed approach, which outperforms the state-of-the-art models by significant margin."
Biros the double layer charge produces only the posontall gap withuut disturbing ng the normal magneto fux donsity, the filld accuranely formulated even by one BIE with one unknown. Once the double layer charge is desermied, Bava-Saarrs low gives easty the magnetic flu density. Findings The BE using double bayer charge is capable treating robusly gecmetnial sigulantee at edges and comers, magnetic parmubilly Orignnityooduu The proposed BIE comsins only the douthi layer charge while conventional equations
also expable salving the pmttiems with estremely high
scalar potential contain multily connected problerms, magnetomotive
single double byye charges untnowns, reprosvent crodwing
patential
muliply conmerted eacling cuments.
Present teyphrases: boundary integral equation, douple layer charge, muliipy
Absent keyphrases: Ennniinear magnetcstato anatyols, elestric current)
Sequsence labeling result: boundaayy integral equation double loyor charge, multityy connected. scoler potendal)
result: [ (magnatostadc energy analysis, magnatomotia force, electric cument)
Figure 1: An example of keyphrase prediction. Present and absent keyphrases are colored blue and red, respec- tively. Overlapping keyphrases are in bold.
et al., 1999; Hulth, 2003; Nguyen and Kan, 2007; Medelyan et al., 2009; Caragea et al., 2014; Zhang et al., 2016; Alzaidy et al., 2019) and keyphrase generation (KG) models (Meng et al., 2017: Ye and Wang. 2018; Chan et al., 2019; Chen et al., 2020b; Yuan et al., 2020; Ye et al., 2021b; Zhao et al., 2022), where the former only extracts present keyphrases from the text and the latter generates both present and absent keyphrases.
1 Introduction
Recently, several methods integrating KE and KG have been proposed (Chen et al., 2019a; Liu et al., 2021; Ahmad et al., 2021; Wu al., 2021, 2022b). These models predict present keyphrases using an extractor and absent keyphrases using a generator, thereby effectively exploiting a rel- atively small search space in extraction. However, current integrated models suffer from two limita- tions. First, they employ sequence labeling models that predict the probability of cach token being constituent of present keyphrase, where such token-level predictions may be a problem when the target keyphrase is fairly long or overlapping. As shown in Figure 1, the sequence labeling model makes an incomplete prediction for the term "multi-
Keyphrase prediction (KP) is task of identifying a set of relevant words or phrases that capture the main ideas or topics discussed in a given document. Prior studies have defined keyphrases that appear in the document as present keyphrases and the oppo- sites as absent keyphrases. High-quality keyphrases are beneficial for various applications such as in- formation retrieval (Kim al., 2013), text summa- rization (Pasunuru and Bansal, 2018), and trans- lation (Tang et al., 2016). KP methods are gener- ally divided into keyphrase extraction (KE) (Witten
Work done during internship Naver Webtoon. 'Our code is publicly available at https: //gi thub. com/ br ght jade/ SimCKP.

---

ply connected problem" because only the tokens for "multiply connected" have yielded a high probabil- ity. We also observe that the model is prone to miss the keyphrase "integral equations" every time be- cause it overlaps with another keyphrase "boundary integral equation" in the text. Secondly, integrated even purely generative models are usually based maximum likelihood estimation (MLE), which predicts the probability of each token given the past seen tokens. This approach scores the most proba- ble text sequence the highest, but as pointed out by Zhao et al. (2022), keyphrases from the maximum- probability sequence are not necessarily aligned with target keyphrases. In Figure 1, the MLE-based model predicts "magnetostatic energy analysis", which is semantically similar to but not aligned with the target keyphrase "nonlinear magnetostatic analysis". This may be consequence of greedy search, which can be remedied by finding the tar- get keyphrases across many beams during beam search, but would also create a large number of noisy keyphrases being generated in the top-k pre- dictions.
heuristic algorithm (see Section 4.1). The model pulls keyphrase embeddings to the document em- bedding and pushes away the rest of the candi- dates in a contrastive manner. Then during infer- ence, top-k keyphrases that are semantically close to the document are predicted. After the model has finished training, it generates candidates for absent keyphrases. These candidates are simply constructed by overgenerating with large beam size for beam search decoding. To reduce the noise introduced by beam search, we train a reranker that allocates new scores for the generated phrases via another round of contrastive learning, where this time the agreement between the document and ab- sent keyphrase representations is maximized. Over- all, major contributions of our work can be summa- rized follows:
We present contrastive learning framework that learns to extract and generate keyphrases by building context-aware phrase-level repre- sentations.
We develop a reranker based on the semantic alignment with the document to improve the absent keyphrase prediction performance.
Existing KE approaches based on represen- tation learning may address the above limita- tions (Bennani-Smires et al., 2018; Sun et al., 2020; Liang et al., 2021; Zhang et al., 2022; Sun al., 2021; Song et al., 2021, 2023). These methods first mine candidates that are likely to be keyphrases in the document and then rank them based on the relevance between the document and keyphrase embeddings, which have shown promising results. Nevertheless, these techniques only tackle present keyphrases from the text, which may mitigate the overlapping keyphrase problem from sequence la- beling, but they not suitable for handling MLE and the generated keyphrases.
To the best of our knowledge, we introduce contrastive learning to a unified keyphrase traction and generation task for the first time and empirically show its effectiveness across multiple KP benchmarks.
2 Related Work
2.1 Keyphrase Extraction
Keyphrase extraction focuses on predicting salient phrases that are present in the source document, Existing approaches can be broadly divided into two-step extraction methods and sequence label- ing models. Two-step methods first determine set of candidate phrases from the text using differ- ent heuristic rules (Hulth, 2003; Medelyan et al., 2008; Liu et al., 2011; Wang et al., 2016). These candidate phrases are then sorted and ranked by either supervised algorithms (Witten et al., 1999; Hulth, 2003; Nguyen and Kan, 2007; Medelyan et al., 2009) or unsupervised learning (Mihalcea and Tarau, 2004; Wan and Xiao, 2008; Bougouin
In this work, we propose two-stage contrastive learning framework that leverages context-aware phrase-level representations on both extraction and generation. First, we train an encoder-decoder network that extracts present keyphrases on top of the encoder and generates absent keyphrases through the decoder. The model learns to extract present keyphrases by maximizing the agreement between the document and present keyphrase repre- sentations. Specifically, we consider the document and its corresponding present keyphrases as pos- itive pairs and the rest of the candidate phrases as negative pairs. Note that these negative candi- date phrases are mined from the document using
et al., 2013; Bennani-Smires et al., 2018). Another line of work is sequence labeling, where a model learns to predict the likelihood of each word being a keyphrase word (Zhang et al., 2016; Luan et al., 2017; Gollapalli et al., 2017; Alzaidy et al., 2019).

---

Extraction and Generation
Reranking
maximize agreement
maximize agreement
Document Embedding
Keyphrase Embodding
keyphrases
Document Embedding
Keyphrase Embedding
hgg
ng
nge
ng
fg
doc
Extractor
Generator
Reranker
Reranker
Figure 2: A contrastive framework for keyphrase prediction. In the first stage (left), the model learns to maximize the relevance between present keyphrases and their corresponding document while generating absent keyphrases. After training, the model generates candidates for absent keyphrases and sends them the second stage (right), where the candidates are reranked after their relevance with the document has been maximized/minimmized.
2.2 Keyphrase Generation
ing process is to pull semantically similar data be close while pushing dissimilar data to far away in the representation space. Contrastive learn- ing has shown great success for various computer vision tasks, especially in self-supervised training (Chen et al., 2020a), whereas Gao et al. (2021) have devised contrastive framework learn univer- sal sentence embeddings for natural language pro- cessing. Furthermore, Liu and Liu (2021) formu- lated a seq2xeq framework employing contrastive learning for abstractive summarization. Similarly, a contrastive framework for autoregressive language modeling (Su et al.. 2022) and open-ended text gen- eration (Krishna et al., 2022) have been presented.
The task of keyphrase generation is introduced to predict both present and absent keyphrases. Meng al. (2017) first proposed CopyRNN, a seq2seq framework with attention and copy mechanism un- der the ONE2ONE paradigm, where a model is trained to generate a single keyphrase per docu- ment. However, due to the problem of having to fix the number of predictions, Yuan et al. (2020) proposed the ONE2SEQ paradigm where model learns to predict dynamic number of keyphrases by concatenating them into single sequence. Several ONE2S#Q-baan models have been pro- posed using semi-supervised learning (Ye and Wang, 2018), reinforcement learning (Chan et al., 2019), adversarial training (Swaminathan et al.. 2020), hierarchical decoding (Chen et al., 2020b), graphs (Ye et al., 2021a), dropout (Ray Chowdhury et al., 2022), and pretraining (Kulkarni al., 2022; Wu al., 2022a) improve keyphrase generation. Furthermore, there have been several attempts to unify KE and KG tasks into a single learning framework. These methods not only focus on gen- erating only the absent keyphrases but also perform presumably easier task by extracting present keyphrases from the document, instead of having to generate them from myriad of vocabularies. Cur- rent methodologies utilize external source (Chen et al., 2019a), selection guidance (Zhao et al., 2021), salient sentence detection (Ahmad et al., 2021), relation network (Wu et al., 2021), and prompt-based learning (Wu et al., 2022b).
There have been endeavors to incorporate con- trastive learning in the context of keyphrase extrac- tion. These methods generally utilized the pairwise ranking loss to rank phrases with respect to the document to extract present keyphrases (Sun et al., 2021; Song et al., 2021, 2023). In this paper, we de- vise a contrastive learning framework for keyphrase embeddings both extraction and generation to improve the keyphrase prediction performance.
Problem Definition
Given document x, the task of keyphrase prediction to identify set of keyphrases y = by),,i,...., where | is the number of keyphrases. In the ONE2ONE training paradigm, each sample pair (x, }) is split into multiple pairs {(x, y y'))i=..... to train the model to generate one keyphrase per document. In ONE2SEQ, each sample pair processed as (x, f())), where f())) is concatenated sequence of keyphrases. In this work, we train for extraction and generation si- multancously; therefore, we decompose y into a present keyphrase set y, = (y'.)-1..... and an absent keyphrase set )a (y wy..i...... 1.l.
2.3 Contrastive Learning
Methods extract rich feature representations based on contrastive learning (Chopra et al., 2005; Hadsell et al., 2006) have been widely studied in numerous literature, The primary goal of the learn-

---

4 SIMCKP
Algorithm 1 HHadd Negative Phrase Mining
Input: Source document maximum n-gram length n, regu- lar expression pattern POS tugging function tag( +), phrase parsing function perse(-)) stemming function stem() Output: Present keyphrase candidate Cpre
In this section, we elaborate on our approach to building contrastive framework for keyphrase pre- diction. In Section 4.1, we delincate our heuristic algorithm for constructing set of candidates for present keyphrase extraction; in Section 4.2, we de- scribe the multi-task learning process for extracting and generating keyphrases; and lastly, we explain our method for reranking the generated keyphrases
1: Cyre + Ø
2: word tag. Lag__piirs__ -- tag(x)
3: phrase_tree trase_tree_ee - parse(pp word_tag_pant
4:
for pluase E phraase_ree do
5:
for Wi, E phrase do
6:
if Tindep and & Tend dep then continue
7:
8:
span; stem(w:)
in Section 4.3. Figure 2 illustrates the overall archi- tecture of our framework.
9:
Cpra - C U span;
10:
for Wiyg pprasees + 1:] do
11:
if sennss),,,.pllit) break
then
4.1 Hard Negative Phrase Mining
12:
13:
if b4 Taep by E Teudddep then span, stem(w,y)
To obtain the candidates for present keyphrases.
14:
we employ a similar heuristic approach from ex-
15:
continue
if Tindep teg E Tatart then span, stemm wxy))
isting extractive methods (Hulth, 2003; Mihalcca
and Tarau, 2004; Wan and Xiao, 2008; Bennani- Smires et al., 2018). A notable difference between prior work and ours is that we keep not only noun phrases but also verb, adjective, and adverb phrases, as well phrases containing preposi- tions and conjunctions. We observe that keyphrases are actually made up of diverse parts of speech, and extracting only the noun phrases could lead to missing a significant number of keyphrases. Following the common practice, we assign part- of-speech (POS) tags to each word using the Stanford POSTagger" and chunk the phrase structure tree into valid phrases using the NLTK RegexpParser"
18:
Cyre + Cpre. span,
19:
return Cpre
tag sets Tstart_ and Tend dep- which include tags that cannot start but end a phrase and tags that can start but not end a phrase, respectively. Lastly, each candidate phrase is iterated over to acquire all n- grams that make up the phrase. For example, if the phrase is ""aplications of machine learning", we lect n- D-grams "upplications", "machine", "learning", "applications machine", "machine learning", and "applications of machine learning" as candidates. Note that phrases such as "applications of",, "of", "of machine". and "of machine learning" are not chosen as candidates because they are not proper phrases. As noted by Gillick et al. (2019), hard neg- atives are important for learning high-quality coder, and we claim that our mining accomplishes this objective.
As shown in Algorithm I, each document con- verted phrase structure tree where each word w is tagged with a POS tag The tagged docu- ment is then split into possible phrase chunks based on our predefined regular expression rules, which must include one or more valid tags such as nouns, verbs, adjectives, etc. Nevertheless, such valid tag sequences sometimes nongrammatical, which cannot be proper phrase and thus may introduce noise during training. response, we filter out such nongrammatical phrases by first categorizing tags as independent or dependent. Phrases generally do not start or end with preposition or conjunc- tion; therefore, preposition and conjunction tags belong to dependent tag set Taep. On the other hand, noun, verb, adjective, and adverb tags can stand alone by themselves, making them belong an independent tag set Tindep- There are also
4.2 Extractor-Generator
In order to jointly train for extraction and gener- ation, we adopt a pretrained encoder-decoder net- work. Given document x, is tokenized and fed as input to the encoder where we take the last hid- den states of the encoder to obtain the contextual embeddings of document:
(1) where T is the token sequence length of the doc- ument and he is the start token (e.g., 
) rep- resentation used as the corresponding document embedding. For each candidate phrase, we con- struct its embedding by taking the sum pool- ing of the token span representations: h, =
[bo, hi...., ....,,y)) = Encoder,
'nttps://star.forbat iot CoreNLP/
'https:// http:///wwwttttrrr..../..

---

SumPooling((hs.... be]), where s and de- note the start and end indices of the span. The
Split
Dataset
aKP,
#KP,
IKPl,
% Absent
# Samples
Trii
KP20k
5.28
3.76
1.94
38.19
530,809
document and candidate phrase embeddings are then passed through a linear layer followed by non- linear activation to obtain the hidden representa- tions:
Valid
KP20k
5.26
3.67
1.94
38.26
20,000
KP20k
5.27
3.74
2.04
37.03
20,000
Inspec
9.81
4.97
2.33
22.25
500
Krapivin
NUS
10.85
6.67
2.13
51.55
211
SemFval
14.97
3.50
2.15
55.74
100
dd = tanh (W a, ho + bdp))
(2)
Table I: Dataset statistics. #KP,; Jr average number of keyphrases; #KP,: standard deviation of the num- ber of keyphrases; IKPl, average length (n-gram) of keyphrases per document.
Zy tanh( ,, 1pp+ + bp):
where Wd, Wp. Dag- bp are learnable parameters. Contrastive Learning for Extraction To extract relevant keyphrases given a document, train our model to learn representations by pulling keyphrase embeddings to the corresponding document while pushing away the rest of the candidate phrase em- beddings in the latent space. Specifically, we follow the contrastive framework in Chen et al. (2020a) and take cross-entropy objective between the doc- ument and each candidate phrase embedding. We set keyphrases and their corresponding document positive pairs, while the rest of the phrases and the document are set as negative pairs. The training objective for a positive pair (Zd,, ap.c) (i.e., docu- ment and present keyphrase v/y) with Np candidate pairs is defined as
4.3 Reranker
As stated by Zhao al. (2022), MLE-driven mod- els predict candidates with the highest probability, disregarding the possibility that target keyphrases may appear in suboptimal candidates. This prob- lem can be resolved by setting a large beam size for beam search; however, this approach would also result in substantial increase in the generation of noisy keyphrases among the top-k predictions. Inspired by Liu and Liu (2021), we aim to reduce this noise by assigning new scores to the generated keyphrases.
Candidate Generation We employ the fine- tuned model from Section 4.2 to generate candi- date phrases that highly likely to be absent keyphrases for the corresponding document. We perform beam search decoding using a large beam size on each training document, resulting in the overgeneration of absent keyphrase candidates. The model generates in an ONE2SEQ fashion where the outputs are sequences of phrases, which means that many duplicate phrases are present across the beams. We remove the duplicates and arrange the phrases such that each unique phrase is indepen- dently fed to the encoder. We realize that the gener- ator sometimes fails to produce even single target keyphrase, in which we filter out such documents for the second-stage training.
simtadp pppll
Ce 1--log llog sim My "
ST Σ " "
(3)
where T is a temperature hyperparameter and sim(u u, v) is the cosine similarity between vectors u and v. The final loss is then computed across all positive pairs for the corresponding document (i.e.. Σ 1''l FCL)
fCL
Our model generates keyphrases by learning probability distribution po(y.) over an absent keyphrase text sequence Ya "o..yy.... (i.e.. in an ONE2SEQ fashion), where & denotes the model parameters. Then, the MLE objective used to train the model to generate absent keyphrases is defined as
Joint Learning
Dual Encoder We adopt two pretrained encoder- only networks and obtain the contextual embed- dings of a document, as well as each candidate phrase c: [h
1 yyl
MMLE
log DogroVaaelya.e))
(4)
0 g h2 ...., h Encoder,, (x) and
Lastly, we combine the contrastive loss with the negative log-like ligeliiel loss to train the model to both extract and generate keyphrases:
[he h. ..., h. = Encoder, (c), where T, is the token sequence length of the candidate phrase and ha and he are the start token representations used as the document and candidate phrase embedding, respectively. Consequently, their hidden representa- tions are obtained by zd, = tanh (anhWWa, (W,, h ba.)
L LME + XLCL-
(5)
where À is a hyperparameter balancing the losses in the objective.

---

Inspec
Krapivin
SemEiaal
KP20k
Model
FiBM
F/85
Fi8M
1985
FP8M
F,85
F.OM
F,85
FPBM
Generative Modela
catSeq (Yaa et al., 2020)
0.225
0.262
0.269
0.354
0.323
0.397
0.242
0.253
0.291
0.367
eateegGG (Chen al., 201
0.229
0.270
0.282
0.366
0.325
0.393
0.246
0.290
0.292
0.366
CaSSegGG22RF) (Chan et al.. 2019)
0.253
0.301
0.300
0.309
0.375
0.433
0.287
0.329
0.321
0.386
ExHiiD-- (Chen al., 2020b)
0.253
0.291
0.286
0.347
0.284
0.335
0.311
0.374
SelThans (Ye et 2021h)
0.285
0.324
0.326
0.364
0.406
0.450
0.331
0.357
0.358
0.392
CorrKG (Zhao et al., 2022)
0.330
0.365
0.405
0.449
0.333
0.350
0.370
0.404
Unifedd Models
SEG-Net (Ahmad et al., 2021)
0.216
0.265
0.276
0.366
0.396
0.461
0.283
0.332
0.311
0.379
UniKeyphrase (Wu et al., 2021)
0.260
0.288
0.415
0.443
0.302
0.322
0.347
0.332
PromptKP (Wa et al., 2022b)
0,260
0.294
0.412
0.439
0.329
0.356
0.351
0.355
SIMCKP
0.356,
0.358
0.405g
0.405g
0.496
0.498
0.387,
0.386
0.426;
0.427
Table 2: Present keyphrase prediction results. The best results are in bold, while the second best are underlined. The subscript denotes the corresponding standard deviation (e.g., 0.427, indicates 0.427 0.001).
and 2a = tanh(W,ni, + ba), where Wd,, Wa. ba, b are learnable parameters.
Generative Models both present and absent keyphrases through gen- eration. Most models follow catSeq (Yuan et al., 2020), seq2seq framework under the ONE2SEQ paradigm. We report the performance of catSeq along with its variants such catSeqTG (Chen et al., 2019b), catseqTG-2RF (Chan al., 2019), and ExHiRD-h (Chen et al., 2020b). We also com- pare with two state-of-the-arde SetTrans (Ye et al., 2021b) and CorrKG (Zhao et al., 2022).
Generative models predict
Contrastive Learning for Generation To rank relevant keyphrases high given a document, we train the dual-encoder framework via contrastive learning. Following a similar process as before, we train our model to learn absent keyphrase rep- resentations by semantically aligning them with the corresponding document. Specifically, we set the correctly generated keyphrases and their corre- sponding document as positive pairs, whereas the rest of the generated candidates and the document become negative pairs. The training objective for a positive pair (Zd,, z0,i) (i.c., document and absent keyphrase v with Na candidate pairs then fol- lows Equation 3, where the cross-entropy objective maximizes the similarity of positive pairs and min- imizes the rest. The final loss is computed across all positive pairs for the corresponding document with summation.
Unified Models Unified models combine extrac- tive and generative methods to predict keyphrases. We compare with the latest models including SEG- Net (Ahmad et al., 2021), UniKeyphrase (Wu et al., 2021), and PromptKP (Wu et al., 2022b).
5.3 Evaluation Metrics
Following Chan et al. (2019), all models are evalu- ated on macro-averaged Fi @5 and F @M. . F @M compares all the predicted keyphrases with the ground truth, taking the number of predictions into account. Fj @5 measures only the top five predictions, but if the model predicts less than five keyphrases, we randomly append incorrect keyphrases until it obtains five. The motivation is to avoid Fj @5 and F, @M reaching similar results when the number of predictions is less than five. We stem all phrases using the Porter Stemmer and remove all duplicates after stemming.
5 Experimental Setup
5.1 Datasets
We evaluate framework on five scientific article datasets: Inspee (Hulth, 2003), Krapivin (Krapivin et al., 2009), NUS (Nguyen and Kan, 2007), Se- val (Kim et al., 2010), and KP20k (Meng et al., 2017). Following previous work (Meng et al., 2017; Chan et al., 2019; Yuan et al., 2020), we concate- nate the title and abstract of each sample as source document and use the training set of KP20k to train all the models. Data statistics are shown in Table 1.
5,4 Implementation Details
Our framework is built on PyTorch and Hugging- face's Transformers library (Wolf al., 2020). We use BART (Lewis et al., 2020) for the encoder- decoder model and uncased BERT (Devlin et al., 2019) for the reranking model. We optimize their weights with A (Loshchilov and Hutter,
5.2 Baselines
We compare our framework with two kinds of KP models: Generative and Unified.

---

Inspec
Krapivin
NUS
SemEval
KP20K
Model
F
FAaM
F,45
FLOM
F1
FLOM
F195
FLOM
F1655
F F.uM
Generative Models
catSeq (Yaun et al., 2020)
0.004
0.008
0.018
0.036
0.016
0.028
0.016
0.028
0.015
0.032
cateeTGG (Chen al., 20196)
0.005
0.011
0.018
0.034
0.011
0.018
0.011
0.018
0.015
0.032
caSSeTGG-2Rfi (Chan at., 2019)
0.012
0.021
0.030
0.053
0.019
0.031
0.021
0.030
0.027
0.050
ExHiRD-h (Chen et al., 2020b)
0.011
0.022
0.022
0.043
0.017
0.025
0.032
SetTrans (Yo al., 2021b)
0.021
0.084
0.047
0.073
0,042
0.060
0.026
0.034
0.086
0.058
CorKKG (Zhao 2022)
0.032
0.045
0.061
0.079
0.039
0.044
0.053
0.071
Unifted Models
SEG-Net (Ahmad 2021)
0.018
0.036
0.021
0.036
0.021
0.030
0.018
0.036
KmiKeyphrase (Wu et al., 2021)
0.026
0.036
0.045
0.056
0.045
0.052
0.046
0.068
PromptKP (Wu al., 20226)
0.017
0,022
0,036
0.042
0.028
0.032
0.082
0.042
SIMCKP
0.033,
0.035g
0.078
0.089
0.076
0.088
0.040g
0.047,
0.073;
0.080;
Table Absent keyphrase prediction results.
2019) and tune our hyperparameters to maximize Fi T on the validation set, incorporating tech- niques such early stopping and linear warmup followed by linear decay to O. We set the maximum n-gram length of candidate phrases 6 during min- ing and fix À to 0.3 for scaling the contrastive loss. When generating candidates for absent keyphrases, we use beam search with the beam size 50. Dur- ing inference, we take the candidate phrases as predictions in which the cosine similarity with the corresponding document is higher than the thresh- old found in the validation set. The threshold is calculated by taking the average of the Fi @.M- maximizing thresholds for each document. If the number of predictions is less than five, we retrieve the top similar phrases until we obtain five. We con- duct our experiments with three different random seeds and report the averaged results.
In-domain
Out-of-domain
Method
Fi 85
Fi &&M
Fi &&&M
Present keyphrase prediction
SIMCKP
0.426
0.427
0.411
0.412
w/o CL
0.295
0.388
0.278
0.356
SEOLABL
0.236
0.384
0.180
0.272
CL= BI=BANRYCLLF
0.209
0.322
0.171
0.251
Absent keyphrase prediction
SIMCKP
0.073
0.080
0.057
0.064
w/o CL w/o RERANKER
0,030
0.066
0.027
0.051
0.035
0.069
0.025
0.041
Table 4: Ablation study. "w/o CL" is the vanilla BART model using heam search for predictions. "w/o reranker" extracts with CL but generates using only beam search.
timal thresholds seems necessary for improving the F1 @M performance. Nonetheless, real-world applications are often focused on identifying the top-k keywords, which we believe our model effec- tively accomplishes.
6 Results and Analyses
6.2 Ablation Study
6.1 Present and Absent Keyphrase Prediction
We investigate each component of our model to understand their effects the overall performance and report the effectiveness of each building block in Table 4. Following Xie et al. (2022), we report on two kinds of test sets: 1) KP20k, which we refer to as in-domain, and 2) the combination of Inspec, Krapivin, NUS, and SemEval, which is out- of-domain.
The present and absent keyphrase prediction results are demonstrated in Table and Table 3, respec- tively. The performance of our model mostly ex- ceeds that of previous state-of-the-a methods by a large margin, showing that our method is effective in predicting both present and absent keyphrases. Particularly, there is notable improvement in the @5 performance, indicating the effectiveness of upproach in retrieving the top-k predictions. On the other hand, we observe that @ val- ues are not much different from F1 a5 and we believe this is due to the critical limitation of a global threshold. The number of keyphrases varies significantly for each document, and finding op-
Effect of CL We notice significant drop in both present and absent keyphrase prediction perfor- mance after decoupling contrastive learning (CL). For fair comparison, we set the beam size to 50, but our model still outperforms the purely genera- tive model, demonstrating the effectiveness of CL. We also compare our model with two extractive methods: sequence labeling and binary classifica-
8,85 "We compared with A 85 and found no difference in determining the best hyperparameter configuration.

---

8.0
F183
Document
FIGH
Poslire
3.45
Negativ
0.35
8
0.25
nogec
Figure 4:
Visualization the semantic space using t- SNE. The left shows the extractor space, while the right depicts the generator space after reranking.
Krepiin
0.15
3.10
srack
Nax N-thram Length
present keyphrase prediction performance from unigrams 6-grams, as shown in Figure 3. For all datasets, the performance steadily increases until the length of 3, which then plateaus to the rest of the lengths. This indicates that the test- ing datasets are mostly composed of unigrams, bi- grams, and trigrams. The performance increases slightly with the length of 6 for some datasets, such as Inspec and SemEval, suggesting that there negligglee number of 6-gram keyphrases. Therefore, the length seems feasible for maxi- mum performance in all experiments.
Figure 3: Comparison of present keyphrase prediction performance w.r.t max n-gram length during extraction.
In-domain
Out-of-domand
Mining Method
F1G5
F,0.M
F,05
F.OM
IN-BATCH Doc NEGATIVES
0.077
0.038
0.092
0.065
RANDOM NEGATIVES
0.210
0.226
0.211
0.267
HARD NEGATIVES (OURS)
0.428
0.427
0.411
0.412
Table 5:
Comparison of negative mining methods for present keyphrase prediction.
tion. For sequence labeling, we follow previous work (Tokala et al., 2020; Liu et al., 2021) and em- ploy BiLSTM-CRF, strong sequence labeling baseline, top of the encoder to predict BIO tag for each token, while for binary classification, a model takes each phrase embedding to predict whether cach phrase is keyphrase or not. CL out- performs both approaches, showing that learning phrase representations is more efficacious.
6.4 Impact of Hard Negative Phrase Mining
In order to assess the effectiveness of our hard neg- ative phrase mining method, we compare it with other negative mining methods and report the re- sults in Table 5. First, utilizing in-batch document embeddings as negatives yields the poorest perfor- mance. This likely due to ineffective differen- tiation between keyphrases and other phrase em- beddings. Additionally, we experiment with using random text spans as negatives and observe that although it aids in representation learning to some degree, the performance improvement is limited. The outcomes of these two baselines demonstrate that our approach successfully mines hard nega- tives, enabling our encoder acquire high-quality representations of keyphrases.
Effect of Reranking We remove the reranker and observe the degradation of performance in absent keyphrase prediction. Note that the vanilla BART (i.e., w/o CL) is trained to generate both present and absent keyphrases, while the other model (i.e., w/o RERANKER) is trained to generate only the absent keyphrases. The former performs slightly better in out-of-domain scenarios, as it is trained to generate diverse keyphrases, while the latter ex- cels in in-domain since absent keyphrases resemble those encountered during training. Nevertheless, the reranker outperforms the two, indicating that plays a vital role in the KG part of our method.
6.5 Visualization of Semantic Space
To verify that our model works as intended, we vi- sualize the representation space of our model with t-SNE (van der Maaten and Hinton, 2008) plots, depicted in Figure 4. From the visualizations, we find that our model successfully pulls keyphrase embeddings close to their corresponding document in both extractor and generator space. Note that the generator space displays lesser number of phrases than the beam size 50 because the duplicates after stemming have been removed.
6.3 Performance over Max N-Gram Length We conduct experiments on various maximum lengths of n-grams for extraction and compare the
"We use the BIO format for our sequence labeling baseline. For example, if the phrase "voip conferencing system" tokenized into "v ##oi #Hi con #afer #tencingg system", it is laheled as "B 111111..

---

Inspec
Krapivin
NUS
SemEval
KP20k
government (MSIT) (No. 2020-0-00368, Neural- Symbolic Model for Knowledge Acquisition and Inference Techniques and No. 2021-0-02068, Ar- tificial Intelligence Innovation Hub), and the Na- tional Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. NRF-2022R I A2B5B02001913), We thank all re- searchers at NAVER WEBTOON Ltd. for their valuable discussions.
Ro050
0.144
0.226
0.18
0.072x
0.220;
Table Upper bound performance for absent keyphrase prediction after overgeneration with the beam size 50.
6.6 Upper Bound Performance
Following previous work (Meng et al., 2021; Ray Chowdhury et al., 2022), we measure the up- per bound performance after overgeneration by cal- culating the recall score of the generated phrases and report the results in Table 6. The high recall demonstrates the potential for reranking to increase precision, and we observe that there is room for improvement by better reranking, opening up an opportunity for future research.
References
Wasi Ahmad, Xiao Bai, Soomin Lee, and Kai-Wei Chang. 2021. Select, extract and generate: Neural keyphrase generation with layer-wise coverage at- tention. In Proceedings of the 59th Anmual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 1389-1404, Online. Association for Computa- tional Linguisties.
7
Conclusion
This paper presents contrastive framework that aims to improve the keyphrase prediction perfor- mance by learning phrase-level representations, rec- tifying the shortcomings of existing unified models that score and predict keyphrases at a token level. To effectively identify keyphrases, we divide our framework into two stages: joint model for ex- tracting and generating keyphrases and reranking model that scores the generated outputs based on the semantic relation with the corresponding docu- ment. We empirically show that our method signif- icantly improves the performance of both present and absent keyphrase prediction against existing state-of-the-art models.
Rabah Alzaidy, Cornelia Caragea, and C. Lee Giles. 2019. Bi-Isum -rf sequence labeling for keyphrase extraction from scholarly documents. In The World Wide Web Conference, WWW '19, page 2551-2557, New York, NY, USA, Association for Computing Machinery.
Kamil Cennani-Smiree Claudiu Musat, Andreea Hoss- mann, Michael Baeriswyl, and Martin Jaggi. 2018. Simple unsupervised keyphrase extraction using sen- tence embeddings. In Proceedings of the 22nd Con- ference on Computational Natural Language Learn- ing, pages 221-229, Brussels, Belgium. Association for Computational Linguistics.
Adrien Bougouin, Florian Boudin, and Béatrice Daille. 2013. TopicRank: Graph-based topic ramking for keyphrase extraction. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 543-551, Nagoya, Japan. Asian Federation of Natural Language Processing.
Limitations
Despite the promising prediction performance of the framework proposed in this paper, there is still room for improvement. A fixed global threshold has limited the potential performance of the frame- work, especially when evaluating F1 @.M. We ex- pect that adaptively selecting threshold value via an auxiliary module for each data sample might overcome such challenge. Moreover, the result of the second stage highly depends on the perfor- mance of the first stage model, directing the next step of research towards an end-to-end framework.
Comella Caragea, Florin Adrian Bulgarov, Andreea Godea, and Sujatha Das Gollapalli, 2014. Citation- enhanced keyphrase extraction from research pa- pers: A supervised approach. In Proceedings of the 2014 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 1135-1446 Doha, Qatar. Association for Computational Linguis- tics.
Hou Pong Chan, Wang Chen, Lu Wang, and Irwin King. 2019. Neural keyphrase generation via reinforcement learning with adaptive rewards. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 2163-2174, Florence. Italy, Association for Computational Linguistics.
Acknowledgements
This work was supported by Institute for Infor- mation& & communications Technology Planning & Evaluation (ITP)) grant funded by the Korea
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020a. A simple framework for

---

contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, volume I 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR.
R. Hadsell, S. Chopra, and Y. LeCun. 2006. Dimension- ality reduction by learning an invariant mapping. In 2006 IEFF Computer Society Conference on Com- puter Vision and Pattern Recognition (CVPR '06), volume 2, pages 1735-1742.
Wang Chen, Hou Pong Chan, Piji Li, Lidong Bing, and Irwin King. 2019a. An integrated appreach for keyphrase generation via exploring the power re- trieval and extraction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies. Volume I (Long and Short Papers), pages 2846 2856, Minneapolis, Minnesota. Association for Computational Linguistics.
Anette Hulth. 2003. Improved automatic keyword ex- traction given more linguistic knowledge. In Pro- ceedings of the 2003 Conference on Empirical Meth- ods in Natural Language Processing, pages 216 223.
Su Nam Kim, Olena Medelyan, Min- Kan, and Tim- othy Baldwin, 2010. SemEval-2010 task 5 Auto- matic keyphrase extraction from scientific articles. In Proceedings of the 5th International Workshop Semantic Evaluation, pages 21 21-26, Uppsala, Sweden. Association for Computational Linguistics.
Wang Chen, Hou Pong Chan, Piji Li, and Irwin King. 2020b. Exclusive hierarchical decoding for deep keyphrase generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1095-1105, Online. Association for Computational Linguistics.
Youngsam Kim, Munhyong Kim, Andrew Cattle, Julia Ormakhova, Suzi Park, and Hyopil Shin. 2013. Ap- plying graph-based keyword extraction to document retrieval. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 864-868, Nagoya, Japan. Asian Federation of Natural Language Processing.
Wang Chen, Yifan Gao, Jiani Zhang, Irwin King, and Michael R Lyu. 2019b. Title- selee-guied encoding for keyphrase generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8268-6275.
Mikalai Krapivin, Aliaksandr Autaeu, and Maurizio Marchese. 2009. Large dataset for keyphrase extrac- tion.
S. Chopra, R. Hadsell, and Y. LeCun. 2005. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference Computer Vision and Pattern Recog- nition (CVPR 005). volume I, pages 539-546 vol. 1.
Kalpesh Krishna, Yapei Chang, John Wieting, and Mo- hit lyyer. 2022. RankGen: Improving text gener- ation with large ranking models. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Procexsing, pages 199-232, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and
Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Mayank Kulkarni, Debanjan Mahata. Ravneet Arora. and Rajarshi Bhowmik. 2022. Learning rich repre- sentation of keyphrases from text. In Findings of the Association for Computational Linguisties: NAACL 2022, pages 891-906, Seattle, United States. Associ- ation for Computational Linguistics.
Tianyu Gao. Xingeheng Yao, and Danqi Chen, 2021. SimCSE: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 6894-6910, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad. Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequencc-to-sqqueeee for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguisties, pages 7871-7880, Online, Association for Computa- tional Linguistics.
pre-training
Daniel Gillick, Sayali Kulkarni, Larry Lansing, Alessan- dro Presta, Jason Baldridge, Eugene Ie, and Diego Garcia-Olano. 2019. Learning dense representations for entity retrieval. In Proceedings of the 23rd Con- ference on Computational Natural Language Learn- ing (CoNLLL), pages 528-537, Hong Kong, China. Association for Computational Linguistics.
Xinnian Liang, Shuangzhi Wu, Mu Li, and Zhoujun Li. 2021. Unsupervised keyphrase extraction by jointly modeling local and global context. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 155-164, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Sujatha Das Gollapalli, Xiao-Li Li, and Peng Yang. 2017. Incorporating expert knowledge into keyphrase extraction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31.
Rui Liu, Zheng Lin, and Weiping Wang- 2021. Addressing extraction and generation separately:

---

Keyphrase prediction with pre-trained language mod- els. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3180-3199
Thuy Dung Nguyen and Min-Yen Yen Kan. 2007. Keyphrase extraction in scientific publications. In Asian Digital Libraries. Looking Back 10 Years and Forging New Frontiers, pages 317-326, Berlin, Hei- delberg. Springer Berlin Heidelberg.
Yixin Liu and Pengfei Liu. 2021. SimCLS: A sim- ple framework for contrastive learning of abstractive summarization, In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 2: Short Pupers), pages 1065-1072, Online. Association for Computational Linguistics.
Ramakanth Pasunuru and Mohit Bansal. 2018. Multi- reward reinforced summarization with saliency and entailment. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Short Papers), pages 646-653, New Orleans, Louisiana. Association for Computa- tional Linguistics.
Zhiyuan Liu, Xinxiong Chen, Yabin Zheng, and Maosong Sun. 2011. Automatic keyphrase extraction by bridging vocabulary gap. In Proceedings of the Fifteenth Conference on Computational Natural Lan- guage Learning, pages 135-144, Portland, Oregon, USA. Association for Computational Linguisties.
Jishnu Ray Chowdhury, Seo Yeon Park, Tuhin Kundu, and Cornelia Caragea. 2022. KPDROP: Improving absent keyphrase generation. In Findings of the Asso- ciation for Computational Linguisties: EMNLP 2022. pages 4853-4870, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Nyy Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Mingyang Song, Liping Jing, and Lin Xiao. 2021. Im- portance Estimation from Multiple Perspectives for Keyphrase Extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 2726-2736, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
I Luan, Mari Ostendorf, and Hannameh Hajishirzi. 2017. Scientific information extraction with semi- supervised neural tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natu- ral Language Processing, pages 2641-2651 Copen- hagen, Denmark. Association for Computational Lin- guistics.
Mingyang Song, Lin Xiao, and Liping Jing. 2023. Learning to extract from multiple perspectives for neural keyphrase extraction. Computer Speech & Language, 11:101502.
Olena Medelyan, Eibe Frank, and Ian H. Witten. 2009. Human- competccmeetiiive tagging using automatic keyphrase extraction. In Proceedings of the 2009 Conference on Empirical Methods Natural Lan- guage Processing, pages 1318-1322 1318, Singapore. As- sociation for Computational Linguistics.
Yixuan Su, Tian Lan, Yan Wang. Dani Yogatama, Ling- peng Kong, and Nigel Collier. 2022. A contrastive framework for neural text generation. In Advances in Neural Information Processing Systems.
Si Sun. Zhenghao Liu, Chenyan Xiong. Zhiyuam Liu, and Jie Bao. 2021. Capturing global informativeness in open domain keyphrase extraction. In Natural Language Processing and Chinese Computing: 10th CCF International Conference, NLPCC 2021, Qing- dao, China, October 13-17, 2021, Proceedings, Part Springer.
Olena Medelyan, Ian H Witten, and David Milne. 2008. Topic indexing with wikipedia. Proceedings of the AAA/ WikiA/ workshop, volume 1, pages 19-24.
Rui Meng, Xingdi Yuan, Tong Wang, Sanqiang Zhao, Adam Trischler, and Daqing He. 2021. An empir- ical study on neural keyphrase generation. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4985-5007.
I1 10, pages
Yi Sun, Hangping Qiu. Yu Zheng, Zhongwei Wang, and Chaoran Zhang. 2020. Sifrank: a new baseline for un- supervised keyphrase extraction based on pre-trained language model. IEEE Access, 8: 10896-10906.
Rui Meng. Sanqiang Zhao, Shuguang Han, Daqing He, Peter Brusilovsky, and Yu Chi. 2017. Deep keyphrase generation. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 582-592, Vancouver, Canada. Association for Computational Linguisties.
Avinash Swaminathan, Haimin Zhang, Dehanjan Ma- hata, Rakesh Gosangi. Rajiv Ratn Shah, and Amanda Stent. 2020. A preliminary exploration of GANs for keyphrase generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 8021-8030, On- line. Association for Computational Linguistics.
Rada Mihalcea and Paul Tarau. 2004. TextRank: Bring- ing order into text. In Proceedings of the 2004 Con- ference Empirical Methods in Natural Language Processing, pages 404 411, Barcelona, Spain. Asso- ciation for Computational Linguistics.
Yaohua Tang, Fandong Meng, Zhengdong Lu, Hang Li, and Philip L. H. Yu. 2016. Neural machine translation with external phrase memory. CoRR,

---

Santosh Tokala, Debarshi Kumar Sanyal. Plaban Kumar Bhowmick, and Partha Pratim Das. 2020. SaSAKE: Syntax and semantics aware keyphrase extraction from research papers. In Proceedings of the 28th International Conference on Computational Linguis- fics, pages 5372-5383, Barcelona, Spain (Online). International Committee Computational Linguis- tics.
Binbin Xie, Xiangpeng Wei, Baosong Yang, Huan Lin, Jun Xie, Xiaoli Wang, Min Zhang, and Jinsong Su. 2022. WR-One2Set: Towards well-calibrated keyphrase generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing. pages 7283-7293, Abu Dhabi, United Arab Emirates. Association for Computa- tional Linguistics.
Laurens van der Maaten and Geoffrey Hinton. 2008 Visualizing data using t-SNE. Journal of Machine Learning Research, 9:2579-2605.
Hai Ye and Lu Wang. 2018. Seme-supe learn- ing for neural keyphrase generation. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4142-4153, Brussels, Belgium. Association for Computational Linguistics.
Xiaojun Wan and Jianguo Xiao. 2008 Single document keyphrase extraction using neighborhood knowledge. In AAAI, volume 8, pages 855-860.
Jiacheng Ye, Ruijian Cai, Tao Gui, and Qi Zhang. 2021a. Heterogeneous graph neural networks for keyphrase generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 2705-27 15, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Minmei Wang, Bo 7hao, and Yihua Huang. 2016. Ptr: Phrase-based topical ranking for automatic keyphrase extraction in scientific publications. In Neural Infor- mation Processing: 23rd International Conference, ICONIP 2016, Kyoto, Japan, October 16-21, 2016, Proceedings, Part IV 23, pages 120-128. Springer.
Jiacheng Ye, Tao Gui, Yichao Luo, Yige Xu, and Qi Zhang. 2021b. One2Set: Generating diverse keyphrases as a set. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 4598-4608, Online. Association for Computational Linguistics.
Tan H Witten, Gordon W Paynter, Eibe Frank, Carl Gutwin, and Craig G Nevill-Manning. 1999. Kea: practical automatic keyphrase extraction. In Pro- ceedings of the fourth ACM conference on Digital libraries, pages 254-255.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu. Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of- the-art art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, Association for Computational Linguistics.
Xingdi Yuan. Tong Wang. Rui Meng, Khushboo Thaker, Peter Brusilovsky, Daqing He, and Adam Trischler. 2020. One size does not fit all: Generating and evalu- ating variable number of keyphrases. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7961-7975, On- line. Association for Computational Linguisties.
Linhan Zhang, Qian Chen, Wen Wang, Chong Deng, ShiLiang Zhang, Bing Li, Wei Wang, and Xin Cao. 2022. MDERank: A masked document embedding rank approach for unsupervised keyphrase extraction. In Findings of the Association for Computational Lin- guistics: ACL 2022, pages 396-409, Dublin, Ireland. Association for Computational Linguisties.
Di Wu, Wasi Ahmad, Sunipa Dev, and Kai-Wei Chang. 2022a. Representation learning for resource- constrained keyphrase generation. Findings of the Association for Computational Linguistics: EMNLP 2022, pages 700-716, Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
Qi Zhang, Yang Wang, Yeyun Gong, and Xuanjing Huang. 2016. Keyphrase extraction using deep recur- rent neural networks on Twitter. In Proceedings of the 2016 Conference on Empirical Methods in Nat- ural Language Processing, pages 836-845, Austin, Texas, Association for Computational Linguisties.
Huanqin Wu, Wei Liu, Lei Li, Dan Nie, Tao Chen, Feng Zhang, and Di Wang. 2021. UniKeyphrase: A unified extraction and generation framework for keyphrase prediction. Findings of the Association for Compu- lational Linguistics: ACL-IJCNLP 2021, pages 825- 835. Online. Association for Computational Linguis- tics.
Guangzhen Zhao, Guoshun Yin, Peng Yang, and Yu Yao. 2022. Keyphrase generation via soft and hard seman- tic corrections. In Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 7757-7768. Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Huanqin Wu. Baijiaxin Ma. Wei Liu, Tao Chen, and Dan Nie. 2022b. Fast and constrained absent keyphrase generation by prompt-based lcarning, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11503
Jing Zhao, Junwei Bao, Yifan Wang, Youzheng Wu, Xi- aodong He, and Bowen Zhou. 2021. SGG: Learning

---

select, guide, and generate for keyphrase gener- ation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 5717-5726, Online. Association for Computational Linguistics.
A Additional Details for SIMCKP
A.1 Details for Hard Negative Phrase Mining
In order to effectively extract present keyphrases, we categorize POS tags to the corresponding tag set as the following:
Tindep: ("CD", "FW", "Gw", "NN*", "Va*", "JJ*", "RBx", "ADD")
Tdep: ("CC", "POS", "HYPH" "IN")
Tstort_depp ("RP")
"Teud_ ("DT", "AFX", "LS")
where each tag is defined in the NLTK library. asterisk (*) refers to any character(s) that come after the tag name. For example, "JJ", "JJR", and "JJS" are adjectives, comparative adjectives, and superlative adjectives, respectively, and all of them are elements of the independent tag set Tindep.
A.2 Hyperparameter Search
We perform grid search to find the best hyperpa- rameter configuration and report the tuning range used for our experiments in Table 7. The evaluation on the validation set is performed for every 5,000 gradient accumulating steps, and the tolerance in- creases by when the validation loss or Fj @M is worse than the previous evaluation.
Hyperparameter
Runge 5o-5
learning warm-up ratio batch size
5c-5, Ic-4
0.10...1
0.0
14.8, 16
8
BASE
00.L
[0.1. LO
0.3
epech
{ 1
10
max tolerance
[10]
10
max grad
L.)
learning (e 2e-5, 3e 5. 5e
3e
warm up ratio
0..0...1 0.1
0.1
batch size
14,8, 16
BERT BASH:
[0.1)
0.1
epoch max tolerance grad
[ [0]
10
[
Table 7: Hyperparameter tuning range and best values used in the experiments.


Reviews:
[{"review_id": "V3rMcwyqJq", "reviewer": "Reviewer_zjaV", "summary": "", "questions": "Question A: do you plan to make the code, models, data from the negative phrase mining, available to the general public?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer zjaV for their valuable time and constructive feedback on improving our paper. The reviewer raised several concerns, which we respond to below:\n- *Computational Cost*: As the reviewer pointed out, the training cost of SimCKP is approximately twice as much as other standalone models, as it does require additional training of the reranker (e.g., BERT). However, compared to the model sizes and GPU power available today, we believe that the cost is not burdensome. Furthermore, the inference cost includes retrieving top *k* phrases in addition to beam search, which is not extremely expensive. Therefore, deploying it to real-world applications by any chance will not be a big problem. In regards to an apples-to-apples comparison with other models, the salient sentence selector in SEG-Net [1] and the stacked relation layer in UniKeyphrase [2] are also additional modules that require extra training. In that sense, we believe that our model is not much different from theirs. Still, to make the comparison as fair as possible, we have incorporated base models for our framework.\n- *Reproducibility*: We recognize that data from the negative phrase mining, as well as the code and model checkpoints, are crucial for reproducing the paper. We plan to release the source code, data, model checkpoints, and prediction results.\n\n[1] Select, Extract and Generate: Neural Keyphrase Generation with Layer-wise Coverage Attention (Ahmad et al., 2021)\n\n[2] UniKeyphrase: A Unified Extraction and Generation Framework for Keyphrase Prediction (Wu et al., 2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "> New framework with multi-task learning objective and reranking to boost evaluation metrics.\n\n>Achieves new state of the art performance over popular keyphrase extraction/generation datasets.\n\n> Ablation studies have been performed to strengthen authors' argument towards novel contribution.", "reasons_to_reject": "> New framework seems computationally very expensive. It aggregates/boosts results from separate individual models. Comparing it to standard keyphrase generation papers in table 2 and 3 is not exactly apples to apples comparison, given they are standalone RNN or transformer based models. \n\n> The negative phrase mining, new learning objective and reranker are a combination of specific heuristics proposed by the authors. If the code, data and models are not made available to the public, it may cause serious reproducibility issues.", "questions_to_authors": "Question A: do you plan to make the code, models, data from the negative phrase mining, available to the general public?", "paper_topic_and_main_contributions": "The paper introduces an interesting approach towards the tasks of keyphrase extraction-generation. The framework designed by the author comprises of a multi-task model and a reranker model. The evaluation results presented are compared with standard keyphrase evaluation state of the art papers across popular datasets. Appropriate ablation study has been performed after the results.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MoFjct6GO9", "reviewer": "Reviewer_TNRt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are encouraged that Reviewer TNRt found our paper making “total sense” and “well written.” We deeply appreciate the reviewer’s positive comments and helpful feedback, which we respond to below:\n- *Motivation*: We appreciate the reviewer for pointing this out. As stated in the paper, our contrastive learning framework brings the document and keyphrase embeddings together in the semantic space while pushing away the rest of the phrase embeddings from the document. Hence, we can say that our model has learned to distinguish representations at a *phrase level*. At inference time, computing the distance (cosine similarity in our case) between document and phrase embeddings will be equivalent to the scoring of the phrases. Such scores will then be ranked, and the top *k* are successfully retrieved. A related work [1] explained this contrastive learning approach as “introducing competition between the candidate keyphrases to extract more salient keyphrases.” Furthermore, learned phrase-level representations via contrastive learning are especially effective in scenarios where phrases overlap. For example, say that only one of the overlapping phrases “boundary integral equations” and “integral equations” is the ground-truth keyphrase. For a standard language model, the two phrases are semantically similar; therefore, the model is very likely to predict both as keyphrases for a given document. However, for our model, contrastive learning has clearly distinguished their representations w.r.t. the document, thereby successfully predicting the one true keyphrase. This motivation may not have been clearly conveyed in the paper; thus, we will update this explanation in our revised manuscript.\n- *Reproducibility*: We recognize that our code and predicted results would be useful for ongoing research efforts. We plan to release the source code, data, model checkpoints, and prediction results.\n- *UniKeyphrase Results*: The authors of UniKeyphrase [2] and PromptKP [3] are the same, and thus we followed the convention in the PromptKP paper, which compared with the greedy search results. However, we agree that the best results should be used for comparison. We will revise our manuscript accordingly.\n\n[1] Importance Estimation from Multiple Perspectives for Keyphrase Extraction (Song et al., 2021)\n\n[2] UniKeyphrase: A Unified Extraction and Generation Framework for Keyphrase Prediction (Wu et al., 2021)\n\n[3] Fast and Constrained Absent Keyphrase Generation by Prompt-Based Learning (Wu et al., 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The author applies a simple contrastive learning framework to generate keyphrases for a given document, which makes total sense. \n2. The experimental setup is described and the experiments show the proposed method seems to achieve good results on five datasets. \n3. The paper is well written. ", "reasons_to_reject": "1. The motivation of the work is very weak. Why the contrastive learning framework would benefit learning phrase-level representations (or benefit observing and scoring keyphrases as a whole)? \n\n2. The source code for training the proposed model and the predicted results are not released. It would be useful for ongoing research efforts if the paper could release its source code and predicted results.\n\n3. In Table 3, the best results of UniKeyphrase (beam=4) should be used for comparing different models.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes to generate the keyphrase for a given document using a simple contrastive learning framework. ", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "avQpBNAf1W", "reviewer": "Reviewer_8xSg", "summary": "", "questions": " (1) The innovation of the paper using the joint learning framework is limited. Existing methods proposed jointly learns to extract and generate keyphrases, such as UniKeyphrase etc. \n\n (2) Why is contrastive learning framework beneficial for learning phrase-level representations? Please give details. \n\n (3)  In Table 3, why is the best result of UniKeyphrase (beam=4) not used as the comparison of experimental results? Please give details.\n\n (4) The paper sets the maximization agreement in both stages. Is it the same to set the maximization agreement measure?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We genuinely appreciate Reviewer 8xSg for their thorough and thoughtful review. The reviewer asked several insightful questions, which we respond to below:\n- *Innovation of the Joint Learning Framework*: We certainly agree that a joint learning framework is not new. However, building a joint learning framework is not the main contribution of our work. We have leveraged joint learning because as we stated in the abstract and introduction, the search space in KE is much smaller than in KG, and therefore it is much more effective to combine both approaches for keyphrase prediction. We want to emphasize that our paper is mainly a work that addresses two limitations of the current joint models – sequence labeling and MLE – and presents contrastive learning as a solution.\n- *Benefits of Contrastive Learning*: We appreciate the reviewer for pointing this out. As stated in the paper, our contrastive learning framework brings the document and keyphrase embeddings together in the semantic space while pushing away the rest of the phrase embeddings from the document. Hence, we can say that our model has learned to distinguish representations at a *phrase level*. At inference time, computing the distance (cosine similarity in our case) between document and phrase embeddings will be equivalent to the scoring of the phrases. Such scores will then be ranked, and the top *k* are successfully retrieved. A related work [1] explained this contrastive learning approach as “introducing competition between the candidate keyphrases to extract more salient keyphrases.” Furthermore, learned phrase-level representations via contrastive learning are especially effective in scenarios where phrases overlap. For example, say that only one of the overlapping phrases “boundary integral equations” and “integral equations” is the ground-truth keyphrase. For a standard language model, the two phrases are semantically similar; therefore, the model is very likely to predict both as keyphrases for a given document. However, for our model, contrastive learning has clearly distinguished their representations w.r.t. the document, thereby successfully predicting the one true keyphrase. This motivation may not have been clearly conveyed in the paper; thus, we will update this explanation in our revised manuscript.\n- *UniKeyphrase Results*: The authors of UniKeyphrase [2] and PromptKP [3] are the same, and thus we followed the convention in the PromptKP paper, which compared with the greedy search results. However, we agree that the best results should be used for comparison. We will revise our manuscript accordingly.\n- *Maximization Agreement*: We apologize if we misunderstood the question. The maximization agreement in both stages uses the same contrastive loss in Equation 3. The only difference is that present keyphrase representations are utilized in Stage 1, whereas the absent keyphrases are used in Stage 2.\n\n[1] Importance Estimation from Multiple Perspectives for Keyphrase Extraction (Song et al., 2021)\n\n[2] UniKeyphrase: A Unified Extraction and Generation Framework for Keyphrase Prediction (Wu et al., 2021)\n\n[3] Fast and Constrained Absent Keyphrase Generation by Prompt-Based Learning (Wu et al., 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": " The proposed method seems to achieve good results on multiple benchmark datasets for keyphrase prediction.", "reasons_to_reject": " (1) The innovation of the paper using the joint learning framework is limited. Existing methods proposed jointly learns to extract and generate keyphrases, such as UniKeyphrase etc.\n\n (2) Why is contrastive learning framework beneficial for learning phrase-level representations? Please give details. \n\n (3)  In Table 3, why is the best result of UniKeyphrase (beam=4) not used as the comparison of experimental results? Please give details.\n\n (4) The paper sets the maximization agreement in both stages. Is it the same to set the maximization agreement measure?\n", "questions_to_authors": " (1) The innovation of the paper using the joint learning framework is limited. Existing methods proposed jointly learns to extract and generate keyphrases, such as UniKeyphrase etc. \n\n (2) Why is contrastive learning framework beneficial for learning phrase-level representations? Please give details. \n\n (3)  In Table 3, why is the best result of UniKeyphrase (beam=4) not used as the comparison of experimental results? Please give details.\n\n (4) The paper sets the maximization agreement in both stages. Is it the same to set the maximization agreement measure?", "paper_topic_and_main_contributions": "The topic of this paper is about approaches for data- and compute efficiency.\nThe main contributions of this paper are as follows: \n(1)\tThis paper presents a contrastive learning framework to extract and generate keyphrases by building context-aware phrase-level representations.\n(2)\tThe paper develop a reranker based on the semantic alignment with the document to improve the absent keyphrase prediction performance. \n(3)\tExperimental results on multiple benchmark datasets demonstrate that the proposed approach is effective.\n", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UaiK3d73dV", "reviewer": "Reviewer_h8yT", "summary": "", "questions": "1. Will the code be released? \n\n2. \"F1@5 measures only the top five predictions, but if the model predicts less than five keyphrases, we randomly append incorrect 428\nkeyphrases until it obtains five\" -- this motivation does not make sense to me. Making changes to an evaluation purely for the sake of making it different from another evaluation sounds ill-formed. I can just add some random noise to make it different - but presumably that would be not meaningful. Adding dummy keyphrases would lead to penalizing perfect predictions when ground truth has less than 5 keyphrases. It seems like an ill-motivated modification by all account. (I won't deduct any point from your paper - in particular - because of this - my criticism is aimed more generally against the whole literature following Chan et al. -- still I would be open to hearing if there is a counter to my point here)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that Reviewer h8yT found our approach “reasonably novel” and “technically sound.” We are deeply thankful for the reviewer’s precious time and critical feedback, which we respond to below:\n- *Sequence Labeling*: We agree that the sequence labeling baseline was naively implemented and that stronger baseline results should be reported to prove the superiority of contrastive learning over sequence labeling. Therefore, we have conducted an additional experiment using BiLSTM-CRF, a widely used strong sequence labeling method. As shown below, the baseline results have been significantly boosted from the naive 3-way classifier; however, they are still outperformed by SimCKP, demonstrating the superiority of our model. We will update this result in our revised manuscript.\n> | Method | In-domain F1@5 | In-domain F1@M | Out-of-domain F1@5 | Out-of-domain F1@M | \n| --- | --- | --- | --- | --- |\n| CL=>3-way classifier | 0.156 | 0.257 | 0.117 | 0.181 |\n| CL=>BiLSTM-CRF | *0.207* | *0.345* | *0.174* | *0.272* |\n| SimCKP | **0.426** | **0.427** | **0.411** | **0.412** |\n\n- *Missing Related Work*: We apologize for missing out on some related work in keyphrase extraction using contrastive learning. We will add the mentioned work, in addition to other studies we might have missed. In terms of novelty, applying contrastive learning for extraction itself is not new; however, all of the aforementioned and previous works have not explored a joint contrastive learning setup for extraction and generation. As we have stated in the introduction, presenting contrastive learning to a unified keyphrase prediction task is one of the main contributions of our work.\n- *Reproducibility*: We recognize that our code would be useful for ongoing research efforts. We plan to release the source code, data, model checkpoints, and prediction results.\n- *Thoughts on F1@5*: We appreciate the reviewer for bringing this to the discussion. We also believe that appending incorrect keyphrases is counterintuitive, as it may represent the model performance as lower than it actually is when the ground truth has less than 5 keyphrases. However, because the literature following Chan et al. measures their models using this procedure, our work follows the same for a fair comparison. With all that said, we do feel that a better evaluation metric is worth exploring for future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Reasonably novel and technically sound combination of methods for unified generation/extraction. \n2. Reranking after over-generation is a pretty reasonable strategy for improving absent keyphrase generation.", "reasons_to_reject": "1. Not sure if the sequence labeling baselines are implemented in the best way. The paper says the reason for the poor performance of sequence labeling is \"due to the employment of a subword tokenizer, as it requires having to predict more tokens.\". However, standard implementations [1] of sequence labeling use some pooling method (for example mean pool, or selecting the first subword token within a word) to transform subwords to words before BIO predictions. It is also standard to use a CRF for sequence labeling. Since one of the main claims in the paper is the superiority of contrastive learning over sequence labeling, the lack of attention to effort in creating stronger sequence labeling baselines is a critical miss. There are also some very strong sequence labeling results in the literature [2, 3]. \n2. Misses an entire branch of related works that do a sort of contrastive learning [4-6] for keyphrase extraction. This also means that the contrastive learning setup for extraction is not as novel as it may seem.\n\n\n[1] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding - Devlin et al. ACL 2019 (even in one of the earliest pre-trained models using subword tokenizer see \"We use the representation of the first sub-token as the input to the token-level\nclassifier over the NER label set\". I have seen other papers trying different poolings.\n\n[2] Addressing Extraction and Generation Separately: Keyphrase Prediction With Pre-Trained Language Models - Liu et al. IEEE 2021\n\n[3] SaSAKE: Syntax and Semantics Aware Keyphrase Extraction from Research Papers - Santosh et al. COLING 2020\n\n\n**Post Rebuttal:** My concerns are mostly addressed.", "questions_to_authors": "1. Will the code be released? \n\n2. \"F1@5 measures only the top five predictions, but if the model predicts less than five keyphrases, we randomly append incorrect 428\nkeyphrases until it obtains five\" -- this motivation does not make sense to me. Making changes to an evaluation purely for the sake of making it different from another evaluation sounds ill-formed. I can just add some random noise to make it different - but presumably that would be not meaningful. Adding dummy keyphrases would lead to penalizing perfect predictions when ground truth has less than 5 keyphrases. It seems like an ill-motivated modification by all account. (I won't deduct any point from your paper - in particular - because of this - my criticism is aimed more generally against the whole literature following Chan et al. -- still I would be open to hearing if there is a counter to my point here)", "paper_topic_and_main_contributions": "The authors combine keyphrase extraction (of present keyphrases) and generation (of absent keyphrases) in a unified setting. Unlike prior methods, the authors motivate the use of contrastive learning (rather than sequence labeling) for extraction (using the contextualized encoder representations from BART) combined with the MLE generation loss (for absent keyphrase generation) in a multi-task manner. Furthermore, the authors use a BERT-based re-ranker (trained using contrastive learning again)  to rerank overgenerated absent keyphrases and select the top keyphrases that pass a threshold (tuned as a hyperparameter). The combined method outperforms existing generation models by a high margin.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[4] Capturing Global Informativeness in Open Domain Keyphrase Extraction - Sun et al. NLPCC 2021\n\n[5] Importance Estimation from Multiple Perspectives for Keyphrase Extraction - Song et al. EMNLP 2021\n\n[6] Learning to extract from multiple perspectives for neural keyphrase extraction - Song et al. Computer Speech & Language 2023\n\nOther missing references:\n\n[2] Addressing Extraction and Generation Separately: Keyphrase Prediction With Pre-Trained Language Models - Liu et al. IEEE 2021 (also combines extraction and generation in a certain manner)\n\n[7] KPDrop: Improving Absent Keyphrase Generation - Ray Chowdhury et al. EMNLP Findings 2022\n\n[8] Learning Rich Representation of Keyphrases from Text - Kulkarni et al. NAACL Findings 2022\n\n[9] Heterogeneous Graph Neural Networks for Keyphrase Generation - Ye et al. EMNLP 2021\n\n[10] Representation Learning for Resource-Constrained Keyphrase Generation - Du et al. EMNLP Findings 2022", "Typos_Grammar_Style_and_Presentation_Improvements": "A better motivation for overgeneration + reranking should be the recorded high recall performance of beam search [7,8]. Even now the recall is much higher (so there is still room for improvement purely by better reranking). \n\n**Post Rebuttal:** I increased my soundness score to 4 (from 2)  based on the discussion below."}}]

Author rebuttal:

Meta-review: This paper presents a joint learning framework based on contrastive learning to extract and generate keyphrases. Concerns lie in the novelty over UniKeyPhrase and the motivation of the contrastive learning framework (partially addressed in the rebuttal). Comparison under the beam search setting is also needed.
Decision: Accept-Findings