Rethinking Model Selection and Decoding for Keyphrase Generation with Pre-trained Sequence-to-Sequence Models
Authors: Di Wu
Wasi Uddin Ahmad
Kai-Wei Chang
Venue: EMNLP 2023 Main (2023)
Keywords: Keyphrase Generation
Pre-trained Language Models
Text Generation Decoding
Performance Evaluation

Abstract: Keyphrase Generation (KPG) is a longstanding task in NLP with widespread applications. The advent of sequence-to-sequence (seq2seq) pre-trained language models (PLMs) has ushered in a transformative era for KPG, yielding promising performance improvements. However, many design decisions remain unexplored and are often made arbitrarily. This paper undertakes a systematic analysis of the influence of model selection and decoding strategies on PLM-based KPG. We begin by elucidating why seq2seq PLMs are apt for KPG, anchored by an attention-driven hypothesis. We then establish that conventional wisdom for selecting seq2seq PLMs lacks depth: (1) merely increasing model size or performing task-specific adaptation is not parameter-efficient; (2) although combining in-domain pre-training with task adaptation benefits KPG, it does partially hinder generalization. Regarding decoding, we demonstrate that while greedy search achieves strong F1 scores, it lags in recall compared with sampling-based methods. Based on these insights, we propose DeSel, a likelihood-based decode-select algorithm for seq2seq PLMs. DeSel improves greedy search by an average of 4.7% semantic F1 across five datasets. Our collective findings pave the way for deeper future investigations into PLM-based KPG.

Rethinking Model Selection and Decoding for Keyphrase Generation with Pre-trained Sequence-to-Sequence Models
Di Wu, Wasi Uddin Ahmad, Kai-Wei Chang
University of California, Los Angeles
(diwu, kwchang [cs. www.sucll... edu, wasiahmad@ucla.ed
Abstract
keyphrase generation (KPG) involves predicting both types of keyphrases.
Keyphrase Generation (KPG) is a longstand- ing task in NLP with widespread applications. The advent of sequence-to-see to-sequence (seq2seq) pre-trained language models (PI.Ms) has ush- ered in a transformative for KPG, yielding promising performance improvements, How- ever, many design decisions remain unexplored and often made arbitrarily. This paper dertakes a systematic analysis of the influence of model selection and decoding strategies on PL.M-based KPG. We begin by elucidating why seq2seq PLMs are apt for KPG, anchored by an attention-driven hypothesis. We then establish that conventional wisdom for selecting seq2seq PLMs lacks depth: (I) merely increasing model size performing task-specific adaptation is not parameter-efficient; (2) although combin- ing in-domain pre- traiiraning with task adaptation benefits KPG, it does partially hinder general- ization. Regarding decoding, we demonstrate that while greedy search achieves strong Fl scores, it lags in recall compared with sampling- based methods. Based on these insights, we propose DESEL. likelihood-bart based decode- select algorithm for seq2seq PLMs. DESEI. improves greedy search by an average of 4.7% semantic F1 across five datasets. Our collec- tive findings pave the way for deeper future investigations into PLM-based KPG.
Recently, pre-trained language models (PLMs) have been widely incorporated in KPG (Chowd- hury et al., 2022; Zhao al., 2022) via sequence-to- sequence (seq2seq) generation, with promising per- formance on zero-shot (Kulkarni al., 2022), mul- tilingual (Gao et al., 2022), and low-resource (Wu et al., 2022a) KPG. However, existing literature typ- ically focuses on specific subset important com- ponents in this pipeline, such as data construction and loss design, while making arbitrary choices for the others (Zhao al., 2022; Ray Chowdhury et al., 2022; Wu et al., 2022a; Garg al., 2022). As result, KPG systems are often compared under different assumptions and the effect of the arbitrary design choices remains unclear. To bridge this gap. this paper focuses on two crucial questions that have not been systematically explored:
Which PLM leads to the best KPG perfor- mance when fine-tuned?
What is the best decoding strategy? In practice, sub-optimal choices for these factors could lead to optimizing unnecessarily large model or sub-optimal results decoded from strong KPG model. To answer these two questions, we conduct in-depth analyses on KPG with (1) PLMs of diverse size and pre-training strategies and (2) a diverse set of decoding strategies.
1 Introduction
Keyphrases encapsulate the core information of document. Due to their high information density, they have been found valuable in areas such as in- formation retrieval (Wu and Bolivar, 2008; Dave and Varma, 2010; Kim et al., 2013; Boudin et al., 2020), document clustering (Hammouda et al., 2005), summarization (Zhang et al., 2004), and text classification (Berend, 2011). A keyphrase is termed a present keyphrase if it explicitly found within the document and an absent keyphrase oth- erwise. The task of identifying present keyphrases is defined as keyphrase extraction (KPE), whereas
To begin with, we posit that seq2seq PLMs are inherently suitable to KPG ($3). By drawing the correlations with a strong graph-based KPE algo- rithm, we show that these PLMs implicitly compute phrase centrality (Boudin, 2013) in their decoder attention patterns. This knowledge is also directly translatable to a strong ranking function for KPE. On the other hand, encoder-only models fail to carry such centrality information.
Next, we search for the best seq2seq PLM for KPG fine-tuning ($4). While common strategies for other NLP tasks might advocate for (1) scaling

---

up the model size, (2) in-domain pre-training, or (3) task adaptation, do these approaches hold the same merit for KPG? Our findings reveal that sin-
4. Our research illuminates the profound impact of under-emphasized factors on KPG perfor- mance. To facilitate future research on KPG, we release our code and models at https: wttptttww..o.c.c./c..c...ec
gular emphasis on scaling or task adaptation does not ensure efficient performance improvement. In contrast, in-domain pre-training consistently bol- sters performance across both keyphrase types and can benefit from task adaptation. A robustness analysis reveals that a proper model choice and data-oriented training approaches are complemen- tary: without the latter, stronger PLMs are more vulnerable to perturbed input, with over 14% recall drop under named variation substitutions and over 5% recall drop under input paraphrasing.
2 Preliminaries
2.1 Keyphrase Generation
Problem Definition We represent an example for KPG as a tuple (X' corresponding to the input document X' = (2,, 22,...,...) z.) and the set of human- written reference keyphrases y {y1; 12. V.. Following Meng et al. (2017), Y/ is classified substring of X' or an absent keyphrase otherwise. The KPG task requires predicting y in any order, and the KPE task only requires predicting present keyphrases (Turney, 2000).
Decoding strategy is also an essential component in PLM-based KPG, but much under-explored by current literature. In $5, we thoroughly compare six decoding strategies including greedy search, beam search, and sampling-based methods. Re- sults suggest that when only generating single sequence consisting of concatenated keyphrases, greedy search achieves a strong Fl score. How- ever, aggregating the predictions from multiple sampled sequences outperforms greedy search due to much higher recall.
a present keyphrase if it is a
Evaluation We adopt lexical-based and semantic- based evaluation to evaluate model's predictions P {pI, P2, ........ Pm against V. For lexical eval- uation, we follow Chan et al. (2019) and use the P@M, R@M, and FI@M scores. P and are stemmed with the Porter Stemmer (Porter, 1980) and the duplicates are removed before the score calculation. For semantic evaluation, we follow Wu et al. (2023) and report SemP, SemR, and SemF1. Note that lexical metrics are separately calculated for present and absent keyphrases while the semantic metrics phrases. We repeat all the experiments with three random seeds and report the averaged performance.
Based on these findings, we introduce DESEL, a likelihood-basse selection strategy that selects from sampled phrases to augment the greedy search predictions. DESEL utilizes the probability of phrases from greedy search's predictions as the baseline to filter out noisy predictions from a set of sampled keyphrase candidates. Experi- ments on five KPG datasets show that DESEL consistently improves greedy decoding by 7.9% FI@M for present keyphrases, 25% FI@M for absent keyphrases, and 4.7% Semantic Fl for all keyphrases, achieving state-of-the-art KPG perfor- mance, underscoring the importance of carefully examining the design choice of KPG.
calculated with all the
Benchmark Meng et al. (2017) introduce KP20k. which contains 500k Computer Science papers. Following their work, we train on KP20k and eval- uate on the title and abstracts from the KP20k test set as well as four out-of-distribution testing datasets: Inspec (Hulth, 2003), Krapivin (Krapivin et al., 2009), NUS (Nguyen and Kan, 2007), and SemEEval (Kim et al., 2010). Table 5 summarizes the statisties of all testing datasets.
To summarize, our primary contributions are:
1. An in-depth exploration of the intrinsic suit- ability of seq2seq PLMS for KPG.
2. A comprehensive examination of effective strategies for model enhancement in KPG, spotlighting the merits of specific combina- tions and their implications for robustness.
Baselines We consider two strong supervised encoder-decoder models from Ye et al. (2021):
1. Copy A Transformer (Vaswani et al.,
3. We establish the trade-off between accuracy and concept coverage for different decoding algorithms. Then, we introduce a probability- based decode-seless mechanism DESEL that consistently improves over greedy search.
2017) with copy mechanism (See al., 2017).
2. The SetTrans model, which performs order- agnostic KPG. The model uses control codes trained via a k-step target assignment algo- rithm to generate keyphrases in parallel.

---

As the goal of this work is thoroughly studying PLM-based methods, we only provide the results of the strongest baselines as a point of reference. We also include other baselines in appendix section G. In our analysis, we also use MultipartiteRank (MPRank, Boudin (2018)), a performant graph- based unsupervised KPE algorithm. More details about MPRank are discussed in $3.1.
Model
Size
Head
Head
T
Encoder-only
PLMs
BERT-base
110M
3-0
0.300
3-0
0.206
BERT-lagge
340M
0-5
0.351
0-6
0.246
Decoder-only
PLMs
gp(2
117M
0-11
0.626
0-11
0.479
gp(2-medium
345M
1-6
0.630
1-6
0.480
gpt2-laree
774M
0-13
0.627
0-13
0.478
gp(2-xl
1.5B
0-6
0.626
0-6
0.476
Seq2seq PLMS
2.2
Sequence-fo-Sequenee PLMs
BART-bor
140M
0-6
0.608
0-6
0.459
In this work, focus on fine-tuning Transformer- based sequence-to-sequence PLMs BART (Lewis al., 2020) and T5 (Raffel et al., 2020) for KPG with the "One2Seq" formulation. Concretely, fol- lowing Yc and Wang (2018) and Yuan et al. (2020), we use a separator token; to join all the target keyphrases as the target sequence y = (y1 3n.. The models are trained with the cross- entropy loss for generating )' based on X. At test time, greedy decoding is used, followed by a post- processing stage to segment the output sequence into individual phrases. We provide implementa- tion details and hyperparameters in appendix D.
BART-
406M
0-9
0.585
0-9
0.438
TS-small
60M
4-4
0.624
4-4
0.471
TS-base
223M
8-4
0.621
8-4
0.466
TS-large
770M
0-2
0.628
0-2
0.471
T5-3B
3B
0-8
0.648
0-8
0.494
Table I: Correlation between keyphrase candidates' at- tention weights and centrality scores. i-h denotes atten- tion head h in layer 1. Both h and start from index 0. We report the attention head that achieves the best scores. The highest score is boldfaced.
performs lexical clustering to group the candidates into topic clusters. Next, each candidate is rep- resented as a graph node and connected with the candidates from other topic clusters. TextRank (Mihalcea and Tarau, 2004) is used to obtain a cen- trality score G for each of the nodes ni. We refer the readers to Boudin (2018) for further details.
3
Do PLMs inherently carry significant knowledge of keyphrases?
Existing studies have justified their of seq2seq PLMs by drawing the close relationship between the pre-training tasks of BART (denoising language modeling) or T5 (unified text-to-text transfer) and the formulation of KPG (Gao et al., 2022; Zhao et al., 2022: Wu et al., 2022a) or KPE (Kong et al.. 2023). However, there is lack of an in-depth understanding of why seq2seq PLMs should be chosen for keyphrase-related tasks. In this section, we reason based on phrase centrality (Litvak and Last, 2008; Boudin, 2013) and show that PL.Ms with autoregressive decoders, including seq2seq PLMs, carry attention heads that approximately function as centrality assigners and naturally as potent keyphrase rankers.
3.2 Attention intensities in BART and T5 decoders encode phrase centrality
Using MPRank lens, first investigate the tent to which PLMs implicitly represent centrality information. We use the paper titles and abstracts from the KP20k test set as the probing set. Each probing instance is fed into a PLM and the attention weights from the self-attention layers are collected. For the hh attention head at layer I, we denote the attention from token to token j as d;-oj. For the jth token in the noun phrase candidate n,, the global attention weight on it is
I.k
3.1 Centrality of Phrases
Lh
Σ
1 2
(1)
The concept of phrase centrality originated from graph-based KPE, where keyphrase candidates are represented as nodes. Various graph centrality mea- sures are used to determine a phrase's importance in the document. We use MPRank in our analysis, which encodes blaseness-based and eigenvector- based centrality (Boudin, 2013). MPRank first uses rules to obtain C' noun phrase candidates and then
where L is the length of the text after tokenization. Then, the attention weight of n, is calculated as
J.b = n Σ
1g
where |ny] denotes the number of tokens in n2- We study four families of models: BART, T5, BERT, and GPT-2 (Radford et al., 2019). For

---

BART and T5, we use their decoder attentions. We correlate de with qj using Spearman correla- tion P and Kendall's Tau т and present the best correlation for each model in Table 1. Surprisingly, BART and T5 decoders contain attention heads that encode phrase centrality similarly as MPRank. The head with the best correlation generally ap- pears in the lower layers. indicating that centrality understanding may be more related to low-level fea- tures. Also, the upper bound of correlation strength grows with model size for T5 while does not grow for BART. Beyond centrality assigners, these at- tention heads are also potent keyphrase extractors: simply ranking the noun phrase candidates by dj achieves similar F1 @5 for present keyphrases or Sem @5 score as MPRank (appendix B).
3.
Using task-adapted PLMs ($4.3).
The scaling law for keyphrase generation Although the effect of model sizes has been ex- plored for a range of tasks, it is poorly understood in the KPG literature, where most recent works em- ploy a single PLM with 100M to 500M parameters (Kulkarni et al., 2022; Wu ct al., 2022b; Zhao et al., 2022). To establish a common ground, we measure the performance of fine-tuning BART-base/large (purple line) and T5-smmal//base//arge//33 (green line) and report the results on KP20k in Figure 1. Surprisingly, fine-tuning BART or T5 ex- tremely parameter-ineffiieen compared to task- specific architectures trained from scratch'. For instance, although T5's performance consistently increases with the model size, around 8x more pa- rameters are required to achieve the same Present F1 Ô M on KP20k as SetTrans and 30x more pa- rameters are required to have better Sem F1. Closer inspection shows that SetTrans excels in recall via its parallel control codes and the set loss. In comparison, limited by the learning formulation and decoding strategy, fine-tuned seq2seq PLMs fall behind in their recall of important keyphrases. In $5, we will show that this problem can be allevi- ated with simple decode-then-seleed strategy.
4.1
Evaluating other types of PLMs, we find that BERT's attention heads only show weak centrality knowledge, with only 0.246 best Kendall Tau with MPRank. On the other hand, GPT-2 exhibits a similar pattern to the decoders from seq2seq PLMs, indicating that the observed pattern is strongly associated with autoregressive decoders. As centrality is generally correlated with global importance, our result aligns with the observations that masked language modeling tends to exploit local dependency while causal language modeling can learn long-range dependencies (Clark et al., 2019; Vig and Belinkov, 2019).
BART vs. T5 BART and T5 display similar scal- ing for FI@M and F1mF1. However, compared to T5, BART's recall scores increase more readily than the precision scores. At the same number of parameters, BART also performs better on absent keyphrases. One possible reason is that BART's text infilling objective is more advantageous for learning the knowledge for constructing spans ab- sent from text (Wu et al., 2022a).
In summary, through attention- based analy- ses, we reveal novel insights into the underlying keyphrase knowledge of PLMs with autoregressive decoders. Such knowledge can be employed ex- plicitly (via ranking) or implicitly (via fine-tuning and prompting) to facilitate KPG. In the rest of the paper, we focus on rethinking two basic designs for KPG with seq2seq PLMMs
Which score is more sensitive to scaling? Com- pared to recall, precision is more sensitive to model size. For example, T5-small achieves 98% SemR compared to the 50x larger T5-3B. In addition, ab- sent keyphrases scores are more sensitive. Overall, this suggests that small models are able to extract relevant keyphrases, but learn selectively omit unimportant keyphrases and create more absent keyphrases as the model size grows. Indeed, the average number of predicted keyphrases decreases from T5-small (6.75), T5-base (5.74), and TS-large (5.66), to T5-3B (5.48), while the number of ab- sent keyphrases increases from T5-small (0.91),
4 Influence of PLM Choice for KPG
Three crucial design options exist for using seq2seq PLMs for KPG: the choice of PLM to fine-tune, the fine-tuning data and objective, and the decoding strutegy. Previous work focuses on fine-tuning ob- jective and data construction (Meng et al., 2021; Ray Chowdhury et al., 2022: Garg et al., 2023) while often making the other two choices in an ad hoc way, making it difficult to compare among ap- proaches. This section dives into the first question by evaluating three "conventional wisdoms":
1. Using PLMs with more parameters ($4.1). 2. Using in-domain PLMs (64.2).
"We note that this claim is orthogonal to the observations that PLMs are fata-efficient (Wu et al., 2022a).

---

Prosent KPs Procision@
Present KPS Recall@M
Present F1@M
5.38
1DTTT
1D TAPPT
0.52
5.35
+ID-TAFT
+
+TPPT
e
0.34
-1D
0.50
++D+ IAPT
0.42
0.42
+
0.48
0.33
0.30
0.47
Copyllars
Sulthana RAKT
Seltrana
0.38
PART
ROAT
0.25
T3 FLAAATS
1 TAPP
15
FLANITS
RLANTTS
0.45
Model
Size
Size
Absent KPs oneCI
Absent kPs Recali@M
Absent KPs Fl@M
0.06
0.05
+DD
+I-TTAPT
+D0
TAPT
0.05
+D
+D+TAPP
5.06
+ID+ TAp's
TAPP
0.05
TAPT
5.05
0.04
5.04
0.03
Tuns
0.03
CopyTrrre
5.03
0.02
FLAGHTS
FLANITS
0.02
RLAN
107
10a
1.0
30a
18'
Model Size
Siz
Size
SemP
SemR
SemF1
* T2-TAPP
0.625
0.02
#IP:IAPP
DD-TAPT
0.59
+Ibb IAP
+ID+ TAPT
0.59
4.58
0.615
IDD
0.58
0.57
+1D
+DD
9.56
0.605
0.57
0.56
Copy Tuma Sectrons
0.595
TAPT
Copy
Copy/hera
30'
10¹
1.0
30a
107
10a
Model Size (M)
Model Size (M)
Model 1M)
Figure 1: KP20k performance of models of various sizes and pre-training strategies. +ID using in-domain SciBART. +TAPT second-stage training on OAGKX. The results are averaged over 3 random seeds.
T5-base (0.99), T5-large (1.01), T5-3B (1.05).
kens in total. The models are pre-trained on text infilling for 250k steps with batch size 2048, learn- ing rate 3e-4, 10k warm-up steps, and polynomial learning rate decay. We present data processing and model training details in appendix C.
4.2 Domain knowledge is crucial to accurate keyphrase generation
In-domain pre-training has been shown effective in a wide range of tasks requiring extensive domain knowledge (Beltagy et al., 2019; Lee et al., 2019). As keyphrases often contain domain-specinfe termi- nologies, we hypothesize that the domain of a PLM greatly affects its keyphrase generation ability. To test this hypothesis, we pre-train in-domain BART models SciBART-base and SciBART-Harge from scratch using the paper titles and abstracts from the S20RC dataset (Lo et al., 2020). The processed dataset contains 1.77M documents or 15.4B to-
The results of fine-tuning SciBART are pre- sented with "+ID" (for "In-Domiin") in Figure 1. As expected, SciBART significantly improves over BART for all three Fl metrics, outperforming the much larger T5-3B. Notably, SciBART also has better parameter efficiency compared general domain models: scaling from SciBART-base SciBART-large provides a much larger growth SemF1 compared to scaling up BART and T5.

---

4.3 Task-adaptive pre-training is more effective with in-domain models
Model
Before
After
A
Name Variation Substitution
BART-large
0.476
0.410
-0.666
(13.9%)
Task-adaptice pre-training (TAPT) is another com- mon technique for adding task-specific supervi- sion signals (Gururangan al.. 2020). In this sec- tion, we analyze the effect on KPG performance of adding two types of TAPT stages to seq2seq PLMs: keyphrase generation or instruction followino
SciBART-lare large
0.531
0.420
-0.131 (20.9%%)
KeyBART
0.463
0.398
-0.065 (14.2%))
SciBART- IIAge-TTTA
0.525
0.434
-0.091 (17.4%))
Paraphrasing
BART-large
0.463
0:453
-0.010 (2.3%L)
SciBART-large
0.524
0.494
-0.030 (5.8%..)
KeyBART
0,466
0.407
-0.059 (2
0.522
0.487
-0.035 (6.8%)
Keyphrase Pre-training We directly use Key- BART (Kulkarni et al., 2022) (denoted as "+TAPT" in Figure 1), which is trained using the OAGKX dataset (çano and Bojar, 2020) KPG with keyphrases corrupted from the input. To inves- tigate the effects of TAPT on in-domain PLMs, we also fine-tune SciBART on OAGKX with batch size 256, learning rate 3c-5, and 250k steps. We denote this model as "+ID+TAPT" in Figure 1.
Table 2: Results for the probing experiments. "Before" and "After" indicate the recall scores before and after perturbation. A denotes the score change from "Before" to "After". The hest score is boldfaced.
4.4 Analysis: are strong KPG models sensitive to input perturbations?
As in-domain and task-adapted PLMs already greatly benefit KPG, are data augmentation tech- niques no longer necessary? In this section, we reveal that these designs increase the model's sensi- tivity to input perturbations, and data augmentation is still desired for better generalization.
Instruction Pre-training Recently, instruction tuning has been introduced to improve the gen- eralization ability of PLMs (Mishra et al., 2022; Ouyang et al.. 2022). As KPG is relevant to clas- sic NLP tasks such as information extraction and summarization, we hypothesize that training with instruction data also serves as TAPT for KPG². To confirm, we benchmark FLAN-T5 (Chung al., 2022), a family of T5 models finc-tuned on instruc- tion following datasets (yellow line in Figure 1).
4.4.1 Method
We design two input perturbations on KP20k check the behaviors of BART-based KPG models.
Name variation substitution We construct 8905 perturbed inputs by replacing present keyphrases with their name variations linked by Chan et al. (2019). Ideally, robust KPG model would have a similar recall for the original phrases and the name variations as they appear in the same context.
TAPT struggles to improve absent KPG but is more effective with in-domain models. Figure I suggests that both TAPT strategies lead to a similar amount of improvement in the present keyphrase FI @M@M and SemF1. Surprisingly, the absolute gain is small and TAPT hardly improves absent keyphrase performance. For KeyBART, although its pre-training data (OAGKX) has a similar per- centage of absent keyphrases as KP20K (32% vs. 37%), its objective (recovering present keyphrases from corrupted input) might still be different from absent keyphrase generation. For FLAN-T5, we find that the KPG-related tasks in its pre- pre-training tasks often contain very short input text, represent- ing a significant distribution mismatch with KP20k. However, when applied on the in-domain SciBART, TAPT can grcatly improve the performance on KP20k. Combined with $4.2, we conclude that in-domain pre-training is more important for KPG and TAPT serves a complementary secondary role.
Paraphrasing We leverage gpt- 1..5-3..50 5-tuubb to paraphrase 1000 documents into a scientific writ- ing style. A good KPG model is expected to retain similar recall for phrases that present both before and after paraphrasing since the inputs describe the same ideas. The detailed prompt and several examples are presented in appendix E.
4.4.2 Results
Table 2 presents the perturbation results. We ob- serve that models often fail to predict name varia- tions when they appear in place of the original syn- onyms, while successfully maintaining their pre- dictions given paraphrased inputs. This indicates that the models may overfit the high-frequency keyphrases in the training set and input augmenta- tion methods such as Garg et al. (2023) are neces- sary to correct this pattern.
In fact, some variants of the keyphrase extraction task are included in popular instruction datascts such as NIv2 (Wang ct al., 2022) and Alpaca (Taori et al., 2023).

---

In addition, domain-specific or task-adapted models exhibit larger performance drop com- pared to BART-large, suggesting a trade-off be- tween domain/task specificity and generalization. Pre-trained on large-scale keyphrase data, Key- BART may rely more on syntax and position in- formation in the data and thus be less sensitive to synonym change. On the other hand. pre-trained on large-scale scientific corpus, SciBART is more ro- bust than KeyBART on different scientific writing styles beyond the ones available in KP20k.
Greedy search. Beam search. We the beam size the number of desired samples.
Diverse beam search (Vijayakumar et al., 2018). We set the number of groups to the number of desired samples and the weight of the dissimilarity term to >g 0.1.
4. Vanilla sampling. We further apply tempera- ture scaling 0.7.
Top-k sampling (Fan et al., 2018). We use temperature т 0.7 and 2 as we find a large harms the generation quality.
4.5 Discussion
Nucleus sampling (Holtzman et al., 2020). We set 0.95 and temperature = 0.5.
We summarize the main conclusions derived from the empirical results presented this section:
Figure 2 presents the semantic-based evalua- tion results as a function sample size. In the single sample setting, greedy search achieves a strong FemFl, only slightly outperformed by di- verse beam search. For other methods, we observe clear trade-off between their information cov- erage (Sem R) and the noise in the final output (SemP) as the number of samples grows. Never- theless, all these methods are able outperform greedy search at a certain sample size, indicating that single-sequence decoding is sub-optimal.
Naively scaling up BART and T5 is parameter- inefficient on KP20k compared to SetTrans.
Domain knowledge is crucial for KPG perfor- mance and improves parameter efficiency.
Task-adaptive training with keyphrase in- struction tuning data only significantly im- proves KPG with in-domain models.
In-domain pre-training and TAPT harm gen- cralization in different ways and data augmen- tation during fine-tuning is desired.
5.2
A simple decode-select strategy boosts the performance of greedy decoding
5 Decoding Strategy for KPG
Greedy search captures the correlations in human- written labels but suffers from local decisions and path dependency: high-quality keyphrases can be missed with improbable first tokens. However, naively outputting the union of multiple sampled sequences brings excessive noise. To achieve the balance between the two, we introduce DESEL, simple and effective three-stage decoding strategy:
While is well-known that decoding strategies can strongly affect text generation quality (Fan et al., 2018; Holtzman et al.. 2020), surprisingly there has been little study about decoding PlM-based KPG models. Previous studies often directly greedy search and variants of beam search (Gao et al., 2022; Zhao et al., 2022; Wu al., 2022a), limiting the understanding of PLMS fine-tuned for KPG. To bridge this knowledge gap, we first carefully evaluate six decoding strategies on the strongest PLM-based KPG model. We then propose simple yet effective decode-select strategy to mitigate the observed deficiencies of greedy search.
1. Decode one sequence G via greedy search.
2. Sample n sequences (S,...., to collect a set of candidate keyphrases S.
3. Select high quality phrases (81, (s1;..,, C and output the sequence (G; 81 ... 8m). For step 3,
estimate Pr(s; for every phrase 8; in the n samples and 22) for ev- ery phrase 9j G. Then, we use G as a baseline to select most m most probable s; that satisfies
5.1 Multi-sequence decoding: the trade-off between coverage and quality
We focus on decoding the SciBART-large++AA model fine-tuned KP20k, under the budget vary- ing from 1 to 20 samples. The following six de- coding algorithms are compared. For each algo- rithm, their hyperparameters are chosen based on the KP20k validation set.
Pr(s;(2) Z |G|
Σ Pr(g, 9,EG
(3)
where a hyperparameter controlling the trade- off between precision and recall. The probabil- ity estimation is obtained with either the original

---

0.65
0.51
0.60
0.75
0.59
0.70
0.58
0.65
0.45
0.57
0.40
18
32
15
18
Number of Samples
Number of Samples
Number Samples
gready smarch
beam search
divrs beamn search
varila sampling
Cop-t samsling
nudlus sampling
Figure 2: A comparison of six strategies for decoding from the SciBART+ model. Greedy search achieves strong performance while performing worse than beam search and sampling with multiple samples.
KP20k
Inspec
Krapivin
NUS
SemEval
Method
P
A
Sem
P
A
Sem
P
A
Sem
P
Sem
P
A
Sem
CopyTrans
0.376
0.046
0.562
0.333
0.023
0.569
0.365
0.063
0.547
0.429
0.044
0.579
0.321
0.022
0.377
SerTrans
0.391
0.058 0.585
0.328
0.030 0.573
0.375
0.072
0.560
0.446
0.055
0.597
0.342
0.029
0.396
ConKG
0.404
0.071
N/A
0.365
0.045
N/A
N/A
N/A
N/A
0.449
0.079
N/A
0.359
0.044
N/A
BART-large
0.392 0.047 0.575 0.333
0.024
0.565
0.347
0.051
0.517
0.435
0.048
0.586
0.311
0.024 0.381
SciBART-large
0.396 0.057 0.587 0.328 0.026
0.557
0.329
0.056 0.503
0.421
0.050
0.567
0.304
0.033 0.382
ТAPT
0.426 0.063 0.597 0.330 0.030 0.569
0.347
0.064
0.519
0.442
0.055
0.585
0.333
0.031 0.386
TAPT DESEL
0.431 0.076 0.612 0.402 0.036 0.611 0.352
0.086 0.546
0.449
0.068 0.610
0.341
0.040 0.402
Table 3: Testing results all datasets. P and A stand for F1 @M for present and absent keyphrases. Sem stands for SemF1. The best performance is boldfaced. 'copied from Zhao et al. (2022). The best entries each column statistically significantly higher than the second hest (p < 0.05) via paired bootstrap test. Full results Table 8.
model or a newly trained "one2one" model that learns to generate single keyphrase based on X.
Method
A Sem
Greedy scarch (G)
0.426 0.063 0.597
We use nucleus sampling with p= 0.95 and T = 0.5 for step 2, and set 10, 10, and 0.78.
Nucleus sampling (S)
0.385 0.074 0.599
Random Selection
0.385 0.072 0.596
Input Overlap
0.402 0.064 0.611
Table 3 presents the test results of important models in this paper. DESEL consistently improves the performance over the base model by large margin. In Table 4, we compare against other se- lection strategies including random selection, input overlap using a sentence transformer model, and FreqFS proposed in Zhao al. (2022). DESEL is the only method that consistently outperforms both greedy search and nucleus sampling.
FreqES (Zhao al., 2022)
0.426 0.072 0.610
DESEL (self) DESEL (one2one)
0.426 0.070 0.608
0.431 0.076 0.612
Table 4: A comparison across different decoding strate- gies. Methods below the dotted line merge G with S.
high-quality phrases from the diverse candidates.
Discussion Compared to the single-sequence de- coding baselines, DESEL wins by bringing in the diversity. Compared the baseline ranking meth- ods, DESEL wins by capturing the correlations be- tween labels (encoded in the greedy search outputs) and using the likelihood-baat criteria to filter out
Efficiency DESEL harms the inference latency generates multiple sequences. To improve the ef- ficiency, one optimization is reusing the encoder's outputs for all the decoding and scoring operations. We implemented this strategy and benchmarked it with the BART (base) model. DESEL. with = 10 greedy search and sampling sequences decoded) takes 3.8x time compared to greedy decoding.
Sttarting from Key BART, the one2one model can he effi- ciently trained. We provide more details in appendix F.

---

6 Related Work
Limitations
Keyphrase Generation Meng al. (2017) pro- pose the task of Deep Keyphrase Generation and a strong baseline model COpyRNN. Later works improve the architecture by adding corre- lation constraints (Chen et al., 2018) and linguis- tic constraints (Zhao and Zhang, 2019), exploiting learning signals from titles (Ye and Wang, 2018; Chen et al., 2019b), and hierarchical modeling the phrases and words (Chen et al., 2020). Ye and Wang (2018) reformulate the problem as gener- ating a sequence of keyphrases, while Ye et al. (2021) further uses a set generation formulation to remove the influence of difference target phrase ordering. Other works include incorporating rein- forcement learning (Chan et al., 2019; Luo et al., 2021), GANs (S Swaminathan et al., 2020), and uni- fying KPE with KPG (Chen et al., 2019a; Ahmad et al., 2021). Meng et al. (2021) conduct an empiri- cal study on architecture, generalizability, phrase order, and decoding strategies, with the main focus on models trained from scratch instead of PLMs.
While our study sheds light on important aspects of keyphrase generation (KPG) models, several limi- tations present opportunities for future research. First, our analysis focuses on model selection and decoding and thus uses default cross entropy loss and original training set without data aug- mentations. Investigating how the discussed de- sign choices with more recent data augmentation (Ray Chowdhury et al., 2022; Garg et al., 2022) or training strategies (Zhao et al., 2022) is an impor- tant future study. In addition, the best approach to combine the conclusions reached in this paper on long input KPG (Garg al., 2022) or KPG models trained with reinforcement learning (Chan et al., 2019; Luo al., 2021) worth future study. Second, while in-domain pre-training combined with task adaptation was found to enhance KPG performance, we did not fully investigate the under- lying mechanisms leading to these improvements. Further research could explore the interplay be- tween these two aspects and uncover more granular insights into how they improve KPG.
PLMs for KPG More recently, Wu et al. (2021), Chowdhury et al. (2022), Wu et al. (2022a), Gao et al. (2022), and Wu et al. (2022b) consider fine- tuning prefix-MMs or seq2seq PLMs for KPG. Kulkarni al. (2022) use KPG pre-training task to learn strong BART-based representations. Zhao et al. (2022) adopt optimal transport for loss design and propose frequency-band based filtering for decoding to improve BART-based KPG.
Finally, although we revealed compromise be- tween performance optimization and model robust- ness, we did not delve into designing new meth- ods for improving the robustness of these models against perturbed inputs. Future research could fur- ther explore techniques to mitigate this trade-off, developing models that maintain high performance while being resistant to input perturbations.
Ethics Statement
7 Conclusion
S2ORC and OAGKX are released under the Cre- ative Commons By 4.0 License. We perform text cleaning and email/URI. filtering on S2ORC to re- move sensitive information, and we keep OAGKX as-is. We use the keyphrase benchmarking datasets distributed by the original authors. No additional preprocessing is performed before fine-tuning ex- cept lower-casing and tokenization. We do not re-distribute any of the datasets used in this work. Potential risks of SciBART include accidental leakage of (1) sensitive personal information and (2) inaccurate factual information. For (1), we care- fully preprocess the data in the preprocessing stage to remove personal information, including emails and URLs. However, we had difficulties desensitiz- ing names and phone numbers the text because they overlapped with the informative content. For (2), since SciBART is pre- trained on scientific pa-
This paper systematically investigated model se- lection and decoding for building KPG models with seq2seq PLMs. Our analyses suggested much more nuanced patterns beyond the "conventional wisdom" assumed by the majority of current litera- ture. Our novel decoding strategy, DESEL, signifi- cantly improved the performance of greedy search across multiple datasets. More broadly, this study underscores the distinct nature of the KPG task. One should not blindly transpose conclusions or assumptions from other text generation tasks. In- stead, they warrant careful re-evaluation and emn- pirical validation. Our work also opens up exciting directions for future work, with deep groundings in keyphrase literature. For instance, making KPG models more robust, interpreting a KPG model, and designing better decoding algorithms for KPG.

---

pers, it may generate scientific-style statements that include inaccurate information. We encourage the potential users of SciBART not to rely fully on its outputs without verifying their correctness.
2018 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 2 (Short Pa- pers), pages 667-672, New Orleans, Louisiana. As- sociation for Computational Linguistics.
Pre-training SciBART and fine-tuning the large T5 models are computationally heavy, and we es- timate the total CO2 emission to be around 3000 kg using the calculation application provided by Lacoste et al. (2019). We will release the fine-tuned checkpoints and we document the hyperparameters in the appendix section to help the community reduce the energy spent optimizing PLMs for KPG and other various NLP applications.
Florian Boudin, Ygor Gallina, and Akiko Aizawa. 2020. Keyphrase generation for scientific document trieval. In Proceedings the 58th Annual Meeting of the Association for Computational Linguisties, pages 1118 1126, Online. Association for Computational Linguistics.
Erion Çano and Ondicj Bojar. 2020. Two huge title and keyword generation corpora of research articles. In Proceedings of the Twelfih Language Resources and Evaluation Conference, pages 6663-6671, 1, Marseille, France. European Language Resources Association.
Acknowledgments
Hou Pong Chan, Wang Chen, Lu Wang, and Irwin King. 2019. Neural keyphrase generation via reinforcement learning with adaptive rewards. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 2163-2174, Florence, Italy. Association for Computational Linguisties.
The research is supported in part by Taboola, NSF CCF-2200274, and an Amazon AWS credit award. We thank the Taboola team for the helpful discus- sion. We also thank anonymous reviewers, Da Yin, Tanmay Parekh, and other members of the UCLA- NLP group for their valuable feedback.
Jun Chen, Xiaoming Zhang. Yu Wu, Zhao Yan, and Zhoujun Li. 2018. Keyphrase generation with corre- lation constraints. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pages 4057-4066, Brussels, Belgium. Association for Computational Linguistics.
References
Wasi Ahmad, Xiao Bai, Soomin Lee, and Kai-Wei Chang. 2021. Select, extract and generate: Neu- ral keyphrase generation with layer-wise coverage attention. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 1389-1404, Online. Association for Computa- tional Linguistics.
Wang Chen, Hou Pong Chan, Piji Li, Lidong Bing, and Irwin King. 2019a. An integrated approach for keyphrase generation via exploring the power of re- trieval and extraction. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pages 2846-2856, Minneapolis, Minnesota. Association for Computational Linguistics.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019, SciB- ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCCNL, pages 3615- 3620, Hong Kong, China. Association for Computa- tional Linguistics.
Wang Chen, Hou Pong Chan, Piji Li, and Irwin King. 2020. Exclusive hierarchical decoding for deep keyphrase generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages (105, Online. Association for Computational Linguistics.
Gábor Berend. 2011. Opinion expression mining by ex- ploiting keyphrase extraction. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 1162-1170, Chiang Mai, Thailand. Asian Federation of Natural Language Processing.
Wang Chen, Yifan Gao, Jiani Zhang, Irwin King, and Michael R. Lyu. 2019b. Title-guided encoding for keyphrase generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 0668-6275.
Florian Boudin. 2013. A comparison of centrality mea- sures for graph- based keyphrase extraction. In Pro- ceedings of the Sixth International Joint Conference on Natural Language Processing, pages 834-838, Nagoya, Japan. Asian Federation of Natural Lan- guage Processing.
Md Faisal Mahbub Chowdhury, Gaetano Rossiello, Michael Glass, Nandana Mihindukulasooriya. and Alfio Gliozzo. 2022. Applying generic sequence-to- sequence model for simple and effective keyphrase generation.
Hyung Won Chung. Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay. William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.
Florian Boudin. 2018 Unsupervised keyphrase extrac- tion with multipartite graphs. In Proceedings of the

---

2022. Scaling instruction finetumedd language models. arXiv preprint 2XXv:22100114166.
Anette Hulth. 2003. Improved automatic keyword ex- traction given more linguistic knowledge. In Pro- ceedings of the 2003 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP '03, page 216-223, USA. Association for Computational Linguisties.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? ynalysis of BERT's attention. In Pro- ceedings of the 2019 ACL Workshop BlackboxNI.P: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy, Association for Com- putational Linguisties.
Su Nam Kim, Olena Medelyan, Min- Yen Kan, and Tim- othy Baldwin. 2010. SemEval-2010 task 5 Auto- matic keyphrase extraction from scientific articles. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 21-26, Uppsala. Sweden, Association for Computational Linguistics.
Kushal S. Dave and Vasudeva Varma. 2010. Pattern based keyword extraction for contextual advertising. In Proceedings of the 19th ACM International Con- ference on Information and Knowledge Management, CIKM '10, page 1885-1888, New York, NY, USA. Association for Computing Machinery.
Youngsam Kim, Munhyong Kim, Andrew Cattle, Julia Ormakhova, Suzi Park, and Hyopil Shin. 2013. Ap- plying graph-based keyword extraction to document retrieval. In Proceedings of the Sixth International Joint Conference on Natural Language Processing. pages 864-868, Nagoya, Japan. Asian Federation of Natural Language Processing.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 689-898, Melbourne, Australia, Association for Computational Linguistics.
Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, and Xiaoyan Bai. 2023. Promptrank: Unsupervised keyphrase extraction using prompt.
Yifan Gao, Qingyu Yin, Zheng Li. Rui Meng, Tong Zhao, Bing Yin, Irwin King, and Michael Lyu. 2022. Retrieval-augmeened multilingual keyphrase gener- ation with retriever- generraor iterative training. In Findings of the Association for Computational Lin- guistics: NAACL 2022, pages 1233-1246, Seattle, United States. Association for Computational Lin- guistics.
Mikalai Krapivin, Aliaksandr Autaeu, and Maurizio Marchese. 2009, Large dataset for keyphrases extrac- tion. Technical report, University of Trento.
Mayank Kulkarni, Debanjan Mahata, Ravneet Arora, and Rajarshi Bhowmik. 2022. Learning rich repre- sentation of keyphrases from text. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 891-906, Seattle, United States. Associ- ation for Computational Linguistics.
Krishna Garg, Jishnu Ray Chowdhury, and Cornelia Caragea. 2023. Data augmentation for low- resource keyphrase generation.
Krishna Garg, Jishnu Ray Chowdhury, and Cornelia Caragea. 2022. Keyphrase generation beyond the boundaries of title and abstract. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5809-5821, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019. Quantifying the carbon emissions of machine learning.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. Biobert: a pre-trained biomedical language representation model for biomedical text mining. CoRR, abs/1901.08746.
Suchin Gururangan. Ana Marasović, Swabha Swayamdipta, Kyle Lo. Iz Beltagy. Doug Downey, and Noah A. Smith. 2020. Don't it stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the A Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguisties. pages 7871 Online. Association for Computa- tional Linguistics.
pre-training
Khaled Hammouda, Diego Matute, and Mohamed S. Kamel. 2005. Corephrase: Keyphrase extraction for document clustering. In International workshop on machine learning and data mining in pattern recog- nition, pages 265-274.
Marina Litvak and Mark Last. 2008 Graph-bases key- word extraction for single-documen summarization. In Coling 2008: Proceedings of the workshop Multi- source Multilingual Information Extraction and Sum- marization, pages 17-24, Manchester, UK. Coling 2008 Organizing Committee.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text de- generation. In International Conference on Learning Representations.

---

Kyle Lo, Lucy Lu Wang. Mark Neumann. Rodney Kin- ney, and Daniel Weld. 2020. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Compu- lational Linguistics, pages 4969-4983 Online. Asso- ciation for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 (1440))1-67.
Yichao Luo, Yige Xu, Jiacheng Ye, Xipeng Qiu, and Qi Zhang. 2021. Keyphrase generation with fine- grained evaluation- guided reinforcement learning. In Findings of the Association for Computational Lin- guistics: EMNLP 2021. pages 497-507, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jishnu Ray Chowdhury, Seo Yeon Park, Tuhin Kundu, and Cornelia Caragea. 2022. KPDROP: Improving absent keyphrase generation. In Findings of the Asso- ciation for Compulational Linguistics: EMNLP 2022. pages 4853-4870, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Abigail See. Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1073- 1083, Vancouver, Canada. Association for Computa- tional Linguistics.
Rui Meng, Xingdi Yuan, Tong Wang, Sanqiang Zhao, Adam Trischler, and Daqing He. 2021. An empir- ical study on neural keyphrase generation. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 4985-5007, Online. Association for Computational Linguistics.
Avinash Swaminathan, Haimin Zhang, Debanjan Ma- hata, Rakesh Gosangi, Rajiv Ratn Shah, and Amanda Stent. 2020. A preliminary exploration of GANs for keyphrase generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 8021-8030, On- line. Association for Computational Linguistics.
Rui Meng. Sanqiang Zhao, Shuguang Han, Daqing He, Peter Brusilovsky, and Yu Chi. 2017. Deep keyphrase generation. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers). pages 582-592, Vancouver. Canada. Association for Computational Linguistics.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois. Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpaca: A strong, replicable instruction-follwwiin model. Stan- ford Center for Research on Foundation Models. hitps://rffm. stanford. edu/2023/03/11 //alppaaa// himl, 3(6):7.
Rada Mihalcea and Paul Tarau. 2004. TextRank: Bring- ing order into text. In Proceedings of the 2004 Con- ference on Empirical Methods in Natural Language Processing, pages 404 411, Barcelona, Spain. Asso- ciation for Computational Linguistics.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannanch Hajishirzi. 2022. Cross-task generaliza- tion via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( VVolume 1: Long Papers), pages 3470-3487, Dublin, Ireland. Association for Computational Linguisties.
Peter D Turney, 2000. Learning algorithms for keyphrase extraction. Information retrieval, 2:303- 336.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Thuy Dung Nguyen and Min Kan. 2007. Keyphrase extraction in scientific publications. In Asian Digital Libraries. Looking Back 10 Years and Forging New Frontiers, pages 317-326, Berlin, Hei- delberg. Springer Berlin Heidelberg.
Jesse Vig and Yonatan Belinkov. 2019. Analyzing the structure of attention in a transformer language model. In Proceedings of the 2019 ACI. Workshop BlackboxNI.P: Analyzing and Interpreting Neural Nenworks for NLP, pages 63-76, Florence, Italy, As- sociation for Computational Linguistics.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroli Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instruc- tions with human feedback. Advances in Neural Information P Processing Systems, 35:27730-2778
Ashwin Vijayakumar, Michael Cogswell, Ramprasaath Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2018. Diverse beam search for improved description of complex scenes. Proceed- ings of the AAAT Conference on Artificial Intelligence, 32(1).
Martin F. Porter. 1980. An algorithm for suffix stripping. Program, 40:211-200 1-218.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 18):9
Yizhong Wang. Swaroop Mishra, Pegah Alipoormo- labashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran,

---

Anjana Arunkumar, David Stap, Eshaan Pathak. Giannis Karamanolakis, Haizhi Lai, Ishan Puro- hit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohicha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Generaliza-
2020. One size does not fit all: Generating and evalu- ating variable number of keyphrases. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7961-7975, On- line. Association for Computational Linguistics.
Yongzheng Zhang, Nur Zincir-Heywood, and Evangelos Milios. 2004. World wide web site summarization. Web Intelli. and Agent Sys., 2(1):39-55 53.
Shen. 2022. Huper-Naturalllnsttuctin tion via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 5085-5109, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Guangzhen Zhao, Guoshun Yin, Peng Yang, and Yu Yao. 2022. Keyphrase generation via soft and hard seman- tic corrections. In Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Pro- cexsing, pages 7757-7768, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Di Wu, Wasi Ahmad, Sunipa Dev, and Kai-Wei Chang. 2022a. Representation learning for resource- constrained keyphrase generation. Findings of the Association for Computational Linguistics: EMNLP 2022, pages 700-716, Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
Jing Zhao and Yuxiang Zhang. 2019. Incorporating linguistic constraints into keyphrase generation. Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 5224- 5233, Florence, Italy. Association for Computational Linguistics.
Di Wu, Da Yin, and Kai-Wa wei Chang. 2023. Kpeval: Towards fine-grained semantic-basee evaluation of keyphrase extraction and generation systems, arXiv preprint 2..3.:2333..522
Huanqin Wu, Wei Liu, Lei Li, Dan Nie, Tao Chen, Feng Zhang, and Di Wang. 2021. UniKeyphrase: unified extraction and generation framework for keyphrase prediction. In Findings of the Association for Computational Linguistics: ACI-IJCNLP 2021, pages 825-835, Online, Association for Computa- tional Linguistics.
Huanqin Wu, Baijiaxin Ma, Wei Liu, Tao Chen, and Dan Nie. 2022b. Fast and constrained absent keyphrase generation by prompt-based learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages -455--1503.
Xiaoyuan Wu and Alvaro Bolivar. 2008 Keyword ex- traction for contextual advertisement. In Proceedings of the 17th International Conference on World Wide Web, WWW '08, page 1195-1196, New York, NY, USA. Association for Computing Machinery.
Hai Ye and Lu Wang. 2018 Semi-supervised learn- ing for neural keyphrase generation. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4142-4155 Brussels, Belgium. Association for Computational Linguistics.
Jiacheng Ye, Tao Gui, Yichao Luo, Yige Xu, and Qi Zhang. 2021. One2Set: Generating diverse keyphrases as set. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence Natural Language Processing (Volume I: Long Papers), pages 4598-4608 Online. Association for Computational Linguistics.
Xingdi Yuan, Tong Wang, Rui Meng. Khushboo Thaker, Peter Brusilovsky, Daqing He, and Adam Trischler.

---

Supplementary Material: Appendices
A Test Set Statisties
entries, we fix wrong Unicode characters, remove emails and URLs, and convert the text ASCII encoding' The final dataset contains 171.7M doc- uments or 15.4B tokens in total. We reserve 10k documents for validation and 10k for testing and use the rest as training data.
Table 5 summarizes the statistics of all testing datasets we use. We use the version distributed by Meng et al. (2017). In this distribution, SemEval's documents and keyphrases are already stemmed while the other datasets are not.
Vocabulary Beltagy et al. (2019) suggest that using a domain-specific vocabulary is crucial to downstream in-domain fine-tuning performance. Following their observations, we build cased BPE vocabulary in the scientific domain using the Sen- tencePiece library the cleaned training data. We set the vocabulary size to 30K.
Dataset
#Examples
#KP
%AKP
IKPI
KP20k
20000
5.3
37.1
2.0
Inspec
500
9.8
26.4
2.5
Krapivin
400
5.9
44.3
2.2
NUS
211
11.7
45.6
2.2
SemEval
100
14.7
57.4
2.4
Table Test sets statistics. #KP, %AKP, and refers the average number of keyphrases per document, the percentage of absent keyphrases, and the average num- ber of words each keyphrase contains.
Training For the pre-training objective, we only use text infilling as introduced in Lewis al. (2020). We mask 30% of all tokens each ex- ample, with the spans randomly sampled from a Poisson distribution (A = 3.5). For 10% of the spans selected to mask, we replace them with a random token instead of the mask token. We set the maximum sequence length to 512. The model is pre-trained for 250k steps with batch size 2048. learning rate 3c-4, 10k warm-up steps, and poly- nomial learning rate decay. We use the Adam optimizer for pre- -reetraining. Using Nvidia A100 GPUs (40G each), the training took eight days for SciBART-base and twelve days for SciBART-large..
B Attention heads as keyphrase extractors
We present the KPE performance of ranking noun phrase candidate using their attention intensities in Table We observe that BART and T5 con- tain attention heads that function well as keyphrase extractors, some of which even surpass the KPE performance of the strong MPRank algorithm.
Model
Size
Head
PKP F1
FemFF155
D Implementation details
MPRank
0.188
0.429
BART-base
140M
5-3
0.182
0.451
D.1 Keyphrase Generation
BART-large
406M
11-9
0.191
0.452
For keyphrase generation with BART and T5, we use Huggingface Transformers and train for 15 epochs with early stopping. We use learning rate 6e-5, polynomial decay, batesh size 64, and the Adam W optimizer. To fine-tune SciBART-basc and SciBART-large, we use the Translation task provided by fairseq? and train for 10 epochs. We use learning rate 3e-5, polynomial decay, and the AdamW optimizer.
T5-small
60M
4-4
0.183
0.420
bS-base
223M
0.164
0.388
T5-large
770M
17-15
0.187
0.436
T5-3B
3B
21-27
0.183
0.426
Table 6: Keyphrase extraction performance utilizing attention intensity as ranking criteria. BART-large's attention head 11-9 is even able to outperform MPRank. PKP present keyphrases.
We perform a careful hyperparameter search over the learning rate, learning rate schedule, batch size, and warm-up steps. The corresponding search spaces are 1e--,, 3e-5, 6e-5, le-4, 3e-4), (linear,
SciBART Pre-training Details
Corpus and Data Preprocessing The S2ORC dataset contains over 100M papers from variety of disciplines (Figure 3). We train on all the titles and abstracts to increase the coverage of different topics. After removing n or title-only
'We clean-text for data cleaning. httepp:://ithhbbe com/ google/ sentencepiede
"https://0e htt.b..ccm/tcccbbteeeeaaaaa f rieg
WWe use guess language for language detection.

---

Domain distribution of S2ORC
Medkinee
Biology
Chemistry
Engineering
Computer Science
Physics
Materials
Science
Mathematics
Psychelogy
Economics
Science
Business
Geology
Seciology
Environmental
History
Philosophy
10
12
Number of documents (million)
Figure 3: Domain distribution of the S2ORC dataset.
polynomial), (16, 32, 64, 128), and (500, 1000, 2000, 4000). The best hyperparameters sented in Table 7. For DESEL,, we tune the hy- perparameters on the validation set of KP20k. the search space for a is (0.5, 0.73, 0.78, 0.83, 0.88, 0.93, 0.98}.
Abstract: Voriginal_abstrae
pre-
Your Paraphrase: Title:
We observe that the large language model can follow the specified format when generating the paraphrased titles and abstracts. During the post- processing, we split the titles and the abstract from the response and directly use them for model test- ing. Figure 4 shows two examples of original and paraphrased text. The paraphrased text has high quality, preserves the scientific writing style, and retains most of the present keyphrases.
The fine-tuning experiments are run on a local GPU server with RTX 2080 Ti (1IG each) and A6000 GPUs (48G each). We use gradient accu- mulation to achieve the desired batch sizes.
D.2 Baselines
For CopyTraas and SetTrans, we rerun with the original implementations to measure their per- formance. We use the earliest version of Key- BART available at https:: org/ record/5745 (ooord/77848445 YOеTONLMJJA. For MPRank, use the implementation from pke*.
F One2one model for DESEL
To better capture Pr(s;(A() and (x) for DE- SEL without the interference from other keyphrases co-occuring in the label, we propose to train an "one2one" model that learns to generate a single keyphrase given input document. Starting from the KeyBART model, fine-tune on the KP20k training set for 0.5 epoch with learning rate 5e-5, batch size 32, and the Adam optimizer. We use a linear learning rate decay with 1000 warmup steps.
E Prompting GPT-3.5 for paraphrasing
To generate the paraphrased titles and abstracts used in $4.4, use the following prompt to query 5et-3.5-turbor
Paraphrase the following title and abstract for scientific paper. Your paraphrase should perfectly preserve
G All model testing results
information in er hill document and should d be as formal as the original text. Do not change the spelling or case the naned eatities in the original sentence. If the original entity all lower-cased, the first letter these names in your paraphrase.
We present the testing scores well as their stan- dard deviation in Table 8. In addition to the models discussed in the main text, we also provide the per- formance of CatSeq (Yuan et al., 2020), Ca- 2RF1 (Chan et al., 2019), ExHiRD-h (Chen et al., 2020) for further reference.
not upper-case
Title: (original_tine hal tittee
'https://gitae
thub. wtw.tttt.bbcooeeeeeooooooooettte cow/boudin fl/pk

---

Model
dropout
wdecay
optimizer bsz
#epoch
#warm-up
Ir
Ir schedule
Seq2seq PLMs
T5-small
0.1
0.01
AdamW
64
25
2000
6e-5
polynomial
T5-base
0.1
0.01
AdamW
64
15
2000
6e-5
polynomial
TS-large
0.1
0.01
AdamW
5
1000
6e-5
polynomial
T5-3B
0.1
0.01
AdamW
64
5
1000
6e-5
polynomial
BART-base
0.1
0.01
A
64
15
2000
6e-5
polynomial
BaRT-large
0.1
0.01
AdamW
64
10
2000
6e-5
polynomial
In-domain Seq2seq PLMs
SciBART-hase
0.1
0.01
A
32
10
2000
3e-5
polynomial
SciBART-large
0.1
0.01
A
64
10
2000
3e-5
polynomial
Task-adaptive Seq2seq Models
FLAN-T5-shn
0.1
0.01
Adam
64
25
2000
6e-5
polynomial
FLAN-T5-base
0.1
0.01
AdamW
15
2000
6e-5
polynomial
FLAN-TS-large
0.1
0.01
AdamW
64
5
1000
6e-5
polynomial
FLAN-TS-XL
0.1
0.01
AdamW
64
5
1000
6c-5
polynomial
KeyBART
0.1
0.01
AdamW
15
2000
3e-5
polynomial
SciBART-base + OAGKX
0.1
0.01
AdamW
1000
1e-5
polynomial
SciBART-large OAGKX
0.1
0.01
AdamW
64
5
1000
le-5
polynomial
Table 7: Hyperparameters for fine-tuning PLMs for keyphrase generation on KP20k. The hyperparameters are determined using the loss on the KP20k validation dataset. "wdecay" weight decay, "bsz" = batch size, "#warm-up" the number of warm-up steps, "Ir" learning rate, "Ir schedule" = learning rate decay schedule. We use carly stopping for all the models and use the model with the lowest validation loss as the final model.
Original title: hybrid analytical modeling of pending cache hits data prefetching and mshrs.. Original abstract: this article proposes techniques to predict the performance impact of pending cache hits, hardware prefetching, and miss status holding register resources on superscalar microprocessors using hybri hybrid analytical models. the proposed models focus on timeliness of pending hits and prefetches and account for a limited number of mshrs. they improve modeling accuracy of pending hits by 3.9 x and when modeling data prefetching, a limited number of mshrs. or both, these techniques result in average errors of 9.5 % to 17.8 % the impact of non uniform dram memory latency is shown to be approximated well by using a moving average of memory access latency.
Paraphrased title: a hybrid analytical model for pending cache hits data prefetching, and mshrs...
Paraphrased abstract: this scientific paper presents novel approach to predicting the performance impact of pending cache hits. hardware prefetching, and miss status holding register resources on superscalar microprocessors the proposed hybrid analytical models focus on the timeliness of pending hits and prefetches. while also accounting for a limited number of mshrs by utilizing these models, the accuracy of pending hit medeling is improved by 3.9 times. when modeling data prefetching. a limited number of mshrs, or both. these techniques result in average errors ranging from 9.5 % to 17.8 % additionally. the paper demonstrates that the impact of non uniform dram memory latency can he approximated well by using moving average of memory access latency
Original title: variant of parallel plane sweep algorithm for multicore systems.
Original abstract: parallel algorithms used in very large scale integration physical design bring significant challenges for their efficient and effective design and implementation. the rectangle intersection problem subset of the plane sweep problem. topic of computational geometry and component in design rule checking parasitic resistance capacitance extraction, and mask processing flows. variant of plane sweep algorithm that embarrassingly parallel and therefore easily scalable on multicore machines and clusters, while exceeding best known parallel plane sweep algorithms on real world tests, is presented in this letter.
Paraphrased title: a modified parallel plane sweep algorithm multicore systems.
Paraphrased abstract: the design and implementation of parallel algorithms for physical design in very large scale integration pose significant challenges.
rectangle intersection problem.. which is component in design rule checking parasitic resistance capacitance extraction. and mask processing flows is subset of plane sweep problem in computational geometry... this letter presents a variant of the plane sweep algorithm that embarrassingly parallel and can casily scaled on multicore machines and clusters. the proposed algorithm outperforms the best known parallel plane sweep algorithms on real world tests.
Figure 4: Examples of documents paraphrased by gpt-3.5 3. 5.5-turbo. We color the present keyphrases in blue.

---

IMI
KP20k
Inspec
Krapivin
NUS
SemEval
Method
Sem
Sem
Sem
Sem
Trained fcomd-craach
CaiSey
21M
0.367
0.032
NIA
0.262
0.008
NAA
0.354
0.036
N/A
0.397
0.028
N/A
0.283
0.028
NAA
ExHiRD-h
22M
0.374,
0.025.
NA
0.291;
0.016;
NA
0,308;
0.033;
N/A
NIA
N/A
N/A
0.282
0,021s
NAA
CopyTrrans
98M
0.376,
0,046,
0.562,
0.335,
0.023,
0.569.
0,365,
0.063,
0.547s
0.429
0.044
0.579,
0.321s
0,022,
0.377
0,391g
0.585
0.030,
0.072s
0.560,
0.446
0.055,r
0.342
0.029;
PLM-bbsed methods
CorrkG
140M
0.404
0.011
"NAA"
0.365
0.045
NVA
"NYA""
'NAA"
N7A"
0.449
0.079
N/A"
"0389"
"0.04""
N/A"
BART-hase
140M
0.388
0.042;
0.571g
0.323,
0.017;
0,561;
0,336,
0.049,
0.514,
0.424,
0.042,
0.581
0.3212:
0.021g
0.372
BART-large
0.392, 0.047;
0.575,
0,333
0.024, 0,565,
0.347,
0.051;
0.517;
D.435
0.048,
0,586;
0.311
0.024, 0.381e
KeyBART
406M
0.398, 0.047; 0.576
0.325;
0.023; 0.561
0,365
0.064, 0,533g
0.430
0,055;
0,582;
0,289,
0.365,
SciBART-base
124M
0.396; 0.052,
0.576:
0.328,
0.028, 0.562,
0.329;
0.054,
0,502,
0.421
0.053,
0,572,
0.304,
0,022) 0.376,
+ TAPT
124M
0.415; 0.052;
0.590,
0,330,
0.027, 0.573g
0,337,
0.057,
0.514;
0.424,
0.048,
0.579,
0.329, 0,024, 0.388,
SoiBART-arre
386M
0.026;
0,329
0.056s
0.503;
0.421,2
0,050;
0.567,
0,304;2
0.033;
+ TAPP
386M
0.426) 0.063;
0.597]
0.330,
0.030, 0.568,
0.347,
0,064;
0,520
0.442,1
0.055;
0,385;
0.333
0.031; 0,386,
T5-small
60M
0,346, 0.019;
0.543g
0,354g
0.026; 0,572g
0,298;
0.026]
0.495;
0,398
0.020,
0.5622
0.315;
0.013; 0,377
T5-base
223M
0,388,
0.034,
0.570g
0.339,
0.020; 0,574g
0,350;
0.043s
0.524;
0,440,
0.0512
0.589,
0.32613
0.020, 0.383,
TS-large
770M
0.39%
0.573
0.045, 0.536;
0.438;
0.042,
0.587;
0.321g
0.020; 0.391a
TS-38
3B
0.419;
0.046;
0.587
0.319;
0.021; 0.559
0.353,
0.048]
0.526)
0.460
0.080
0.600,
0,337;
0,0274 0.391a
FLAN-TS--mae small
60M
0.362,
0.024,
0.552,
0.354;
0.018; 0.569
0,313;
0.024, 0.498;
0.398
0,030,
0.568;
0.321
0.017, 0.385,
0.393
0.035,
0.574,
0.358
0.020, 0.577
0.340,
0.649,
0.514;
0.436,
0.029;
0.586
0,333,
0.017; 0.387
FLAN
770M
0,404,
0.036)
0.579
0.347
0.018;
0.347
0.44g
0.526,
0.441,
0,046,
0.59%,
0.327
0,021; 0.39%
FLAN TS XL.
3B
0.427;
0.049;
0.592
0.313,
0.024; 0.556,
0.356,
0.047s
0.517,
0.455,
0,054,
0.595,
0.332:
0.025, 0.392.
Table 8: Testing results all five datasets. P and A stand for F1@M for present keyphrases and absent keyphrases. Sem stands for SemF1. The reported results are averaged across three runs with different random seeds. The standard deviation each entry presented in the subscript. For example, 23.5g means an average 23.5 with a standard deviation of 0.6. We omit the subscript for methods with single run. tcopied from Zhao et al. (2022).


Reviews:
[{"review_id": "esH0PiecOO", "reviewer": "Reviewer_FavJ", "summary": "", "questions": "The lower F1-scores in case of \"absent keyphrases\". Have you investigated the reasons behind this?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful review.\n\n(1) The performance gain is inconsistent. Absent keyphrase performance is low. \n\nWe first note that the score improvement trends of present and absent keyphrases are consistent across the studied PLM settings (Figure 1) and datasets (last four rows of Table 3). In most of the cases (P, A, and Sem) SciBART+TAPT is better than BART or SciBART, and in all the cases, SciBART+TAPT+DECEL is the best among the four PLM settings. Indeed, our results show that SciBART+TAPT cannot outperform SetTrans. We hypothesize the difference comes from SetTrans’ novel set loss formulation, which enhances the training and decoding efficiency. On the other hand, SciBART+TAPT is limited by MLE training and greedy decoding. To bridge the gap, we introduce DESEL which significantly boosts the decoding performance of SciBART+TAPT. \n\nWe also note that it is expected that the results may not be state-of-the-art compared to CorrKG, as CorrKG introduces an effective loss term to boost the performance of BART. This is complementary to the main argument of the paper, that only considering the best model and decoding algorithm choices would bring significant performance gains without the optimizations made e.g., in CorrKG. \n\n(2) Typo.\n\nThank you for pointing this out. We will proofread and improve the writing in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well written and well organized.\n2. The authors provided detailed analysis and ablation study to claim their contribution.\n3. The paper tried to follow a clear reasoning behind the experimental results.\n4. The findings of the paper (in domain pre-training with task adaptive pre-training and proposed decoding strategy) will help the community in keyphrase generation task.", "reasons_to_reject": "The reasons behind the improvements of in domain pre-training+task adaptive pre-training is not clearly discussed. The experimental results showed improvement in all cases (present keyphrase, absent keyphrase) only for the KP20k dataset. However, in case of other datasets, the F1-score for absent keyphrase is lower than the traditional approaches.", "questions_to_authors": "The lower F1-scores in case of \"absent keyphrases\". Have you investigated the reasons behind this?", "paper_topic_and_main_contributions": "Paper topic: Keyphrase Generation, performance evaluation\n\nMain contributions:\n1. The paper systematically studied scaling up the model size, in-domain pre-training, and task adaptation and show that only certain combinations are effective in improving the KPG performance, with a compromise on robustness.\n2. The auhtors established the trade-off between accuracy and concept coverage for different decoding algorithms. Then, we introduce a probability-based decode-select mechanism DESEL that consistently improves over greedy search.\n3. The study sheds light on how previously under-explored factors can have profound influence on keyphrase generation performance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The line at 1153, \"We observe that the large language model can following the specified format when generating the paraphrased titles and abstracts.\" is not fully complete."}, "extra_text": {}}, {"review_id": "5tiTRyhu92", "reviewer": "Reviewer_TvwR", "summary": "", "questions": "Question A:\nSemEval, Krapivin and NUS datasets contains full texts apart from abstracts. The paper does not mention anywhere whether the full texts was used as input or only abstracts. Can you clarify which one?\n\nQuestion B:\nIf the answer of Question A is abstracts only, then how do you envisage your proposed DESEL approach will work when the input is full text (i.e. text consisting of several thousand words)?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful review.\n\n(1) How can the proposed approach scale for long input text?\n\nWe argue that scaling the approach to longer sequences is natural and in fact explored by previous work. Specifically, efficient attention has been shown effective in fine-tuning BART-like PLMs [1, 2]. In addition, the PLM can be trained to first compress the context and then generate the keyphrases [3]. The first section of the paper complements the long document fine-tuning approaches by exploring understudied factors such as model size, objective, or training data, providing useful guidance on selecting the best PLM to fine-tune. \n\nIn addition, it is unclear whether AutoKeyGen will be a better candidate with long document inputs. The main reason is that AutoKeyGen uses BiLSTM as the encoder, which may not handle long sequences as well as sparse transformers [1, 4]. There is also no existing work showing the AutoKeyGen can excel in the existing long document KPG datasets.\n\n(2) Paper organization. \n\nThank you for the suggestions. In the final revision, we will keep more relevant details within the main text and reserve space for a detailed discussion of the limitations of this study. \n\n(3) Which version of the datasets do you test on?\n\nAs this paper already comprehensively explores various settings, we follow the established KPG literature to restrict the scope of the study to extracting keyphrases from titles and abstracts [5, 6, 7, 8]. We leave the study for long documents to future work. \n\n(4) How can DESEL scale for long input text?\n\nDESEL can operate on any sequence-to-sequence KPG models and it is agnostic to the input size. To employ DESEL on long sequences, the main challenge is on the KPG model itself. As mentioned in (1), we would expect the best performance to be achieved by using DESEL with a BART model fine-tuned with efficient attentions on long-document KPG [2]. \n\nReferences\n\n[1] Longformer: The Long-Document Transformer (Beltagy et al., arXiv 2020)\n\n[2] Keyphrase Generation Beyond the Boundaries of Title and Abstract (Garg et al., Findings 2022)\n\n[3] Select, Extract and Generate: Neural Keyphrase Generation with Layer-wise Coverage Attention (Ahmad et al., ACL-IJCNLP 2021)\n\n[4] Generating long sequences with sparse transformers (Child et al., arXiv 2019)\n\n[5] Deep Keyphrase Generation (Meng et al., ACL 2017)\n\n[6] Neural keyphrase generation via reinforcement learning with adaptive rewards (Chan et al., ACL 2019)\n\n[7] One Size Does Not Fit All: Generating and Evaluating Variable Number of Keyphrases (Yuan et al., ACL 2020)\n\n[8] ONE2SET: Generating Diverse Keyphrases as a Set (Ye et al., ACL 2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Various analysis of the impact of existing design choices for KPG solutions\n2. Proposed approach achieved state-of-the-art results in several benchmarks.", "reasons_to_reject": "1. The proposed supervised approach will work for input small text (e.g. abstracts). But it is not obvious how this will scale for large text (i.e. full documents) where existing unsupervised approaches (e.g. AutoKeyGen (Shen et al, AAAI 2022) will likely do a better job with respect to the proposed approach.\n2. The writing and organization of the paper could be improved. The authors frequently referred to the Appendixes inside the main body (8 pages) of the paper for various optional things (e.g. analysis of MPRank) but could have used such spaces in the main body to discuss important things such as what limitations their approach have w.r.t. the length of the input text.", "questions_to_authors": "Question A:\nSemEval, Krapivin and NUS datasets contains full texts apart from abstracts. The paper does not mention anywhere whether the full texts was used as input or only abstracts. Can you clarify which one?\n\nQuestion B:\nIf the answer of Question A is abstracts only, then how do you envisage your proposed DESEL approach will work when the input is full text (i.e. text consisting of several thousand words)?", "paper_topic_and_main_contributions": "This paper proposes a new keyphrase generation (KPG) approach. Basically, the authors analyzed previous work by posing several hypotheses and then proposed an decode selection algorithm called DESE improves greedy search output considerably. The proposed approach achieved state-of-the-art results in several benchmarks.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZkLNZe6z5P", "reviewer": "Reviewer_3t8S", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful review.\n\n(1) Why DESEL improves performance compared with other decoding methods?\n\nThe success of DESEL is based on two insights collected from the empirical study in section 5.1: 1) greedy search produces high quality keyphrases and 2) path dependency reduces the diversity of greedy search, which is less of an issue for other decoding methods. Therefore, DESEL adopts the novel formulation of selecting high quality phrases from a diverse set of candidates to complement greedy search. Compared to the single-sequence decoding baselines, DESEL wins by bringing in the diversity. Compared to the ranking-based baselines (such as using ranking-based KPE methods to rerank KPG models’ outputs), DESEL wins by capturing the correlations between labels (encoded in the greedy search outputs) and using the probability-based formulation to filter out high quality phrases from the diverse candidates. We will further explain this point with qualitative examples in the final revision.\n\n(2) RL is not considered in this work.\n\nWe note that RL is an orthogonal factor to the dimensions considered in this work. Based on the existing works that report good performance on using RL for KPG [1, 2, 3], it is usually assumed that a supervised fine-tuned KPG model is required before starting RL training. However, how to obtain a strong PLM-based supervised model is not yet clear. We bridge this gap in this paper and delegate combining the proposed approaches with RL to future work. We will acknowledge this point in the limitation section.\n\nReferences\n\n[1] Neural keyphrase generation via reinforcement learning with adaptive rewards (Chan et al., ACL 2019)\n\n[2] Keyphrase Generation with GANs in Low-Resources Scenarios (Lancioni et al., sustainlp 2020)\n\n[3] Keyphrase Generation with Fine-Grained Evaluation-Guided Reinforcement Learning (Luo et al., Findings 2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper reviews the PLM-based KG methods and focuses on the decoding stage, which is exciting and lacks exploration.\n2. This paper summarizes and empirically explores the factors influencing generation performance.\n3. DESEL, the proposed probability-based decode-select algorithm, can improve the greedy search.", "reasons_to_reject": "1. It would be better if the authors could discuss in depth why DESEL improves performance compared with other decoding methods.\n2. Reinforcement learning has been proven effective in RNN-based KG methods, but this factor is not considered in this revisiting work.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper reviews the Keyphrase Generation (KG, or named KPG in this paper) task and concludes a better model selection strategy and decoding strategy for the KG task. From the perspective of model selection, this paper first provides some empirical results about the PLM selection and then explores the influence of in-domain pre-training and task-adaptive pre-training. From the perspective of decoding strategy, this paper compares six decoding algorithms and proposes DESEL which includes the decoding, sampling, and selection stages.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vC1q0iYzfT", "reviewer": "Reviewer_vDDj", "summary": "", "questions": "1.\tDoes the DESEL decoding strategy designed in this paper reduce the efficiency of inference? Has the author thoroughly explored this aspect?\n2.\tHow should one select the appropriate DESEL decoding hyperparameter α setting? There is a lack of relevant hyperparameter setting experiments in the paper.\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful review.\n\n(1) The baseline is limited and a little old. \n\nIn our preliminary studies, we also considered baselines beyond SetTrans, KeyBART, and CorrKG: AutoKeyGen [1], SEG-Net [2], CatSeqTG-2RF1 [3], ExHiRD-h [4], and UniKeyphrase [5]. However, we find that the non-PLM baselines have weaker results than SetTrans, and that UniKeyphrase is weaker than CorrKG. As the goal of this work is thoroughly studying PLM-based methods, we only provide the results of the strongest baselines as a point of reference. In our final revision, we will include the results of [1] - [5] in the appendix. \n\n(2) Scaling the results to larger models. \n\nWe did not consider models larger than 3B due to a limited computation budget. We note that in KPG applications, the set of desired keyphrases often changes quickly and the model may need to be frequently updated. Therefore, it is usually infeasible to fully fine-tune and serve models much larger than 3B. For these models, prompting may be a more feasible solution. However, a more fine-grained evaluation protocol may be required to assess their ability as it is unfair to directly compare with models trained with large supervised datasets [6, 7]. Therefore, we leave the investigation on them to future work. \n\n(3) Statistical significance.\n\nThank you for the suggestion. We have performed paired bootstrap testing for the results reported in the main text. For Table 3, the best entries in each column are statistically significant than the second best (p < 0.05) except the “P” column in NUS and the “Sem” column in SemEval. In Table 1 and Table 4, the best entries in each column are statistically significant than the second best (p < 0.05). We will add the statistical testing results in the final revision.\n\n(4) Inference efficiency of DESEL.\n\nWe acknowledge that DESEL will harm the inference latency because it generates multiple sequences. To reduce the latency, a simple optimization is caching and reusing the encoder’s outputs for all the decoding and scoring operations. We implemented this strategy and benchmarked with BART-base. DESEL with n = 10 (1 greedy search and 10 sampling sequences decoded) takes 3.8x time compared to greedy decoding. We will discuss the latency issue in our final revision.\n\n(5) How to tune α for DESEL?\n\nWe tuned on the KP20k validation set and used the same α=0.78 for all the testing datasets. The search space is {0.5, 0.73, 0.78, 0.83, 0.88, 0.93, 0.98}. We will add this to our final revision.\n\n\nReferences\n\n[1] Unsupervised Deep Keyphrase Generation (Shen et al., AAAI 2022)\n\n[2] Select, Extract and Generate: Neural Keyphrase Generation with Layer-wise Coverage Attention (Ahmad et al., ACL-IJCNLP 2021)\n\n[3] Neural keyphrase generation via reinforcement learning with adaptive rewards (Chan et al., ACL 2019)\n\n[4] Exclusive Hierarchical Decoding for Deep Keyphrase Generation (Chen et al., ACL 2020)\n\n[5] UniKeyphrase: A Unified Extraction and Generation Framework for Keyphrase Prediction (Wu et al., Findings 2021)\n\n[6] KPEval: Towards Fine-grained Semantic-based Evaluation of Keyphrase Extraction and Generation Systems (Wu et al., arXiv 2023)\n\n[7] Is ChatGPT A Good Keyphrase Generator? A Preliminary Study (Song et al., arXiv 2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper showcases thorough experiment on multiple datasets and various backbones. The experiment design is comprehensive, demonstrating a solid workload.\n2. The decoding strategy devised in this study is both straightforward and powerful, leading to significant improvements across multiple data sets.\n3. The paper is easy to follow and the question explored is of great important.\n", "reasons_to_reject": "1.\tThe baseline for comparison is too limited and a little old. Consider incorporating more strong baselines for a comprehensive evaluation.\n2.\tThe parameter scale of the selected backbone is relatively small. Hence, it raises questions about the existence of parameter inefficient on a larger scale model. And are the relevant experimental results still valid under these conditions?\n3.\tThere have been no tests to determine the statistical significance of the results.\n", "questions_to_authors": "1.\tDoes the DESEL decoding strategy designed in this paper reduce the efficiency of inference? Has the author thoroughly explored this aspect?\n2.\tHow should one select the appropriate DESEL decoding hyperparameter α setting? There is a lack of relevant hyperparameter setting experiments in the paper.\n", "paper_topic_and_main_contributions": "This paper presents systematic experiments on PLM choice, pre-training corpus, pre-training tasks, and decoding strategies in Keyphrase Generation and provides corresponding explanations and analyses. The motivation is that the current work often overlooks the choices of models, leading to KPG systems being compared under different backbones, which is not conducive to effectively comparing the effects of various designs.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Although the reviewers overall acknowledge several contributions of the paper, such as 1) detailed experiments and ablation study,  2) improvements over the baselines, and 3) analyses of under-explored factors that influence keyphrase generation performance, there are concerns such as the scalability of the method to longer texts, as well as further clarification on the results obtained.
Decision: Accept-Main