SlowGuess/NAACL_2021_OCR huggingface dataset:
Rui Meng† Xingdi Yuan‡ Tong Wang‡ Sanqiang Zhao†
Adam Trischler $^{\ddagger}$  Daqing He $^{\dagger}$
†School of Computing and Information, University of Pittsburgh
$^{\ddagger}$ Microsoft Research, Montréal
rui.meng@pitt.edu
Recent years have seen a flourishing of neural keyphrase generation (KPG) works, including the release of several large-scale datasets and a host of new models to tackle them. Model performance on KPG tasks has increased significantly with evolving deep learning research. However, there lacks a comprehensive comparison among different model designs, and a thorough investigation on related factors that may affect a KPG system's generalization performance. In this empirical study, we aim to fill this gap by providing extensive experimental results and analyzing the most crucial factors impacting the generalizability of KPG models. We hope this study can help clarify some of the uncertainties surrounding the KPG task and facilitate future research on this topic.
Keyphrases are phrases that summarize and highlight important information in a piece of text. Keyphrase generation (KPG) is the task of automatically predicting such keyphrases given the source text. The task can be (and has often been) easily misunderstood and trivialized as yet another natural language generation task like summarization and translation, failing to recognize one key aspect that distinguishes KPG: the multiplicity of generation targets; for each input sequence, a KPG system is expected to output multiple keyphrases, each a mini-sequence of multiple word tokens.
Despite this unique nature, KPG has been essentially "brute-forced" into the sequence-to-sequence (Seq2Seq) (Sutskever et al., 2014) framework in the existing literature (Meng et al., 2017; Chen et al., 2018; Ye and Wang, 2018; Chen et al., 2019b; Yuan et al., 2020; Chan et al., 2019; Zhao and Zhang, 2019; Chen et al., 2019a). The community has approached the unique challenges with much ingenuity in problem formulation, model design, and evaluation. For example, multiple target phrases
have been reformulated by either splitting into one phrase per data point or joining into a single sequence with delimiters (Figure 1), both allowing straightforward applications of existing neural techniques such as Seq2Seq. In accordance with the tremendous success and demonstrated effectiveness of neural approaches, steady progress has been made in the past few years — at least empirically — across various domains, including sub-areas where it was previously shown to be rather difficult (e.g., in generating keyphrases that are not present in the source text).
Meanwhile, with the myriad of KPG's unique challenges comes an ever-growing collection of studies that, albeit novel and practical, may quickly proliferate and overwhelm. We are therefore motivated to present this study as — to the best of our knowledge — the first systematic investigation on such challenges as well as the effect of interplay among their solutions. We hope this study can serve as a practical guide to help researchers to gain a more holistic view on the task, and to profit from the empirical results of our investigations on a variety of topics in KPG including model design, evaluation, and hyper-parameter selection.
The rest of the paper is organized as follows. We first enumerate specific challenges in KPG due to the multiplicity of its target, and describe general setups for the experiments. We subsequently present experimental results and discussions to answer three main questions:
1. How well do KPG models generalize to various testing distributions?
2. Does the order of target keyphrases matter while training One2Seq?
3. Are larger training data helpful? How to better make use of them?
2 Unique Challenges in KPG
Due to the multiplicity of the generation targets, KPG is unique compared to other NLG tasks such
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4985-5007 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 1: Top: comparison between One2One (left) and One2Seq (right) paradigms on the same data point. Bottom: demonstration of the decoding process for One2One (left) and One2Seq (mid/right) models. One2Seq can apply both beam search (mid) and greedy decoding (right).
as summarization and translation. In this section, we start from providing background knowledge of the KPG problem setup. Then we enumerate the unique aspects in KPG model designing and training that we focus on in this work.
Problem Definition Formally, the task of keyphrase generation (KPG) is to generate a set of keyphrases  $\{p_1,\dots ,p_n\}$  given a source text  $t$  (a sequence of words). Semantically, these phrases summarize and highlight important information contained in  $t$ , while syntactically, each keyphrase may consist of multiple words. A keyphrase is defined as present if it is a sub-string of the source text, or as absent otherwise.
Training Paradigms To tackle the unique challenge of generating multiple targets, existing neural KPG approaches can be categorized under one of two training paradigms: One2One (Meng et al., 2017) or One2Seq (Yuan et al., 2020), both based on the Seq2Seq framework. Their main difference lies in how target keyphrase multiplicity is handled in constructing data points (Figure 1).
Specifically, with multiple target phrases  $\{p_1,\ldots ,p_n\}$  ,One2One takes one phrase at a time and pairs it with the source text  $t$  to form  $n$  data points  $(t,p_i)_{i = 1:n}$  . During training, a model learns a one-to-many mapping from  $t$  to  $p_i$  's, i.e., the same source string usually has multiple corresponding target strings. In contrast, One2Seq concatenates all ground-truth keyphrases  $p_i$  into a single string:  $P = <  \mathrm{bos} > p_{1} <   \mathrm{sep} > \dots <  \mathrm{sep} > p_{n} <   \mathrm{eos}>$  (i.e., prefixed with  $<  \mathrm{bos}>$  , joint with  $<  \mathrm{sep}>$  , and suffixed with  $<  \mathrm{eos}>$  ), thus forming a single data point  $(t,P)$  . A system is then trained to predict the concatenated sequence  $P$  given  $t$  . By default, we con
struct  $P$  follow the ordering strategy proposed in (Yuan et al., 2020). Specifically, we sort present phrases by their first occurrences in source text, and append absent keyphrases at the end. This ordering is denoted as PRES-ABS in  $\S 4$ .
Architecture In this paper, we adopt the architecture used in both Meng et al. (2017) and Yuan et al. (2020), using RNN to denote it. RNN is a GRU-based Seq2Seq model (Cho et al., 2014) with a copy mechanism (Gu et al., 2016) and a coverage mechanism (See et al., 2017). We also consider a more recent architecture, Transformer (Vaswani et al., 2017), which is widely used in encoder-decoder language generation literature (Gehrmann et al., 2018). We replace both the encoder GRU and decoder GRU in RNN by Transformer blocks, and denote this architecture variant as TRANS. Both the RNN and TRANS models can be trained with either the One2One or One2Seq paradigm.
In recent years, a host of auxiliary designs and mechanisms have been proposed and developed based on either One2One or One2Seq (see §6). In this study, however, we focus only on the "vanilla" version of them and we show that given a set of carefully chosen architectures and training strategies, base models can achieve comparable, if not better performance than state-of-the-art methods. We assume that KPG systems derived from either One2One or One2Seq model would be affected by these factors of model designing in similar ways.
Decoding Strategies KPG is distinct from other NLG tasks since it expects a set of multi-word phrases (rather than a single sequence) as model predictions. Depending on the preference of po
Dataset	Present (F1@O)	Present (F1@10)	Absent (R@50)
One2One	One2Seq	One2One	One2Seq	One2One	One2Seq
RNN	TRANS	RNN	TRANS	RNN	TRANS	RNN	TRANS	RNN	TRANS	RNN	TRANS
D0	KP20K	35.3	37.4	31.2	36.2	27.9	28.9	26.1	29.0	13.1	22.1	3.2	15.0
KRAPIVIN	35.5	33.0	33.5	36.4	27.0	26.4	26.9	28.1	13.7	23.8	3.3	16.6
D0 Average	35.4	35.2	32.3	36.3	27.4	27.7	26.5	28.5	13.4	23.0	3.2	15.8
D1	INSPEC	33.7	32.6	38.8	36.9	32.5	30.8	38.7	36.6	8.2	9.2	3.7	6.7
NUS	43.4	41.1	39.2	42.3	35.9	36.1	36.6	37.3	11.2	18.9	2.9	12.5
SEMEVAL	35.2	35.1	36.2	34.8	34.6	33.0	35.0	34.2	6.1	18.9	1.7	12.5
D1 Average	37.4	36.3	38.1	38.0	34.4	33.3	36.7	36.0	8.5	12.7	2.8	9.2
D2	DUC	13.4	7.8	15.0	11.0	13.7	8.4	16.0	11.4	0.0	0.2	0.0	0.0
All	Average	32.8	31.2	32.3	32.9	28.6	27.3	29.9	29.4	8.7	14.0	2.5	9.8
Table 1: Testing scores across different model architectures, training paradigms, and datasets. In which,  $\mathbf{D}_0$ : in-distribution;  $\mathbf{D}_1$ : out-of-distribution, and  $\mathbf{D}_2$ : out-of-domain. We provide the average score over each category.
tential downstream tasks, a KPG system can utilize different decoding strategies. For applications that favor high recall (e.g., generating indexing terms for retrieval systems), a common practice is to utilize beam search and take predictions from all beams<sup>1</sup>. This is applicable in both One2One- and One2Seq-based models to proliferate the number of predicted phrases at inference time. In this work, we use a beam width of 200 and 50 for One2One and One2Seq, respectively. On the contrary, some other applications favor high precision and small number of predictions (e.g., KG construction), a One2Seq-based model is capable of decoding greedily, thanks to its nature of generating multiple keyphrases in a sequential manner.
As an example, we illustrate the two decoding strategies in Figure 1. Specifically, a One2One model typically collects output keyphrases from all beams and use the top  $k$  phrases as the model output ( $k = 5$  in the example). In One2Seq, either beam search or greedy decoding can be applied. For beam search, we use both the order of phrases within a beam and the rankings of beams to rank the outputs. In the shown example, top 5 beam search outputs are obtained from the 2 beams with highest rankings. As for greedy decoding, the decoder uses a beam size of 1, and takes all phrases from the single beam as outputs. In this way, the One2Seq model can determine the number of phrases to output by itself conditioned on  $t$ .
Evaluation Due to the multiplicity of targets in KPG task, the evaluation protocols are distinct from typical NLG tasks. A spectrum of evaluation metrics have been used to evaluate KPG systems, including metrics that truncate model outputs at a fixed number such as  $\mathbf{F}_1@5$  and  $\mathbf{F}_1@10$  (Meng et al., 2017); metrics that evaluate a model's ability of generating variant number of phrases such as
$\mathbf{F}_1@O$  and  $\mathbf{F}_1@M$  (Yuan et al., 2020); metrics that evaluate absent keyphrases such as Recall@50 (R@50). Detailed definitions of the metrics are provided in Appendix A. Due to space limit, we mainly discuss  $\mathbf{F}_1@O$ ,  $\mathbf{F}_1@10$  and R@50 in the main content, complete results with all common metrics are included in Appendix E. We save model checkpoints for every 5,000 training steps and report test performance using checkpoints that produce the best  $\mathbf{F}_1@O$  or R@50 on the KP20K validation set.
Datasets A collection of datasets in the domain of scientific publication (KP20K, INSPEC, KRAPIVIN, NUS, and SEMEVAL) and news articles (DUC) have been widely used to evaluate KPG task. Following previous work, we train models using the training set of KP20K since its size is sufficient to support the training of deep neural networks. Evaluation is performed on KP20K's test set as well as all other datasets without fine-tuning. Details of the datasets are shown in Appendix B.
3 Generalizability
In this section, we show and analyze the generalization performance of KPG systems from 2 dimensions: model architecture and training paradigm. Specifically, we compare the two model architectures (i.e., RNN and TRANS) as described in §2. For each model architecture, we train the KPG model using either of the training paradigms (i.e., One2One or One2Seq) also as described in §2.
To better understand model variants' generalization properties, we categorize the 6 testing sets into 3 classes according to their distribution similarity with the training data (KP20K), as shown in Table 1. Concretely, KP20K and KRAPIVIN are in-distribution test sets (denoted as  $\mathbf{D}_0$ ), since they both contain scientific paper abstracts paired with keyphrases provided by their authors. INSPEC, NUS and SEMEVAL are out-of-distribution test sets
This is in contrast to only taking the single top beam as in typical NLG tasks.
(denoted as  $\mathrm{D}_1$ ), they share the same type of source text with  $\mathrm{D}_0$ , but with additionally labeled keywords by third-party annotators. DUC is a special test set which uses news articles as its source text. Because it shares the least domain knowledge and vocabulary with all the other test sets, we call it out-of-domain test set (denoted as  $\mathrm{D}_2$ ).
Model Architecture: RNN vs TRANS The first thing to notice is that on present KPG, the models show consistent trends between  $\mathbf{F}_1@10$  and  $\mathbf{F}_1@O$ . We observe that TRANS models significantly outperform RNN models when trained with the One2Seq paradigm on  $\mathrm{D_0}$  test sets. However, when test data distribution shift increases, on  $\mathrm{D_1}$  test sets, RNN models starts to outperform TRANS; eventually, when dealing with  $\mathrm{D_2}$  test set, RNN outperforms TRANS by a large margin. On models trained with One2One paradigm, we observe a similar trend. On  $\mathrm{D_0}$  data, TRANS models achieve comparable  $\mathbf{F}_1@10$  and  $\mathbf{F}_1@O$  scores with RNN, when data distribution shift increases, RNN models produce better results.
On the contrary, for absent KPG, TRANS outperforms RNN by a significant margin in all experiment settings. This is especially obvious when models are trained with One2Seq paradigm, where RNN models barely generalize to any of the testing data and produce an average  $\mathbf{R}@\mathbf{50}$  of 2.5. In the same setting, TRANS models get an average  $\mathbf{R}@\mathbf{50}$  of 9.8, which is  $4\times$  higher than RNN.
To further study the different generation behaviors between RNN and TRANS, we investigate the average number of unique predictions generated by either of the models. As shown in Figure 12 in Appendix D, comparing results of order PRES-ABS in sub-figure a/b (RNN) with sub-figure c/d (TRANS), we observe that TRANS is consistently generating more unique predictions than RNN, in both cases of greedy decoding (4.5 vs 4.2) and beam search (123.3 vs 96.8). We suspect that generating a more diverse set of keyphrases may have a stronger effect on in-distribution test data. The generated outputs during inference are likely to represent the distribution learned from the training data, when the test data share the same (or similar) distribution, a larger set of unique predictions leads to a higher recall — which further contributes to their F-scores. In contrast, on test sets which data distribution is far from training distribution, the extra predictions may not be as useful, and even hurts precision. Similarly, because we evaluate ab
Figure 2: Present KPG testing scores  $(\mathbf{F}_1@ \mathcal{M})$ . Colors represent the relative performance, normalized per row.
sent KPG by the models' recall, TRANS models — produce more unique predictions — can always outperform RNN models.2
Training Paradigm: One2One vs One2Seq We observe that on present KPG tasks, models trained with the One2Seq paradigm outperforms One2One in most settings, this is particularly clear on  $\mathrm{D}_1$  and  $\mathrm{D}_2$  test sets. We believe this is potentially due to the unique design of the One2Seq training paradigm where at every generation step, the model conditions its decision making on all previously generated tokens (phrases). Compared to the One2One paradigm where multiple phrases can only be generated independently by beam search in parallel, the One2Seq paradigm can model the dependencies among tokens and the dependencies among phrases more explicitly.
However, on absent KPG, One2One consistently outperforms One2Seq. Furthermore, only when trained with One2One paradigm, an RNN-based model can achieve R@50 scores close to TRANS-based models. This may because a One2Seq model tends to produce more duplicated predictions during beam search inference. By design, every beam is a string that contains multiple phrases that concatenated by the delimiter  $<\mathrm{sep}>$ , there is no guarantee that the phrase will not appear in multiple beams. In the example shown in Figure 1, "topic tracking" is such a duplicate prediction that appears in multiple beams. In fact, the proportion of duplicates in One2Seq predictions
$^{2}$ Our TRANS and RNN models follow Vaswani et al. (2017) and Meng et al. (2017)'s hyper-parameter settings respectively. RNN is significantly lighter than TRANS. We conduct experiments with a much larger RNN but only observe marginal performance boost against Meng et al. (2017)'s setting.
Figure 3: Present KPG testing scores (top:  $\mathbf{F}_1@10$ , bottom:  $\mathbf{F}_1@O$ ). Colors represent the relative performance, normalized per row.
is more than  $90\%$ . This is in contrast with beam search on One2One models, where each beam only contains a single keyphrase thus has a much lower probability of generating duplication.3
4 Does Order Matter in One2Seq?
In the One2One paradigm (as shown in Figure 1), each data example is split to multiple equally weighted data pairs, thus it generates phrases without any prior on the order. In contrast, One2Seq training has the unique capability of generating a varying number of keyphrases in a single sequence. This inductive bias enables a model to learn dependencies among keyphrases, and also to implicitly estimate the number of target phrases conditioned on the source text. However, the One2Seq approach introduces a new complication. During training, the Seq2Seq decoder takes the concatenation of multiple target keyphrases as target. As pointed out by Vinyals et al. (2016), order matters in sequence modeling tasks; yet the ordering among the target keyphrases has not been fully investigated and its effect to the models' performance remains unclear. Several studies have noted this problem (Ye and Wang, 2018; Yuan et al., 2020) without further exploration.
Ordering Definition To explore along this direction, we first define nine ordering strategies for concatenating target phrases.
- RANDOM: Randomly shuffle the target phrases.
Because of the set generation nature of KPG, we expect randomly shuffled target sequences help to learn an order-invariant decoder.
- ORI: Keep phrases in their original order in the data (e.g., provided by the authors of source texts). This was used by Ye and Wang (2018).
- ORI-REV: Reversed order of ORI.
- S->L: Phrases sorted by lengths (number of tokens, from short to long).
- L->S: Reversed order of S->L.
- ALPHA: Sort phrases by alphabetical order.
- ALPHA-REV: Reversed order of ALPHA.
- PRES-ABS: Sort present phrases by their first occurrences in source text. Absent phrases are shuffled and appended to the end of the present phrase sequence. This was used by (Yuan et al., 2020).
- ABS-PRES: Similar to PRES-ABS, but preponding absent phrases to the beginning.
Greedy Decoding In Figure 2, we show the RNN and TRANS model's  $\mathbf{F}_1@\mathcal{M}$  on present KPG task, equipped with greedy decoding. In this setting, the model simply chooses the token with the highest probability at every step, and terminates either upon generating the <eos> token or reaching the maximum target length limit (40). This means the model predicts phrases solely relying on its innate distribution learned from the training data, and thus this performance could somewhat reflect to which
<sup>3</sup>Due to post-processing such as stemming, One2One model may still produce duplication.
degree the model fits the training distribution and understands the task.
Through this set of experiments, we first observe that each model demonstrates consistent performance across all six test datasets, indicating that ordering strategies play critical roles in training One2Seq models when greedy decoding is applied. When using the RNN architecture, RANDOM consistently yields lower  $\mathbf{F}_1@\mathcal{M}$  than other ordering strategies on all datasets. This suggests that a consistent order of the keyphrases is beneficial. However, TRANS models show a better resistance against randomly shuffled keyphrases and produce average tier performance with the RANDOM ordering. Meanwhile, we observe that PRES-ABS outperforms other ordering strategies by significant margins. A possible explanation is that with this order (of occurrences in the source text), the current target phrase is always to the right of the previous one, which can serve as an effective prior for the attention mechanism throughout the One2Seq decoding process. We observe similar trends in greedy decoding models'  $\mathbf{F}_1@\mathcal{O}$  and  $\mathbf{F}_1@\mathbf{10}$ , due to space limit, we refer readers to Figure 9, 10 in Appendix D.
Beam Search Next, we show results obtained from the same set of models equipped with beam search (beam width is 50) in Figure 3 (a/b). Compared with greedy decoding (Figure 10, Appendix D), we can clearly observe the overall  $\mathbf{F}_1@10$  scores have positive correlation with the beam width (greedy decoding is a special case where beam width equals to 1). We observe that compared to the greedy decoding case, the pattern among different ordering strategies appears to be less clear, with the scores distributed more evenly across different settings (concretely, the absolute difference between max average score and min average score is lower).
We suspect that the uniformity among different ordering strategies with beam search may be due to the limitation of the evaluation metric  $\mathbf{F}_1@10$ . The metric  $\mathbf{F}_1@10$  truncates a model's predictions to 10 top-ranked keyphrases. By investigation, we find that during greedy decoding, the number of predictions acts as a dominant factor, this number varies greatly among different ordering. With greedy decoding, PRES-ABS can generally predict more phrases than the others, which explains its performance advantage (Figure 13 (a/c), Appendix D). However, as the beam width increases,
all models can predict more than 10 phrases (Figure 13 (b/d), Appendix D). In this case, the  $\mathbf{F}_1@10$  is contributed more by a model' ability of generating more high quality keyphrases within its top-10 outputs, rather than the amount of predictions. Therefore, the performance gap among ordering strategies is gradually narrowed in beam search. For instance, we observe that the  $\mathbf{F}_1@10$  difference between PRES-ABS and S->L produced by RNN is  $3.5 / 2.0 / 1.0 / 0.2$  when beam width is 1/10/25/50.
To validate our assumption, we further investigate the same set of models' performance on  $\mathbf{F}_1@O$ , which strictly truncates the generated keyphrase list by the number of ground-truth keyphrases  $\mathcal{O}$  (where in most cases  $\mathcal{O} < 10$ ). Under this harsher criterion, a model is required to generate more high quality keyphrases within its top- $\mathcal{O}$  outputs. From Figure 3 (c/d), we observe that the scores are less uniformly distributed, this indicates a larger difference between different order settings. Among all orders, ORI produces best average  $\mathbf{F}_1@O$  with RNN, whereas ALPHA-REV and ORI-REV produce best average  $\mathbf{F}_1@O$  with TRANS.
In our curated list of order settings, there are 3 pairs of orderings with reversed relationship (i.e., S->L vs L->S, ALPHA vs ALPHA-REV, ORI vs ORI-REV). Interestingly, we observe that when beam search is applied, these orderings often show a non-negligible score difference with their counterparts. This also suggests that order matters since specific model architecture and training paradigm often has its own preference on the phrase ordering.