SlowGuess/NAACL_2021_OCR huggingface dataset:
We also utilize five other datasets for evaluation purposes, as shown in Table 3. All except DUC come from scientific publications in Computer Science domain. KRAPIVIN uses keywords provided by the authors as targets, which is the same as KP20K. INSPEC, NUS, and SEMEVAL contain author-assigned keywords and additional keyphrases provided by third-party annotators. DUC, different from all above, is a keyphrase dataset based on news articles. Since it represents a rather different distribution from scientific publication datasets, hypothetically, obtaining decent test score on DUC requires extra generalizability.
C Other Model Designing Aspects
Besides the findings we discuss in the paper, there exist other important factors affect the general performance of KPG models. We provide two addi
Dataset	#Data	#kp	#unique kp	#(word) per kp
KP20K-train	514K	2.7M	700K	1.92
MAGKP	2.7M	41.6M	6.9M	3.42
MAGKP-LN	522K	2.3M	579K	2.73
MAGKP-Nlarge	1.5M	35.5M	5.8M	3.38
MAGKP-Nsmall	522K	12.2M	2.2M	3.37
Table 2: Statistics of training datasets.
Dataset	#Data	Avg#kp	%Pre
KP20K	≈20K	5.26	63.5%
INSPEC	500	9.83	79.8%
KRAPIVIN	460	5.74	56.5%
NUS	211	11.66	51.2%
SEMEVAL	100	15.07	44.7%
DUC	308	8.06	97.5%
Table 3: Statistics of testing datasets. Avg#kp indicates the average numbers of target keyphrases, %Pre denotes percentage of present keyphrases.
tional empirical results that we think might be of interest to certain readers.
C.1 Effect of Copy Mechanism
Copy mechanism (Gu et al., 2016) (also referred to as Pointer Generator (See et al., 2017) or Pointer Softmax (Gulcehre et al., 2016)) has demonstrated to play a critical role in tasks where texts on the source and target side may overlap, such as summarization (See et al., 2017) and keyphrase generation (Meng et al., 2017). Basically, it is an additional loss that enables models to extract information from the source side with the help of attentions. Prior studies (Meng et al., 2017) have shown the importance of copy mechanism with RNN+One2One, but no further comparison has been made.
In Figure 7, we present the results of four KPG model variants, equipped with and without copy mechanism. The results show that copy mechanism leads to considerable improvements on present KPG, especially for RNN. TRANS benefits less from the copy, which may be because its multi-head attentions behave similarly to the copy mechanism even without explicit training losses. With regard to the absent KPG results, copy mechanism only helps RNN+One2One. This suggests that TRANS can achieve consistently better abstractiveness (absent performance) by disabling the copy mechanism at the cost of weaker extractiveness. This dilemma cautions researchers to use copy mechanism more wisely according to specific applications.
C.2 Effect of Beam Width
As discussed in §2, one unique challenge of the KPG task is due to its multiplicity of target out
Figure 7: Averaged scores of models with and without utilizing copy mechanism.
Figure 8: Averaged scores of models using different widths of Beam Search.
puts. As a result, a common strategy is to take multiple beams during decoding in order to obtain more phrases (as opposed to greedy decoding). This choice is at times not only practical but in fact necessary: under the One2One paradigm, for example, it is crucial to have multiple beams in order to generate multiple keyphrases for a given input.
Generally speaking, KPG and its evaluation metrics are in general favors higher recall. It is thus not totally unexpected that the high precision scores of greedy decoding are often undermined by notable disadvantages in recall, which in turn leads to losing by large margins in F-scores when compared to results of beam search (with multiple beams). Empirically, as shown in Figure 8, we observe that beam search can sometimes achieve a relative gain of more than  $10\%$  in present phrase generation, and a much larger performance boost in absent phrase generation, over greedy decoding.
We are also interested in seeing if there exists an optimal beam width. In Figure 8, we show models' testing performance when various beam widths are
used. In present KPG task with One2One (upper left), beam width of 16 already provides an optimal score, larger beam widths (even 200) do not show any further advantage. Replacing the training paradigm with One2Seq (upper right), we observe a positive correlation between beam width and testing score — larger beam widths lead to marginally better testing scores. However, the improvement (from beam size of 10 to 50) is not significant.
On absent KPG task (lower), both One2One and One2Seq paradigms seem to benefit from larger beam widths. Testing score shows strong positive correlation with beam width. We observe that this trend is consistent across different model architectures.
Overall, a larger beam width provides better scores in most settings, but the performance gain diminishes quickly towards very large beam width. In addition, it is worth noting that larger beam width also comes with more intense computing demands, for both space and time. As an example, in Figure 8 (top left), we observe that with the One2One training paradigm, a beam width of 200 does not show a significant advantage over 16, however, in terms of computation, beam width of 200 takes about  $10 \times$  of the resources compared to 16. There clearly exists a trade-off between beam width and computational efficiency (e.g., carbon footprint (Strubell et al., 2019)). We thus hope our results can serve as a reference for researchers, to help them choose beam width more wisely depending on specific tasks.
D Does Order Matter in One2Seq? — Additional Results
In §4, we show models' performance trained with the One2Seq paradigm using different target ordering strategies. Here we provide the complete set of heat maps.
In Figure 9, we show present KPG testing scores in  $\mathbf{F}_1@O$ , when using either greedy decoding or beam search as decoding strategy.
In Figure 10, we show present KPG testing scores in  $\mathbf{F}_1@10$ , when using either greedy decoding or beam search as decoding strategy.
In Figure 11, we show absent KPG testing scores in  $\mathbf{R}@\mathbf{50}$ , when using either greedy decoding or beam search as decoding strategy.
In addition, we shown in Figure 12, 13, and 14 the number of unique predictions on all/present/absent KPG tasks.
E Complete Results
In this section, we report the full set of our experimental results.
In Table 4, we report all the testing scores on present keyphrase generation tasks. For all experiments, we use  $\mathbf{F}_1@5$ ,  $\mathbf{F}_1@10$ ,  $\mathbf{F}_1@\mathcal{O}$  and  $\mathbf{F}_1@\mathcal{M}$  to evaluate a model's performance. Additionally, we provide an average score for each of the 4 metrics over all datasets (over each row in Table 4).
In Table 5, we report all the testing scores on absent keyphrase generation tasks. For all experiments, we use R@10 and R@50 to evaluate a model's performance. Additionally, we provide an average score for each of the 2 metrics over all datasets (over each row in Table 5).
In table 7, we report detailed present testing scores when model trained with One2Seq paradigm, using different ordering strategies. For all experiments, we use  $\mathbf{F}_1@5$ ,  $\mathbf{F}_1@10$ ,  $\mathbf{F}_1@0$  and  $\mathbf{F}_1@M$  to evaluate a model's performance.
In table 8, we report detailed absent testing scores when model trained with One2Seq paradigm, using different ordering strategies. For all experiments, we use R@10 and R@50 to evaluate a model's performance.
We also provide scores against all ground-truth phrases (without splitting present/absent) in Table 6 and 9 to avoid the inconsistency in data processing (present/absent split may vary by ways of tokenization).
All the code and data have been released at https: //github.com/memray/OpenNMT-kpg-release, including the new MAGKP dataset.
We use the concatenation of title and abstract as the source text. When training with data points contains more than 8 ground-truth keyphrases, we randomly sample 8 from the list to build training target labels. This is to prevent jobs from out-of-memory issues and speed up the training.
We train RNN models for 100k steps and TRANS for 300k steps. TRANS generally benefits from longer training, especially for absent KPG performance. For the FT setting, we train models for additional 100k steps.
During the evaluation phase, we replace all punctuation marks with whitespace and tokenizer texts/phrases using Python string method split(), in order to reduce the errors in phrase matching and present/absent split.
Table 4: Detailed performance (F1-score) of present keyphrase prediction on six datasets. Best checkpoints are selected by present F1@O scores on KP20K-VALID. Boldface/underline text indicates the best/2nd-best performance in corresponding columns.
Table 5: Detailed performance (recall scores) of absent keyphrase prediction on six datasets. Best checkpoints are selected by absent R@50 scores on KP20K-VALID. Boldface/Underline text indicates the best/2nd-best performance in corresponding columns.
Table 6: Detailed performance (F1-score) of all (present+absent) keyphrase prediction on six datasets. Best checkpoints are selected by F1@O scores on KP20K-VALID. Boldface/Underline text indicates the best/2nd-best performance in corresponding columns.
Table 7: Detailed present keyphrase prediction performance (F1-score) of One2Seq trained with different orders. Best checkpoints are selected by present F1@O scores on KP20K-VALID.
Table 8: Detailed absent keyphrase prediction performance (recall scores) of One2Seq trained with different orders. Best checkpoints are selected by absent R@50 scores on KP20K-VALID.
Table 9: Detailed keyphrase prediction performance all phrases (present+absent) of One2Seq trained with different orders. Best checkpoints are selected by  $\mathbf{F}_1@\mathcal{O}$  scores on KP20K-VALID.
Figure 9: Present keyphrase generation testing scores  $(\mathbf{F}_1@O)$ . Colors represent the relative performance, normalized per row.
Figure 10: Present keyphrase generation testing scores  $(\mathbf{F}_1@10)$ . Colors represent the relative performance, normalized per row.
(a) Greedy Decoding, RNN
(b) Beam Size 50, RNN
(c) Greedy Decoding, Transformer
Figure 11: Absent keyphrase generation testing scores on  $\mathbf{R}@\mathbf{50}$ . Colors represent the relative performance, normalized per row.
Alpha Alpha-  $\mathsf{R e v}^{-\mathsf{S}} - \mathsf{L}$  L--S Ori ori-Rev Pres-AbsAbs-PresRandom
(d) Beam Size 50, Transformer
Alpha  $A l p h a_{-R e V}S_{- > L}$ $L - > S$  ori  $O r i_{- R e V}P r e s_{- A b s}A b s_{- P r e s}R a n d o m$
(c) Greedy Decoding, Transformer
Figure 12: Unique number of keyphrases generated during test. Colors represent the relative performance, normalized per row. Best checkpoints are selected by  $\mathbf{F}_1@O$  scores on KP20K-VALID.
Alpha Alpha-  $\mathsf{Re}_{\mathsf{v}}^{\mathsf{s}}\rightarrow \mathsf{L}$  L->S Ori ori-Rev Pres-AbsAbs-PresRandom
(d) Beam Size 50, Transformer
> L -->S ori-Rev Pres-Abs-PresRandom
(c) Greedy Decoding, Transformer
Figure 13: Unique number of present keyphrases generated during test. Colors represent the relative performance, normalized per row. Best checkpoints are selected by  $\mathbf{F}_1@O$  scores on KP20K-VALID.
Alpha Alpha-  $\mathsf{ReV}^{S - > L}$ $L\sim >S$  Ori ori-Rev Pres-Absb-PresRandom
(d) Beam Size 50, Transformer
Alpha Alpha-  $\mathsf{R}\mathsf{e}\mathsf{v}_{\mathsf{R}}^{\mathsf{S}\cdots >L}$  L-->S Ori ori-Rev Pres-Abs-PresRandom
(c) Greedy Decoding, Transformer
Figure 14: Unique number of absent keyphrases generated during test. Colors represent the relative performance, normalized per row. Best checkpoints are selected by  $\mathbf{R}@\mathbf{50}$  scores on KP20K-VALID.
Alpha Alpha-  $R_{\mathrm{ev}}^{S - > L}$ $L - > S$  ori Ori-Rev Pres-AbsAbs-PresRandom
(d) Beam Size 50, Transformer
Alpha Alpha-  $\mathsf{R}\mathsf{e}\mathsf{v}_{\mathsf{R}}^{\mathsf{S}\mathsf{-} > L}$ $L\rightarrow >S$  ori Ori-Rev Pres-Abs-PresRandom