SlowGuess/ACL_2020_OCR huggingface dataset:
- Transformer (Vaswani et al., 2017). A transformer-based sequence to sequence model incorporating with copy mechanism.
- catSeq (Yuan et al., 2018). An RNN-based attentional encoder-decoder model with copy mechanism. Both the encoding and decoding are sequential.
- catSeqD (Yuan et al., 2018). An extension of catSeq which incorporates orthogonal regularization (Bousmalis et al., 2016) and target encoding into the sequential decoding process to improve the generation diversity and accuracy.
- catSeqCorr (Chan et al., 2019). Another extension of catSeq, which incorporates the sequential decoding with coverage (See et al., 2017) and review mechanisms to boost the generation diversity and accuracy. This method is adjusted from Chen et al. (2018) to fit the new setting.
In this paper, we propose two novel models that are denoted as follows:
- ExHiRD-s. Our Exclusive HieRarchical Decoding model with the soft exclusion mechanism. In experiments, the window size  $K_{EL}$  is selected as 4 after tuning on the KP20k validation dataset.
- ExHiRD-h. Our Exclusive HieRarchical Decoding model with the hard exclusion mechanism. In experiments, the values of the window size  $K_{ES}$  are selected as 4, 1, 1, 1 for Inspec, Krapivin, SemEval, and KP20k respectively after tuning on the corresponding validation datasets.
We choose the bilinear attention from Luong et al. (2015) and the copy mechanism from See et al. (2017) for all the models.
We engage  $F_{1}$ @ $M$  which is recently proposed in Yuan et al. (2018) as one of our evaluation metrics.  $F_{1}$ @ $M$  compares all the predicted keyphrases by the model with ground-truth keyphrases, which means it does not use a fixed cutoff for the predictions. Therefore, it considers the number of predictions.
We also use  $F_{1}$  as another evaluation metric. When the number of predictions is less than five, we randomly append incorrect keyphrases until it obtains five predictions instead of directly using the original predictions. If we do not adopt such an appending operation,  $F_{1}$  will become the same with  $F_{1}$  when the prediction number is less than five.
The macro-averaged  $F_{1}$  and  $F_{1}$  scores are reported. When determining whether two keyphrases are identical, all the keyphrases are stemmed first. Besides, all the duplicated keyphrases are removed after stemming.
5.4 Implementation Details
Following previous work (Meng et al., 2017; Yuan et al., 2018; Chen et al., 2019a; Chan et al., 2019), we lowercase the characters, tokenize the sequences, and replace digits with “<digit>” token. Similar to Yuan et al. (2018), when training, the present keyphrase targets are sorted according to the orders of their first occurrences in the document. Then, the absent keyphrase targets are put at the end of the sorted present keyphrase targets. We use “<p_start>” and “<a_start>” as the
Model	Inspec	Krapivin	SemEval	KP20k
F1@M	F1@5	F1@M	F1@5	F1@M	F1@5	F1@M	F1@5
Transformer	0.2545	0.2107	0.32814	0.2524	0.3105	0.2574	0.3603	0.28210
catSeq	0.2765	0.2334	0.34414	0.2695	0.3138	0.26211	0.3681	0.2952
catSeqD	0.2803	0.2361	0.3449	0.2688	0.3116	0.2636	0.3682	0.2962
catSeqCorr	0.2533	0.2086	0.3439	0.2589	0.31818	0.26014	0.3673	0.2814
ExHiRD-s	0.2785	0.2353	0.3383	0.2780	0.3225	0.2765	0.3721	0.3070
ExHiRD-h	0.2913	0.2534	0.3474	0.2864	0.33517	0.28415	0.3740	0.3111
Table 2: Present keyphrase prediction results of all models on all datasets. The best results are bold. In all the tables of this paper, the subscript represents the corresponding standard deviation (e.g.,  $0.311_{1}$  indicates  $0.311 \pm 0.001$ ).
Model	Inspec	Krapivin	SemEval	KP20k
F1@M	F1@5	F1@M	F1@5	F1@M	F1@5	F1@M	F1@5
Transformer	0.0131	0.0061	0.0305	0.0143	0.0201	0.0131	0.0242	0.0111
catSeq	0.0083	0.0041	0.0334	0.0152	0.0172	0.0121	0.0231	0.0100
catSeqD	0.0104	0.0041	0.0337	0.0153	0.0161	0.0111	0.0231	0.0101
catSeqCorr	0.0072	0.0041	0.0226	0.0113	0.0215	0.0143	0.0231	0.0101
ExHiRD-s	0.0217	0.0092	0.0335	0.0162	0.0245	0.0164	0.0291	0.0140
ExHiRD-h	0.0223	0.0111	0.0436	0.0223	0.0256	0.0174	0.0320	0.0160
"[neopd]" token of present and absent keyphrases respectively. ";" is employed as the "[eowd]" token for both present and absent keyphrases. "
" is used as the "[eopd]" token.
The vocabulary with 50,000 tokens is shared between the encoder and decoder. We set  $d_{e}$  as 100 and  $d$  as 300. The hidden states of the encoder layers are initialized as zeros. In the training stage, we randomly initialize all the trainable parameters including the embedding using a uniform distribution in  $[-0.1, 0.1]$ . We set batch size as 10, max gradient norm as 1.0, and initial learning rate as 0.001. We do not use dropout. Adam (Kingma and Ba, 2014) is used as our optimizer. The learning rate decays to half if the perplexity on KP20k validation set stops decreasing. Early stopping is applied when training. When inference, we set the minimum phrase-level decoding step as 1 and the maximum as 20.
6.1 Present and Absent Keyphrase Predictions
We show the present and absent keyphrase prediction results in Table 2 and Table 3 correspondingly. As indicated in these two tables, both the ExHiRD-s model and the ExHiRD-h outperform the state-of-the-art baselines on most of the metrics, which demonstrates the effectiveness of our exclusive hierarchical decoding methods. Besides, the ExHiRD-h model consistently achieves the best results on both present and absent keyphrase pre
Table 3: Absent keyphrase prediction results of all models on all datasets. The best results are bold.
Model	Inspec	Krapivin	SemEval	KP20k
Transformer	0.28625	0.29746	0.22038	0.22341
catSeq	0.30211	0.2778	0.2002	0.2174
catSeqD	0.30414	0.2839	0.1991	0.2158
catSeqCorr	0.35238	0.3544	0.24923	0.28214
ExHiRD-s	0.21014	0.18212	0.1198	0.1376
ExHiRD-h	0.0306	0.1406	0.09110	0.1101
Table 4: The average DupRatios of predicted keyphrases on all datasets. The lower the score, the better the performance.
diction in all the datasets2.
6.2 Duplication Ratio of Predicted Keyphrases
In this section, we study the model capability of avoiding producing duplicated keyphrases. Duplication ratio is denoted as "DupRatio" and defined as follows:
$$
D u p R a t i o = \frac {\# d u p l i c a t i o n s}{\# p r e d i c t i o n s}, \tag {10}
$$
where # means "the number of". For instance, the DupRatio is 0.5 (3/6) for [A, A, B, B, A, C].
We report the average DupRatio per document in Table 4. From this table, we observe that our ExHiRD-s and ExHiRD-h consistently and significantly reduce the duplication ratios on all datasets. Moreover, we also find that our ExHiRD-h model achieves the lowest duplication ratios on all datasets.
2We also tried to simultaneously incorporate the soft and the hard exclusion mechanisms into our hierarchical decoding model, but it still underperforms ExHiRD-h.
Model	Inspec	Krapivin	SemEval	KP20k
#PK	#AK	#PK	#AK	#PK	#AK	#PK	#AK
Oracle	7.64	2.10	3.27	2.57	6.28	8.12	3.32	1.93
Transformer	3.1710	0.704	3.5729	0.634	3.2420	0.673	3.4417	0.584
catSeq	3.332	0.584	3.7010	0.635	3.455	0.643	3.704	0.512
catSeqD	3.334	0.582	3.6610	0.611	3.475	0.637	3.743	0.502
catSeqCorr	3.077	0.532	3.3914	0.561	3.153	0.621	3.364	0.501
ExHiRD-s	3.565	0.812	4.337	0.863	3.6914	0.796	3.942	0.691
ExHiRD-h	4.004	1.506	4.419	1.027	3.6513	0.994	3.973	0.811
Table 5: Results of average numbers of predicted unique keyphrases per document. "#PK" and "#AK" are the number of present and absent keyphrases respectively. "Oracle" is the gold average keyphrase number. The closest values to the oracles are bold.
Model	Present	Absent	DupRatio
F1@M	F1@5	#PK	F1@M	F1@5	#AK
ExHiRD-h	0.335	0.284	3.65	0.025	0.017	0.99	0.091
w/o HRD	0.320	0.274	3.58	0.018	0.013	0.97	0.093
w/o ES	0.330	0.278	3.51	0.022	0.014	0.70	0.191
6.3 Number of Predicted Keyphrases
We also study the average number of unique keyphrases predictions per document. Duplicated keyphrases are removed. The results are shown in Table 5. One main finding is that all the models generate an insufficient number of unique keyphrases on most datasets, especially for predicting absent keyphrases. We also observe that our methods can improve the number of unique keyphrases by a large margin, which is extremely beneficial to solve the problem of insufficient generation. Correspondingly, it also leads to overgenerate more keyphrases than the ground-truth for the cases that do not have this problem, such as the present keyphrase predictions on Krapivin and KP20k datasets. We leave solving the over-generation of present keyphrases on Krapivin and KP20k as our future work.
6.4 ExHiRD-h: Ablation Study
Since our ExHiRD-h model achieves the best performance on almost all of the metrics, we select it as our final model and probe it more subtly in the following sections. In order to understand the effects of each component of ExHiRD-h, we conduct an ablation study on it and report the results on the SemEval dataset in Table 6.
We observe that both our hierarchical decoding process and exclusive search mechanism are help
Table 6: Ablation study of our ExHiRD-h model on SemEval dataset. "w/o HRD" means the hierarchical decoder is replaced with a sequential decoder and the exclusive search is still incorporated. "w/o ES" represents our hierarchical decoding model without utilizing exclusive search mechanism.
KES	Present	Absent	DupRatio
F1@M	F1@5	#PK	F1@M	F1@5	#AK
Oracle	-	-	3.32	-	-	1.93	-
0	0.376	0.303	3.76	0.028	0.013	0.61	0.195
1	0.374	0.311	3.97	0.033	0.016	0.86	0.110
2	0.371	0.314	4.11	0.034	0.017	1.00	0.069
3	0.368	0.316	4.21	0.034	0.017	1.08	0.038
4	0.366	0.316	4.27	0.033	0.017	1.16	0.017
5	0.366	0.316	4.30	0.033	0.017	1.19	0.010
all	0.365	0.316	4.32	0.032	0.017	1.25	0.002
Table 7: Results of ExHiRD-h on KP20k with different window size  $K_{ES}$ . When  $K_{ES} = 0$ , ExHiRD-h equals to "w/o ES". The "all" means we taking the first words of all the previously-predicted keyphrases into consideration. The "DupRatio" is the average DupRatio per document. We show the average numbers of ground-truth keyphrases in the "Oracle" row.
ful to generate more accurate present and absent keyphrases. Besides, we also find that the significant performance margins on the duplication ratio and the keyphrase numbers are mainly from the exclusive search mechanism.
6.5 ExHiRD-h: Window Size of Exclusive Search
For a more comprehensive understanding of our exclusive search mechanism in our ExHiRD-h model, we also study the effects of the window size  $K_{ES}$ . We conduct the experiments on KP20k dataset and list the results in Table 7.
We note that a larger window size  $K_{ES}$  leads to a lower DupRatio as we anticipated. It is because the exclusive search can observe more previously-generated keyphrases to avoid generating duplicated keyphrases when  $K_{ES}$  is larger. When  $K_{ES}$  is "all", the DupRatio is not absolute zero because we stem keyphrases when determining whether they are duplicated. Besides, we also find that larger  $K_{ES}$  leads to better  $F_{1}@5$  scores. The reason is that for  $F_{1}@5$  scores, we append incorrect keyphrases to obtain five predictions when the number of predictions is less than five. A larger  $K_{ES}$  leads to predict more unique keyphrases, append less absolutely incorrect keyphrases and improve the chance to output more accurate keyphrases. However, generating more unique keyphrases may also lead to more incorrect predictions, which will degrade the  $F_{1}@M$  scores since  $F_{1}@M$  considers all the unique predictions without a fixed cutoff.
Model	Present	Absent	DupRatio
F1@M	F1@5	#PK	F1@M w	F1@5	#AK
Oracle	-	-	3.32	-	-	1.93	-
Transformer	0.360	0.282	3.44	0.024	0.011	0.58	0.223
catSeq	0.368	0.295	3.70	0.023	0.010	0.51	0.217
catSeqD	0.368	0.296	3.74	0.023	0.010	0.50	0.215
catSeqCorr	0.367	0.281	3.36	0.023	0.010	0.50	0.282
Transformer w/ ES	0.359	0.294	3.75	0.027	0.013	0.79	0.114
catSeq w/ ES	0.366	0.305	3.95	0.025	0.012	0.68	0.138
catSeqD w/ ES	0.366	0.306	3.99	0.026	0.012	0.65	0.137
catSeqCorr w/ ES	0.366	0.298	3.74	0.027	0.013	0.72	0.159
ExHiRD-h	0.374	0.311	3.97	0.032	0.016	0.81	0.110
6.6 ExHiRD-h: Incorporate Baselines with Exclusive Search
Our exclusive search is a general method that can be easily applied to other models. In this section, we study the effects of our exclusive search on other baseline models. We show the experimental results on KP20k dataset in Table 8.
From this table, we note that the effects of exclusive search on baselines are similar to the effects on our hierarchical decoding. We also see our ExHiRD-h still achieves the best performance on most of the metrics, even if baselines are also incorporated with exclusive search, which exhibits the superiority of our hierarchical decoding again.
6.7 ExHiRD-h: Case Study
We display a prediction example in Figure 3. Our ExHiRD-h model generates more accurate keyphrases for the document comparing to the four baselines. Besides, we also observe much less repeated keyphrases are generated by our ExHiRD-h. For instance, all the baselines produce the keyphrase "debugging" at least three times. However, our ExHiRD-h only generates it once, which demonstrates that our proposed method is more powerful in avoiding duplicated keyphrases.
In this paper, we propose an exclusive hierarchical decoding framework for keyphrase generation. Unlike previous sequential decoding methods, our hierarchical decoding consists of a phrase-level decoding process to capture the current aspect to summarize and a word-level decoding process to generate keyphrases based on the captured aspect. Besides, we also propose a soft and a hard exclusion mechanisms to enhance the diversity of the generated keyphrases. Extensive experimental results demonstrate the effectiveness of our meth
Table 8: Results of applying our exclusive search to other baselines on KP20k. The "w/ ES" means our exclusive search is applied.
SOC HW/SW co-verification based debugging technique. Purpose – Increasingly complex and sophisticated VLSI design, coupled with shrinking design cycles, requires shorter verification time and efficient debug method. ... SOC HW/SW co-verification technique seems to draw a balance, but Design Under Test (DUT) still resides in FPGA and remains hard for debugging. The purpose of this paper is to study a run-time RTL debugging methodology for a FPGA-based co-verification system. ...
Targets {computer hardware; computer software; co-verification; debugging}
Transformer: 1. co-verification (1), 2. debugging (7), 3. fpga (3) catSeq: 1. debugging (3), 2. logic programming (2) catSeqD: 1. debugging (4), 2. design (3), 3. verification (3) catSeqCorr: 1. debugging (3), 2. computer aided design (3) ExHiRD-h: 1. verification (2), 2. debugging (1), 3. simulation (1), 4. co-verification (1), 5. hdl (1), 6. computer software (2), 7. logic testing (1)
Figure 3: An example of generated keyphrases by baselines and our ExHiRD-h. The correct predictions are bold and the present keyphrases are underlined. The digit in parentheses represents the frequency that the corresponding keyphrase is generated by the model (e.g., "debugging (3)" means the keyphrase "debugging" is generated three times by the model).
ods. One interesting future direction is to explore whether the beam search is helpful to our model.
The work described in this paper was partially supported by the Research Grants Council of the Hong Kong Special Administrative Region, China (CUHK 2300174 (Collaborative Research Fund, No. C5026-18GF)). We would like to thank our colleagues for their comments.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. In ICLR 2014.
Gábor Berend. 2011. Opinion expression mining by exploiting keyphrase extraction. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 1162-1170, Chiang Mai, Thailand. Asian Federation of Natural Language Processing.
Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. 2016. Domain separation networks. In NeurIPS 2016, pages 343-351.
Hou Pong Chan, Wang Chen, Lu Wang, and Irwin King. 2019. Neural keyphrase generation via reinforcement learning with adaptive rewards. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2163-2174, Florence, Italy. Association for Computational Linguistics.
Jingqiang Chen and Hai Zhuge. 2018. Abstractive text-image summarization using multi-modal attentional hierarchical RNN. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4046-4056, Brussels, Belgium. Association for Computational Linguistics.
Jun Chen, Xiaoming Zhang, Yu Wu, Zhao Yan, and Zhoujun Li. 2018. Keyphrase generation with correlation constraints. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4057-4066, Brussels, Belgium. Association for Computational Linguistics.
Wang Chen, Hou Pong Chan, Piji Li, Lidong Bing, and Irwin King. 2019a. An integrated approach for keyphrase generation via exploring the power of retrieval and extraction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2846-2856, Minneapolis, Minnesota. Association for Computational Linguistics.
Wang Chen, Yifan Gao, Jiani Zhang, Irwin King, and Michael R. Lyu. 2019b. Title-guided encoding for keyphrase generation. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 6268-6275. AAAI Press.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gul- cehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724- 1734, Doha, Qatar. Association for Computational Linguistics.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889-898, Melbourne, Australia. Association for Computational Linguistics.
Sujatha Das Gollapalli, Xiaoli Li, and Peng Yang. 2017. Incorporating expert knowledge into keyphrase extraction. In AAAI 2017, pages 3180-3187.
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1631-1640, Berlin, Germany. Association for Computational Linguistics.
Anette Hulth. 2003. Improved automatic keyword extraction given more linguistic knowledge. In Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing, pages 216-223.
Anette Hulth and Beáta B. Megyesi. 2006. A study on automatically extracted keywords in text categorization. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 537-544, Sydney, Australia. Association for Computational Linguistics.
Su Nam Kim, Olena Medelyan, Min-Yen Kan, and Timothy Baldwin. 2010. SemEval-2010 task 5: Automatic keyphrase extraction from scientific articles. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 21–26, Uppsala, Sweden. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. CoRR, abs/1412.6980.
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017. OpenNMT: Open-source toolkit for neural machine translation. In Proceedings of ACL 2017, System Demonstrations, pages 67-72, Vancouver, Canada. Association for Computational Linguistics.
Mikalai Krapivin, Aliaksandr Autaeu, and Maurizio Marchese. 2009. Large dataset for keyphrases extraction. Technical report, University of Trento.
Yi Luan, Mari Ostendorf, and Hannaneh Hajishirzi. 2017. Scientific information extraction with semi-supervised neural tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2641-2651, Copenhagen, Denmark. Association for Computational Linguistics.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412-1421, Lisbon, Portugal. Association for Computational Linguistics.
Olena Medelyan, Eibe Frank, and Ian H. Witten. 2009. Human-competitive tagging using automatic keyphrase extraction. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 1318-1327, Singapore. Association for Computational Linguistics.
Rui Meng, Sanqiang Zhao, Shuguang Han, Daqing He, Peter Brusilovsky, and Yu Chi. 2017. Deep keyphrase generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 582-592, Vancouver, Canada. Association for Computational Linguistics.
Rada Mihalcea and Paul Tarau. 2004. TextRank: Bringing order into text. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 404-411, Barcelona, Spain. Association for Computational Linguistics.
Thuy Dung Nguyen and Min-Yen Kan. 2007. Keyphrase extraction in scientific publications. In ICADL 2007, pages 317-326.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. In NIPS-W.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Jiwei Tan, Xiaojun Wan, and Jianguo Xiao. 2017. Abstractive document summarization with a graph-based attentional neural model. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1171-1181, Vancouver, Canada. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 5998-6008.
Lu Wang and Claire Cardie. 2013. Domain-independent abstract generation for focused meeting summarization. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1395-1405, Sofia, Bulgaria. Association for Computational Linguistics.
Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 347-354, Vancouver, British Columbia, Canada. Association for Computational Linguistics.
Ian H. Witten, Gordon W. Paynter, Eibe Frank, Carl Gutwin, and Craig G. Nevill-Manning. 1999. KEA: practical automatic keyphrase extraction. In Proceedings of the Fourth ACM conference on Digital Libraries 1999, pages 254-255.
Denis Yarats and Mike Lewis. 2018. Hierarchical text generation and planning for strategic dialogue. In ICML 2018, pages 5587-5595.