PromptRank: Unsupervised Keyphrase Extraction Using Prompt
Year: 2023

Abstract:
The keyphrase extraction task refers to the automatic selection of phrases from a given document to summarize its core content. Stateof-the-art (SOTA) performance has recently been achieved by embedding-based algorithms, which rank candidates according to how similar their embeddings are to document embeddings. However, such solutions either struggle with the document and candidate length discrepancies or fail to fully utilize the pretrained language model (PLM) without further fine-tuning. To this end, in this paper, we propose a simple yet effective unsupervised approach, PromptRank, based on the PLM with an encoder-decoder architecture. Specifically, PromptRank feeds the document into the encoder and calculates the probability of generating the candidate with a designed prompt by the decoder. We extensively evaluate the proposed PromptRank on six widely used benchmarks. PromptRank outperforms the SOTA approach MDERank, improving the F1 score relatively by 34.18%, 24.87%, and 17.57% for 5, 10, and 15 returned results, respectively. This demonstrates the great potential of using prompt for unsupervised keyphrase extraction. We release our code at this url.

heading: 1 Introduction
text: Keyphrase extraction aims to automatically select phrases from a given document that serve as a succinct summary of the main topics, assisting readers in quickly comprehending the key information, and facilitating numerous downstream tasks like information retrieval, text mining, summarization, etc. Existing keyphrase extraction work can be divided into two categories: supervised and unsupervised approaches. With the development of deep learning, supervised keyphrase extraction methods have achieved great success by using advanced architectures, such as LSTM (Alzaidy et al., 2019;
∗Qicheng Li is the corresponding author.
Sahrawat et al., 2020) and Transformer (Santosh et al., 2020; Nikzad-Khasmakhi et al., 2021; Martinc et al., 2022). However, supervised methods require large-scale labeled training data and may generalize poorly to new domains. Therefore, unsupervised keyphrase extraction methods, mainly including statistics-based (Florescu and Caragea, 2017a; Campos et al., 2020b), graph-based (Bougouin et al., 2013; Boudin, 2018), and embedding-based methods (Bennani-Smires et al., 2018; Zhang et al., 2022), are more popular in industry scenarios.
Recent advancements in embedding-based approaches have led to SOTA performances that can be further divided into two groups. The first group of methods, such as EmbedRank (Bennani-Smires et al., 2018) and SIFRank (Sun et al., 2020), embed the document and keyphrase candidates into a latent space, calculate the similarity between the embeddings of the document and candidates, then select the top-K most similar keyphrases. Due to the discrepancy in length between the document and its candidates, these approaches perform less than optimal and are even worse for long documents. To mitigate such an issue, the second kind of approach is proposed. By leveraging a pre-trained language model (PLM), MDERank (Zhang et al., 2022) replaces the candidate’s embedding with that of the masked document, in which the candidate is masked from the original document. With the similar length of the masked document and the original document, their distance is measured, and the greater the distance, the more significant the masked candidate as a keyphrase. Though MDERank solves the problem of length discrepancy, it faces another challenge: PLMs are not specifically optimized for measuring such distances so contrastive fine-tuning is required to further improve the performance. This places an additional burden on training and deploying keyphrase extraction systems. Furthermore, it hinders the rapid adoption of large language models when more powerful PLMs
9788
emerge. Inspired by the work CLIP (Radford et al., 2021), in this paper, we propose to expand the candidate length by putting them into a well-designed template (i.e., prompt). Then to compare the document and the corresponding prompts, we adopt the encoder-decoder architecture to map the input (i.e., the original document) and the output (i.e., the prompt) into a shared latent space. The encoderdecoder architecture has been widely adopted and has achieved great success in many fields by aligning the input and output spaces, including machine translation (Vaswani et al., 2017), image captioning (Xu et al., 2015), etc. Our prompt-based unsupervised keyphrase extraction method, dubbed PromptRank, can address the aforementioned problems of existing embedding-based approaches simultaneously: on the one hand, the increased length of the prompt can mitigate the discrepancy between the document and the candidate. On the other hand, we can directly leverage PLMs with an encoder-decoder architecture (e.g., T5 (Raffel et al., 2020)) for measuring the similarity without any fine-tuning. Specifically, after selecting keyphrase candidates, we feed the given document into the encoder and calculate the probability of generating the candidate with a designed prompt by the decoder. The higher the probability, the more important the candidate.
To the best of our knowledge, PromptRank is the first to use prompt for unsupervised keyphrase extraction. It only requires the document itself and no more information is needed. Exhaustive experiments demonstrate the effectiveness of PromptRank on both short and long texts. We believe that our work will encourage more study in this direction.
The main contributions of this paper are summarized as follows:
• We propose PromptRank, a simple yet effective method for unsupervised keyphrase extraction which ranks candidates using a PLM with an encoder-decoder architecture. According to our knowledge, this method is the first to extract keyphrases using prompt without supervision.
• We further investigate the factors that influence the ranking performance, including the candidate position information, the prompt length, and the prompt content.
• PromptRank is extensively evaluated on six widely used benchmarks. The results show that
PromptRank outperforms the SOTA approach MDERank by a large margin, demonstrating the great potential of using prompt for unsupervised keyphrase extraction.

heading: 2 Related Work
text: Unsupervised Keyphrase Extraction. Mainstream unsupervised keyphrase extraction methods are divided into three categories (Papagiannopoulou and Tsoumakas, 2020): statisticsbased, graph-based, and embedding-based methods. Statistics-based methods (Won et al., 2019; Campos et al., 2020a) rank candidates by comprehensively considering their statistical characteristics such as frequency, position, capitalization, and other features that capture the context information. The graph-based method is first proposed by TextRank (Mihalcea and Tarau, 2004), which takes candidates as vertices, constructs edges according to the co-occurrence of candidates, and determines the weight of vertices through PageRank. Subsequent works, such as SingleRank (Wan and Xiao, 2008), TopicRank (Bougouin et al., 2013), PositionRank (Florescu and Caragea, 2017b), and MultipartiteRank (Boudin, 2018), are improvements on TextRank. Recently, embedding-based methods have achieved SOTA performance. To name a few, EmbedRank (Bennani-Smires et al., 2018) ranks candidates by the similarity of embeddings between the document and the candidate. SIFRank (Sun et al., 2020) follows the idea of EmbedRank and combines sentence embedding model SIF (Arora et al., 2017) and pre-trained language model ELMo (Peters et al., 2018) to get better embedding representations. However, these algorithms perform poorly on long texts due to the length mismatch between the document and the candidate. MDERank (Zhang et al., 2022) solves the problem by replacing the embedding of the candidate with that of the masked document but fails to fully utilize the PLMs without fine-tuning. To address such problems, in this paper, we propose PromptRank which uses prompt learning for unsupervised keyphrase extraction. In addition to statistics-based, graph-based, and embedding-based methods, AttentionRank (Ding and Luo, 2021) calculates self-attention and crossattention using a pre-trained language model to determine the importance and semantic relevance of a candidate within the document.
Prompt Learning. In the field of NLP, prompt learning is considered a new paradigm to replace
fine-tuning pre-trained language models on downstream tasks (Liu et al., 2021). Compared with fine-tuning, prompt, the form of natural language, is more consistent with the pre-training task of models. Prompt-based learning has been widely used in many NLP tasks such as text classification (Gao et al., 2021; Schick and Schütze, 2021), relation extraction (Chen et al., 2022), named entity recognition (Cui et al., 2021), text generation (Li and Liang, 2021), and so on. In this paper, we are the first to use prompt learning for unsupervised keyphrase extraction, leveraging the capability of PLMs with an encoder-decoder architecture, like BART (Lewis et al., 2020) and T5 (Raffel et al., 2020). Our work is also inspired by CLIP (Radford et al., 2021), using the prompt to increase the length of candidates and alleviate the length mismatch.

heading: 3 PromptRank
text: In this section, we introduce the proposed PromptRank in detail. The core architecture of our method is shown in Figure 1. PromptRank consists of four main steps as follows: (1) Given a document d, generate a candidate set C = {c1, c2, . . . , cn} based on part-of-speech sequences. (2) After feeding the document into the encoder, for each candidate c ∈ C, calculate the probability of generating the candidate with a designed prompt by the decoder, denoted as pc. (3) Use position information to calculate the position penalty of c, denoted as rc. (4) Calculate the final score sc based on the probability and the position penalty, and then rank
candidates by their sc in descending order.

heading: 3.1 Candidates Generation
text: We follow the common practice (Bennani-Smires et al., 2018; Sun et al., 2020; Zhang et al., 2022) to extract noun phrases as keyphrase candidates using the regular expression <NN. *|JJ> * <NN.*> after tokenization and POS tagging.

heading: 3.2 Probability Calculation
text: In order to address the limitations of embeddingbased methods as mentioned in Section 1, we employ an encoder-decoder architecture to transform the original document and candidate-filled templates into a shared latent space. The similarity between the document and template is determined by the probability of the decoder generating the filled template. The higher the probability, the more closely the filled template aligns with the document, and the more significant the candidate is deemed to be. To simplify the computation, we choose to place the candidate at the end of the template, so only the candidate’s probability needs to be calculated to determine its rank.
A sample prompt is shown in Figure 1. In Section 4.4, we investigate how the length and content of the prompt affect the performance. Specifically, we fill the encoder template with the original document and fill the decoder template with one candidate at a time. Then we obtain the sequence probability p(yi | y<i) of the decoder template with the candidate based on PLM. The length-normalized
log-likelihood has been widely used due to its superior performance (Mao et al., 2019; Brown et al., 2020; Oluwatobi and Mueller, 2020). Hence we calculate the probability for one candidate as follows:
pc = 1
(lc)α
j+lc−1∑
i=j
log p(yi | y<i), (1)
where j is the start index of the candidate c, lc is the length of the candidate c, and α is a hyperparameter used to regulate the propensity of PromptRank towards candidate length. We use pc whose value is negative to evaluate the importance of candidates in descending order.

heading: 3.3 Position Penalty Calculation
text: When writing an article, it is common practice to begin with the main points of the article. Research has demonstrated that the position of candidates within a document can serve as an effective statistical feature for keyphrase extraction (Florescu and Caragea, 2017b; Bennani-Smires et al., 2018; Sun et al., 2020).
In this paper, we use a position penalty to modulate the log probability of the candidate (as shown in Equation 1) by multiplication. The log probabilities are negative, so a larger value of the position penalty is assigned to unimportant positions. This results in a lower overall score for candidates in unimportant positions, reducing their likelihood of being selected as keyphrases. Specifically, for a candidate c, PromptRank calculates its position penalty as follows:
rc = pos
len + β, (2)
where pos is the position of the first occurrence of c, len is the length of the document, and β is a
parameter with a positive value to adjust the influence of position information. The larger the value of β, the smaller the role of position information in the calculation of the position penalty. That is, when β is large, the difference in contribution to the position penalty rc between two positions will decrease. Therefore, we use different β values to control the sensitivity of the candidate position.
We also observe that the effectiveness of the position information correlates with the document length. The longer the article, the more effective the position information (discussed in Section 4.4). Therefore, we assign smaller value to β for longer documents. Empirically, we formulate β which depends on the length of the document as follows:
β = γ
len3 , (3)
where γ is a hyperparameter that needs to be determined experimentally.

heading: 3.4 Candidates Ranking
text: After obtaining the position penalty rc, PromptRank calculates the final score as follows: sc = rc × pc. (4) The position penalty is used to adjust the log probability of the candidate, reducing the likelihood of candidates far from the beginning of the article being selected as keyphrases. We rank candidates by the final score in descending order. Finally, the top-K candidates are chosen as keyphrases.

heading: 4 Experiments

heading: 4.1 Datasets and Evaluation Metrics
text: For a comprehensive and accurate evaluation, we evaluate PromptRank on six widely used datasets,
in line with the current SOTA method MDERank (Zhang et al., 2022). These datasets are Inspec (Hulth, 2003), SemEval-2010 (Kim et al., 2010), SemEval-2017 (Augenstein et al., 2017), DUC2001 (Wan and Xiao, 2008), NUS (Nguyen and Kan, 2007), and Krapivin (Krapivin et al., 2009), which are also used in previous works (Bennani-Smires et al., 2018; Sun et al., 2020; Saxena et al., 2020; Ding and Luo, 2021). The statistics of the datasets are summarized in Table 1. Following previous works, we use F1 on the top 5, 10, and 15 ranked candidates to evaluate the performance of keyphrase extraction. When calculating F1, duplicate candidates will be removed, and stemming is applied.

heading: 4.2 Baselines and Implementation Details
text: We choose the same baselines as MDERank. These baselines include graph-based methods such as TextRank (Mihalcea and Tarau, 2004), SingleRank (Wan and Xiao, 2008), TopicRank (Bougouin et al., 2013), and MultipartiteRank (Boudin, 2018), statistics-based methods such as YAKE (Campos et al., 2020a), and embedding-based methods such as EmbedRank (Bennani-Smires et al., 2018), SIFRank (Sun et al., 2020), and MDERank(Zhang et al., 2022) itself. We directly use the results of the baselines from MDERank. For a fair comparison, we ensure consistency in both pre-processing and post-processing of PromptRank with MDERank. We also use T5-base (220 million parameters) as our model, which has a similar scale to BERT-base (Devlin et al., 2019) used in MDERank. Additionally, to match the settings of BERT, the maximum length for the inputs of the encoder is set to 512.
PromptRank is an unsupervised algorithm with only two hyperparameters to set: α and γ. PromptRank is designed to have out-of-the-box generalization ability rather than fitting to a single dataset. Hence we use the same hyperparameters to evaluate PromptRank on six datasets. We set α to 0.6 and γ to 1.2 × 108. The effects of these hyperparameters are discussed in Section 4.4.

heading: 4.3 Overall Results
text: Table 2 presents the results of the F1@5, F1@10, and F1@15 scores for PromptRank and the baseline models on the six datasets. The results show that PromptRank achieves the best performance on almost all evaluation metrics across all six datasets, demonstrating the effectiveness of the proposed method. Specifically, PromptRank out-
F1 Document Length
0
5
10
15
20
25
30
35
40
0
50
100
150
200
3000
5000
7000
9000
Inspec SemEval2010 SemEval2017 DUC2001 NUS Krapivin
EmbedRank MDERank PromptRank Document Length
Figure 2: Performance comparison of EmbedRank, MDERank, and PromptRank as the document length increases.
performs the SOTA approach MDERank, achieving an average relative improvement of 34.18%, 24.87%, and 17.57% for F1@5, F1@10, and F1@15, respectively. It is worth noting that while MDERank mainly improves the performance on two super-long datasets (Krapivin, NUS) compared to EmbedRank and SIFRank, our approach, PromptRank, achieves the best performance on almost all datasets. This highlights the generalization ability of our approach, which can work well on different datasets with different length of documents.
As the document length increases, the length discrepancy between documents and candidates becomes more severe. To further investigate the ability of PromptRank to address this issue, we compare its performance with EmbedRank and MDERank on the average of F1@5, F1@10, F1@15 across the six datasets. As the length of the document increases, the number of candidates increases rapidly, and the performance of keyphrase extraction deteriorates. As shown in Figure 2, EmbedRank is particularly affected by the length discrepancy and its performance drops quickly. Both MDERank and PromptRank mitigate this decline. However, the masked document embedding used in MDERank does not work as well as expected. This is due to the fact that BERT is not trained to guarantee that the more important phrases are masked, the more drastically the embedding changes. BERT is just trained to restore the masked token. By leveraging a PLM of the encoder-decoder structure and using prompt, PromptRank not only more effectively solves the performance degradation of EmbedRank on long texts compared to MDERank but also performs better on short texts than both of them.

heading: 4.4 Ablation Study
text: Effects of Position Penalty To evaluate the contribution of the position penalty to the overall performance of PromptRank, we conducted experiments in which candidates were ranked solely based on their prompt-based probability. The results are shown in Table 3. PromptRank without the position penalty outperforms MDERank significantly. When the position penalty is included, the performance is further improved, particularly on longtext datasets. This suggests that prompt-based probability is at the core of PromptRank, and position information can provide further benefits.
Effects of Template Length PromptRank addresses the length discrepancy of EmbedRank by filling candidates into the template. To study how long the template can avoid the drawback of EmbedRank, we conduct experiments using templates of different lengths, namely 0, 2, 5, 10, and 20.
Each length contains 4 hand-crafted templates (see details in Appendix A.2), except for the group with length 0, and the position information is not used. To exclude the impact of template content, for each template, we calculate the ratio of the performance of each dataset compared to the dataset Inspec (short text) to measure the degradation caused by an increase in text length. As shown in Figure 3, the higher the polyline is, the smaller the degradation is. Templates with lengths of 0 and 2 degenerate severely, facing the same problem as EmbedRank, making it difficult to exploit prompt. Templates with lengths greater than or equal to 5 better solve the length discrepancy, providing guidance for template selection.
Effects of Template Content The content of the template has a direct impact on the performance of keyphrase extraction. Some typical templates and their results are shown in Table 4 (no position in-
formation used). Template 1 is empty and gets the worst results. Templates 2-5 are of the same length 5 and outperform Template 1. Template 4 achieves the best performance on all metrics. Therefore, we conclude that well-designed prompts are beneficial. Note that all templates are manually designed and we leave the automation of template construction to future work.
Effects of Hyperparameter α The propensity of PromptRank for candidate length is controlled by α. The higher α is, the more PromptRank tends to select long candidates. To explore the effects of different α values, we conduct experiments where the position information is not used. We adjust α from
0.2 to 1, with a step size of 0.1. The optimal values of α on six datasets are shown in Table 5. Lgk is the average number of words in gold keyphrases. Intuitively, the smaller Lgk of the dataset, the smaller the optimal value of α. Results show that most datasets fit this conjecture. Note that SemEval2017 with the highest Lgk is not sensitive to α. The reason is that the distribution of gold keyphrases in the SemEval2017 dataset is relatively more balanced (see table 1). To maintain the generalization ability of PromptRank, it is recommended to select α that performs well on each benchmark rather than pursuing the best average F1 across all datasets. Therefore, we recommend setting the value of α to 0.6 for PromptRank.
Effects of Hyperparameter γ The influence of position information is controlled by β in Equation 2. The larger the β, the smaller the impact of the position information on ranking. Previous works (Bennani-Smires et al., 2018; Sun et al., 2020) show that the inclusion of position information can lead to a decrease in performance on short texts while improving performance on long texts. To address this, we dynamically adjust β based on the document length through the hyperparameter γ as shown in Equation 3, aiming to minimize the impact on short texts by a large β
while maximizing the benefits on long texts by a small β. Through experimentation, we determine the optimal value of γ to be 1.2 × 108. The average values of β calculated via γ on six datasets are shown in Table 5. As shown in Table 3, the performance of PromptRank on short texts remains unchanged while performance on long texts improves significantly. Effects of the PLM PromptRank uses T5-base as the default PLM, but to explore whether the mechanism of PromptRank is limited to a specific PLM, we conduct experiments with models of different sizes and types, such as BART (Lewis et al., 2020). The results, shown in Table 6, indicate that even when the hyperparameters and the prompt are optimized for T5-base, the performance of all models is better than the current SOTA method MDERank. This demonstrates that PromptRank is not limited to a specific PLM and has strong versatility for different PLMs of encoder-decoder structure. Our approach enables rapid adoption of new PLMs when more powerful ones become available.

heading: 4.5 Case Study
text: To demonstrate the effectiveness of PromptRank, we randomly select a document from the Inspec
dataset and compare the difference between the scores produced by MDERank and PromptRank in Figure 4. We normalize the original scores and present them in the form of a heat map, where the warmer the color, the higher the score, and the more important the candidate is. Gold keyphrases are underlined in bold italics. The comparison shows that compared to MDERank, PromptRank gives high scores to gold keyphrases more accurately and better distinguishes irrelevant candidates. This illustrates the improved performance of PromptRank over the SOTA method MDERank.

heading: 5 Conclusion
text: In this paper, we propose a prompt-based unsupervised keyphrase extraction method, PromptRank, using a PLM of encoder-decoder architecture. The probability of generating the candidate with a designed prompt by the decoder is calculated to rank candidates. Extensive experiments on six widelyused benchmarks demonstrate the effectiveness of our approach, which outperforms strong baselines by a significant margin. We thoroughly examine various factors that influence the performance of PromptRank and gain valuable insights. Additionally, our method does not require any modification to the architecture of PLMs and does not introduce any additional parameters, making it a simple yet powerful approach for keyphrase extraction.
Limitations
The core of PromptRank lies in calculating the probability of generating the candidate with a designed prompt by the decoder, which is used to rank the candidates. Our experiments have shown that the design of the prompt plays a crucial role in determining the performance of the method. While we have manually designed and selected some prompts to achieve state-of-the-art results, the process is time-consuming and may not guarantee an optimal result. To address this limitation, future research could focus on finding ways to automatically search for optimal prompts.

heading: Acknowledgements
text: The work was supported by National Key R&D Program of China (No.2022ZD0116307), National Natural Science Foundation of China (No. 62271270) and Lenovo Research ECR lab university collaboration program.

heading: A Appendix
text: A.1 Effects of the Noun Word We also design experiments to study the impact of the noun word representing the document (no position information used). We consistently use the best-performing template, and only vary the noun word. A total of five different words were tested. As illustrated in Table 7, the choice of noun word does affect the performance of the template, with "Book" achieving the highest results.
A.2 Templates for the Length Study We use five groups of templates of different lengths to explore the effect of template length. All the templates are shown in Table 8 and F1 here is the average of six datasets.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: We discuss the limitations after the conclusion, and before the references.
7 A2. Did you discuss any potential risks of your work? This paper discusses keyphrase extraction, which basically does not bring risks.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: The abstract is at the beginning of the article and the introduction is in Section 1. We summarize the paper’s main claims clearly in these two parts.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? The data we use to evaluate the proposed PromptRank is described in Section 4.1. The model PromptRank uses is described in Section 2, 4.2, and 4.4.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: The data we use to evaluate the proposed PromptRank is cited in Section 4.1. The model PromptRank uses is cited in Section 2, 4.2, and 4.4.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? The data and the model are so widely used in previous research works. Not discussing the license will not cause ambiguity or bring potential risks. For example, T5 is widely known and is publically available in Transformers (a python library) hence spending space on discussing the license of T5 in the paper is meaningless.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Our use of existing artifacts is consistent with their intended use and there is no potential risk. Spending space on this will make the paper a little strange
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We use the data as previous works did. For example, MDERank, a paper accepted by ACL 2022, does not discuss this. For keyphrase extraction, there is no potential risk.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? The domain of data is shown in Section 4.1.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. The relevant statistics of data are shown in Section 4.1. PromptRank is unsupervised so there are no train/test/dev splits.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? We run computational experiments and discuss relevant information in Section 4.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? We report the number of parameters of T5-base in Section 4.2.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? The setup of hyperparameters and prompts are discussed in Section 4.2 and 4.4.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We report our results in Section 4.3 and relevant descriptions are clear and accurate. There is no random element in the operation process of our method, so there is no need to discuss whether it is a single run or not.
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We do use some existing packages like NLTK for stemming or Stanford CoreNLP for pos-tagging. But the use of them does not involve the setting of parameters or something other worth reporting. No relevant description does not affect others to reproduce our work.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: To address this limitation, future research could focus on finding ways to automatically search for optimal prompts