Empirical Study of Zero-shot Keyphrase Extraction with LLMs
Authors: Byungha Kang, Youhyun Shin


Reviewer paper summaries: This paper explores the effectiveness of large language models (LLMs) in zero-shot keyphrase extraction. It proposes several prompting strategies, including vanilla prompting, role prompting, candidate-based prompting, and a hybrid approach. Experimental results on six benchmark datasets reveal that the Llama3-8B-Instruct model, even with simple vanilla prompts, can significantly outperform classic keyphrase extraction methods.
Summary of strengths: 1. This paper is the first to systematically demonstrate that a prompting-based approach using LLMs can outperform existing unsupervised keyphrase extraction methods.
2. The paper proposes and thoroughly compares four different prompting strategies, covering vanilla prompting, role prompting, candidate-based prompting, and a hybrid approach.
3. Experiments are conducted on six widely-used benchmark datasets, showing that the Llama3-8B-Instruct model significantly outperforms the current state-of-the-art methods in F1@5, F1@10, and F1@15.
Summary of weaknesses: 1. While multiple prompting strategies are proposed, the theoretical and mechanistic explanations behind each strategy are not sufficiently deep, potentially affecting the reader's understanding of the method's effectiveness.
2. All experiments are conducted in a zero-shot setting, lacking exploration of few-shot or supervised learning scenarios, which may limit the applicability of the results.
3. The study focuses primarily on the Llama3-8B-Instruct model, lacking a broader comparison of different LLMs on the same task. At the same time, the comparison models are all small-size models. Does the improvement in results come from the difference in model capabilities rather than the method proposed in this article?
4. There is a lack of recent papers on unsupervised keyphrase extraction, e.g., [1-3].

[1] Is ChatGPT A Good Keyphrase Generator? A Preliminary Study 
[2] ChatGPT vs State-of-the-Art Models: A Benchmarking Study in Keyphrase Generation Task 
[3] Large Language Models as Zero-Shot Keyphrase Extractors: A Preliminary Empirical Study
Comments, suggestions and typos: see Weaknesses
Overall assessment scores: 2.5