While keyword-oriented evaluation metrics like ROUGE-K offer valuable insights into how well system-generated summaries include essential words, there are some limitations and drawbacks associated with their use:
Dependency on Keyword Extraction: The effectiveness of ROUGE-K relies heavily on accurate keyword extraction from reference documents. If keywords are not extracted correctly or if important terms are missed during this process, it could lead to inaccurate evaluations.
Limited Semantic Understanding: ROUGE-K focuses primarily on surface-level matching between candidate summaries and predefined keywords without considering semantic nuances or contextual relevance. This limitation may result in overlooking subtle variations in meaning or missing out on synonyms/phrases conveying similar concepts.
Scalability Issues: Manually defining keywords for every dataset or domain may not be feasible for large-scale applications due to time constraints and resource-intensive nature. Automated methods for extracting keywords may introduce noise or inaccuracies affecting evaluation outcomes.
Subjectivity in Keyword Selection: The choice of which words constitute "keywords" is subjective and may vary depending on annotators' interpretations or biases. This subjectivity could introduce inconsistencies in evaluations across different datasets or evaluators.
Inability to Capture Overall Summary Quality: While ROUGE-K provides insights into keyword inclusion levels within summaries, it does not offer a comprehensive assessment of overall summary quality encompassing other critical aspects such as coherence, fluency, informativeness beyond specific keywords.