Benchmarking LLMs on the Semantic Overlap Summarization Task
Authors: John Salvador, Naman Bansal, Mousumi Akter, Souvika Sarkar, Debajyoti Karmaker, Anupam Das, Shubhra Kanti Karmaker Santu


Reviewer paper summaries: This paper presents a benchmark study on the performance of large language models (LLMs) in the Semantic Overlap Summarization (SOS) task. Utilizing the recently proposed TELeR prompt design taxonomy, the authors generated and evaluated 905,216 summaries to assess 16 popular LLMs. The evaluation was conducted using automated metrics such as ROUGE, BERTscore, and SEM-F1, and supplemented with human evaluations on a subset of the data.
Summary of strengths: 1. This study evaluated various models and prompt variants.
2. Prompt engineering was conducted using the TELeR taxonomy.
3. The paper provides a detailed analysis of the performance of different models and prompt styles.
Summary of weaknesses: 1. Although the paper highlights the value of SEM-F1, the authors do not sufficiently justify why it is superior to ROUGE and BERTscore, nor do they discuss the potential weaknesses of SEM-F1.
2. Despite discussing a large amount of data and experimental evaluations, the paper prioritizes breadth over depth.
3. The human evaluation section lacks depth. The criteria for selecting the "best-evaluated prompts" are not detailed enough, and the negotiation process for the annotators' final scores is unclear.
Comments, suggestions and typos: The authors should provide a more in-depth analysis of why certain models and prompts perform better or worse, including a detailed analysis of failure cases.
Overall assessment scores: 2.5