LLMs as NLP Researchers: Paper (Meta-)Reviewing as a Testbed
Authors: Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi LIU, Renze Lou, Henry Peng Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, Haoran Ranran Zhang, Vipul Gupta, Yinghui Li, Tao Li, Fei Wang, Qin Liu, Tianlin Liu, Pengzhi Gao, Congying Xia, Chen Xing, Cheng Jiayang, Zhaowei Wang, Ying Su, Raj Sanjay Shah, Ruohao Guo, Jing Gu, Haoran Li, Kangda Wei, Zihao Wang, Lu Cheng, Surangika Ranathunga, Meng Fang, Jie Fu, Fei Liu, Ruihong Huang, Eduardo Blanco, Yixin Cao, Rui Zhang, Philip S. Yu, Wenpeng Yin


Reviewer paper summaries: This paper examines the performance of current SOTA LLMs as paper reviewers and meta-reviewers, comparing their abilities to those of human reviewers. Additionally, the authors contribute an annotated dataset, ReviewCritique, which can serve as a valuable resource for future researchers working in this area.
Summary of strengths: 1. This paper is very well-written and easy to follow, I enjoy reading it.

2. The paper's motivation is compelling, as it aims to address the pressing need for assisting researchers in today's fast-paced academic environment. By having a LLM as a reviewer, it could not only benefit the reviewers by reducing their workload, but also the authors themselves by providing them with valuable feedback and suggestions to improve their research work.

3. The paper demonstrates rigorous execution, from the carefully designed data annotation process to the thorough analysis comparing LLM-generated reviews with human reviews. The experiments are convincing and provide valuable insights into the limitations of current LLMs in writing reviews and identifying deficient reviews.

4. A great deal of effort has gone into collecting the ReviewCritique dataset and ensuring its quality. The dataset will certainly contribute to the research community. The authors' conclusions serve as a timely reminder that researchers should not blindely rely on LLMs for writing reviews, despite the increasing burden of reviewing responsibilities.
Summary of weaknesses: One potential weakness of the paper is the reliance on ROUGE and BERTScore as the sole metrics for evaluating the quality of the generated explanations (lines 617-636). While these metrics can measure similarity to human-written ones, they may not fully capture the nuances of why certain explanations are effective or ineffective. To gain a deeper understanding of the quality of the generated explanations, it would be beneficial to conduct a more qualitative analysis of the explanations themselves, examining their content in greater detail. This could provide valuable insights into why the model falls short in identifying deficiencies in reviews, and potentially inform future improvements to the model's performance.
Comments, suggestions and typos: See strengths and weaknesses.
Overall assessment scores: 4