LLMs as Meta-Reviewers' Assistants
Authors: Shubhra Kanti Karmaker Santu, Sanjeev Kumar Sinha, Naman Bansal, R. Alexander Knipper, Souvika Sarkar, John Salvador, Yash mahajan, Sri Ram Pavan Kumar Guttikonda, Mousumi Akter, Matthew Freestone, Matthew C. Williams Jr.


Reviewer paper summaries: The study investigates a new potential application area of large language models (LLMs) to assist academic meta-reviewers. The study includes a detailed qualitative analysis of the LLM performance in this task.
Summary of strengths: 1. Studied a new application area for LLMs.
2. Demonstrated a detailed qualitative analysis on the performance of LLMs in this task.
Summary of weaknesses: 1. Lack of details behind the dataset construction provided. How are they sampled? Are those papers related to NLP or any application areas of Machine Learning or Deep Learning? What are the paper topic? More details should be provided.
2. Since no details behind the data sampling were provided, it creates concerns on the validity of utilizing NLP experts to evaluate the meta-review of other Machine Learnibg domains.
3. LLaMA-2 has various versions: 7B, 13B, 70B, which one was used?
4. Since only 40 papers from ICLR were used, the paper did not mention is the finding generalizable to other domains?
5. The Figure 1 and 2 should be explained better.
6. The authors observed a surprising finding but did not conduct any analysis. Is this because of using only 40 reviews?
Comments, suggestions and typos: See above.
Overall assessment scores: 2