LLMs’ Classification Performance is Overclaimed
Authors: Hanzi Xu, Renze Lou, Jiangshu Du, Vahid Mahzoon, Elmira Talebianaraki, Zhuoan Zhou, Elizabeth Garrison, Slobodan Vucetic, Wenpeng Yin


Reviewer paper summaries: The paper evaluates the ability of large language models to answer a question when the right answer is not provided by the user.
Summary of strengths: - This paper provides a good test benchmark and metric for verifying how good LLMs can be at picking an answer when no right answer is provided.
- They also draw attention to this problem in the first place and point out that humans are much better at this task.
Summary of weaknesses: ### Smaller methodological issues:
**124 to 134**: it would be better to discuss the methodologies used in the cited studies. Rather than just reporting the results, critically analyze the findings. Discuss any limitations or potential biases in the studies mentioned.

### Bigger issues:
- WARNING: releasing their dataset to the public will cause future large language models to perform better on this task. [Because they will be scraped by future models during training.]
Comments, suggestions and typos: ### Writing issues:
**- from 001 to 0016**: This should serve as an explanation to clarify why we are addressing the issue. This paragraph can start like "While large language models (LLMs) often demonstrate strong performance in classification tasks where gold labels are included by default, typically posed as ‘which of the following is correct?’, they also reveal limitations when these gold labels are excluded, still choosing from incorrect options." To address this...

**031**: Note that "advocate" is not common in this context; instead, use "propose"

**- from 017 to 035**: The abstract is usually formatted as a single paragraph. You can continue like "...we argue that the perceived performance of LLMs is overstated due to... We make a threefold contribution..."

**- 035**: We need to include sentences in the abstract that discusses some findings. e.g., start with: "We find that...". 

**- from 036 to 039**: Such a starting! You should begin with the most important sentence related to your work and frame it appropriately. This is not a practical teaching approach that starts with "Let's begin!" Instead, start with a compelling statement that directly addresses the core of your research. 

**- from 040- 047**: In the first paragraph of the introduction, you should focus on the purpose of your study, providing evidence and references to prove that you are working on a significant and worthwhile topic. For instance, revise the first paragraph of introduction to highlight the goals of your study, rather than starting with an example. 

**-from 048 to 073**: Please merge the second paragraph with the first to clearly outline the purpose and main goals of your study. For instance, you started the second paragraph with "Why?" This is expected to be stated implicitly, e.g., why these goals are important, providing references to support your claims. i.e., answer this question: why the topic of LLM generation for classification problems is significant in the current research landscape?

**- from 086 to 097**: Please merge the 3rd paragraph with the 4th. (i.e., write 074 to 097 in a single paragraph)

**- from 098 to 116**: Please write in one paragraph and only number the bullet points like "In summary, our contributions are: (1) uncover the limitations of LLMs..., (2) introduce a benchmark..., and (3) propose a new...". Please note that "advocate" is not common in this context; instead, use "propose"

**118**: Please number the bold part in this case as **2.1.LLMs’ Performance on Classification Tasks** 

**119 to 124**: Too short! Make the sentences flow logically. e.g., the transition from discussing standalone classification tasks to multi-task learning could be smoother.

**124 to 134**: Consider restructuring the sentence, make it clearer. it would be better to first discuss Sun et al. (2023a) and then mention Sun et al. (2023b) to maintain a logical flow and clarity for example: "Sun et al. (2023a) demonstrate remarkable performance metrics from GPT, achieving 95-98% accuracy in sentiment analysis... In their subsequent study, Sun et al. (2023b) report an average accuracy above 90% on five well-known NLP benchmark...". Additionally, while mentioning performance metrics is important, provide more detail on how these metrics were achieved. For instance, explain what **"zero-shot prompting"** and **"few-shot demonstrations"** mean in this context. 

**134 to 137**: In this section, it's important to discuss any conflicting results or differing viewpoints found in the literature and emphasize any existing gaps in the current research. Clearly explain how the reviewed literature contributes to the field. For example, discuss the significance of the findings of these studies in advancing LLMs to solve NLP classification tasks and then lead to the conclusion that "while the latest LLMs are considered reliable solutions for NLP classification, their true understanding of the essence of the classification task has not been properly evaluated". 

**138**: Again, please number the bold part in this case as **2.2.LLMs’ Challenges in Task Comprehension**

**139 to 183**: You wrote this part far better than the previous paragraph. However, you also need to follow the same guidelines I mentioned for the previous part regarding discussing the significance of the findings of these studies and any existing gaps, then lead to a conclusion that highlights our contributions and makes the logical flow smoother. You should end this part by mentioning what we are focusing on in this work. Here are some steps to improve this part:
		1- Start with a clear statements outlining the focus of this section. 
		2- Expand on why understanding the rationale behind model predictions is essential.
		3- It would be better to specify some of the studies mentioned in line 148 and briefly summarize their methodologies and critically analyze the findings rather than just reporting them. You already mention them sparsely, but it would be better to make them coherent and elaborate on them more.
		4- Avoid repeating the same information in different ways.

**199 and 208**: remove the citation from "BANK-77 (Casanueva et al., 2020)" since it has already been cited once, and it is not necessary to cite it again, especially in the bold part. and do the same for line 208.

**572 to 575**: Highlight the contributions, i.e., emphasize how the KNOW-NO benchmark and OMNIACCURACY metrics contribute to the field. For example, answer this question: to what extent is the evaluation by OMNIACCURACY metrics a comprehensive reflection of LLMs’ classification performance?
Overall assessment scores: 4