Battle of the Wordsmiths: Comparing ChatGPT, GPT-4, Claude, and Bard

Authors: ali borji, Mehrdad Mohammadian
Primary area: datasets and benchmarks
Keywords: Large language models, ChatGPT, GPT-4, Claude, Bard, datasets, natural language processing, language modeling

Abstract:
Although informal evaluations of modern LLMs
can be found on social media, blogs, and news
outlets, a formal and comprehensive comparison among them has yet to be conducted. In
response to this gap, we have undertaken an extensive benchmark evaluation of LLMs and conversational bots. Our evaluation involved the collection of 1002 questions encompassing 27 categories,
which we refer to as the “Wordsmiths dataset.”
These categories include reasoning, logic, facts,
coding, bias, language, humor, and more. Each
question in the dataset is accompanied by an accurate and verified answer. We meticulously assessed four leading chatbots: ChatGPT, GPT-4, Bard, and Claude, using this dataset. The results
of our evaluation revealed the following key findings: a) GPT-4 emerged as the top-performing
chatbot across almost all categories, achieving a
success rate of 84.1%. On the other hand, Bard
faced challenges and achieved a success rate of
62.4%. b) Among the four models evaluated,
one of them responded correctly approximately
93% of the time. However, all models were correct only about 44%. c) Bard is less correlated
with other models while ChatGPT and GPT-4 are
highly correlated in terms of their responses. d)
Chatbots demonstrated proficiency in language
understanding, facts, and self-awareness. However, they encountered difficulties in areas such
as math, coding, IQ, and reasoning. e) In terms of
bias, discrimination, and ethics categories, models
generally performed well, suggesting they are relatively safe to utilize. To make future model evaluations on our dataset easier, we also provide a multiple-choice version of it (called WordsmithsMCQ).

Decision: reject
Reviewer scores: 5, 3, 3, 6, 3

Reviews and discussion:
<thread>
creation date:
1729951084127

thread_type:
official_review

<thread_object>
creator: reviewer_8kbp

<content>
summary:
This paper is about a thorough evaluation of LLMs, especially the four closed-source models named in the title. Five annotators (without further specification) defined tasks, ranging from math to reasoning and also subjective tasks such as humor, and rated the outputs of the LLMs, without performing an agreement analysis however. The findings then indicate that GPT4 is the best model and the others are substantially worse.

strengths:
The paper has several strengths, including:

1) apparently a lot of human effort to evaluate the LLMs

2) LLM evaluation is an important and very timely topic

3) Several interesting analyses including diversity of model outputs and accuracy across different tasks

weaknesses:
The paper has, however, also several limitations:

W1) We do not know much about the five annotators, their agreements and their expertise for the tasks. The paper says they are well-educated and their age ranges and nationality, but other key information is missing.

W2) I am not sure why only 4 closed-source models are included. It would be very interesting to include open-source models such as LLama, because the scientific community has no control over commerical closed-source models, which can in addition change at any time.

W3) The model versions are unclear. What is ChatGPT - is that GPT3.5? What is Claude? Claude 3 supposedly is a very strong model, but here it performs a lot below GPT4; why? 

W4) In a sense, while this analysis has some generalization value, it's specific to those 4 models which will all be outdated in 2025. Section 4.6 has some ideas on how to generalize the evaluation and dataset to other LLMs in the future (i.e., by automated analysis), but the main part of the paper is focussed just on those 4. What about automatic evaluation metrics for future LLMs?

W5) In addition, the results aren't so insightful / interesting as well: GPT4 seems quite adept so maybe the benchmark is not difficult enough?

W6) The paper does not explore the effect of prompting on the results. Prompting may have big effects [1,2] and potentially the wrong prompts were used for the weaker models?

W7) What about existing LLM leaderboards and their performance evaluation? (ChatBot Area, etc.)

W8) What about task/data contamination? Are your tasks strongly similar to others and some of the LLMs already have "seen" the answers?

W9) I miss more examples, e.g., in the appendix, about questions, LLM answers, and human ratings.

W10) The paper claims it's heavily inspired by two 2023 papers, but a lot has happened since then in the LLM world and its evaluation. I note that section 2.2 very surprisingly contains no single work from 2024 (the same is true for 2.1). This gives me the impression the authors are not fully aware of the most recent trends in LLM evaluation. 

[1] https://arxiv.org/abs/2401.00595

[2] https://arxiv.org/abs/2406.18528

questions:
Q1) Suprisingly, Bard is the best model in Table 3, while GPT4 had dominated throughout. Why?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730540626687

thread_type:
official_review

<thread_object>
creator: reviewer_f4rb

<content>
summary:
This paper aims to conduct a formal and comprehensive comparison among ChatGPT, GPT-4, Claude, Bard. Their evaluation involve the collection of 1002 questions encompassing 27 categories, which they refer to as the “Wordsmiths dataset. The results of the evaluation revealed the following key findings: a) GPT-4 emerged as the top-performing chatbot across almost all categories, achieving a success rate of 84.1%. On the other hand, Bard faced challenges and achieved a success rate of 62.4%. b) Among the four models evaluated, one of them responded correctly approximately 93% of the time. However, all models were correct only about 44%. c) Bard is less correlated with other models while ChatGPT and GPT-4 are highly correlated in terms of their responses. d) Chatbots demonstrated proficiency in language understanding, facts, and self-awareness. However, they encountered difficulties in areas such as math, coding, IQ, and reasoning. e) In terms of bias, discrimination, and ethics categories, models generally performed well, suggesting they are relatively safe to utilize.

strengths:
(1)This work conducts a formal and comprehensive comparison among ChatGPT, GPT-4, Claude, Bard.

(2)This paper gathers a dataset of more than 1K questions spanning various domains to this end.

(3)To support future research endeavors, the authors have made their collected comparison corpus, evaluations, and analysis code openly accessible.

weaknesses:
(1) I do not think that comparing the four large models will have a significant impact on the field.

(2) Some findings have already been drawn (e.g., GPT-4 performs best among the four LLMs).

questions:
see Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730768344501

thread_type:
official_review

<thread_object>
creator: reviewer_eje5

<content>
summary:
The authors create an evaluation benchmark called Wordsmiths dataset that includes 1002 questions across 27 categories, including reasoning, logic, facts, coding, bias, etc. This benchmark addresses the shortcoming of previous evaluation benchmarks in that it is comprehensive, has answers meticulously verified by the authors, and manually assessed as to go beyond multiple choice-based performance.

strengths:
- The authors present a thorough investigation of performance differences between ChatGPT, GPT-4, Claude, and Bard with 1002 questions and answers that were thoroughly selected based on quality and agreement among humans. They manually assess the responses for a detailed qualitative analysis of model behavior.   
- The authors also create a multiple-choice question version of the dataset to facilitate future work to use Wordsmiths for assessing models.

weaknesses:
- The authors find contradicting results between Wordsmiths and Wordsmiths MCQ, where the former shows that ChatGPT and GPT-4 performs the best and Bard the worst, while the latter shows that Bard is the best while ChatGPT is the worst. The implications of these results for the value of Wordsmiths and Wordsmiths MCQ is not discussed and the reader is left confused in what to make use of these contradicting results. 
- While the analysis that the authors have done seem valuable, it is not replicable or scalable given the manual efforts required and the discrepancy in results seen in the MCQ version of the dataset. If there is an MCQ version that can still capture the same trends found in Table 2, this weakness can be alleviated, though the amount of insights may not be as rich as those described throughout the paper. 
- The Wordsmiths dataset may have comprehensive coverage of various categories for evaluating LLMs, but it is unclear whether the number of samples per subcategory is sufficient for getting statistically meaningful results in performance differences of each subcategory between models. For example, there are only an average of about 18 questions per reasoning subcategory. The authors do not provide any p-values. 
- While the authors provide many findings, the analysis is only conducted for closed-source models and therefore does not provide actionable insights. A more interesting analysis might have been one that compares open-source models and at different scales.

questions:
- Lines 57-58: Is this manual examination replicable and scalable for other researchers? Lines 75-76 seems to indicate that this is difficult, and thus opts back to multiple-choice questions. I would suggest providing some examples where the manual examination revealed insights that were not discoverable through a MCQ setup in the introduction to persuade the reader that this is a meaningful contribution. Otherwise, it feels like something that is one-off and cannot be done as it's too much manual effort.
- line 158: what makes someone an expert rater as opposed to a novice rater? What makes the authors more reliable than crowd-sourced annotators? Elaborate more on lines 224-226.
- line 182: what makes something an informal test set as opposed to a formal one?
- 183: what was the motivation for the questions that were formulated by the authors? How are they different from the ones already included in existing benchmarks?
- While this level of meticulous effort is noteworthy, I wonder whether the number of samples per subcategory is sufficient to show statistical significance in performance differences among compared models. For instance, 127/7 is only about 18 questions on average per reasoning subcategory (line 197).
- Reference for lines 263-265?
- Figure 2: Why is the accuracy chart inside the distribution chart? It would be less confusing if it was simply put side by side. Also, the accuracy chart should not be a line chart unless there's some meaningful order for the horizontal axis. I suggest that it be a bar chart as well to prevent confusion.
- Table 2 is missing statistical significance.
- Figure 3: If the inset figure doesn't really tell a different story, it shouldn't be included. Otherwise, it would help to have the caption explain the significance of the comparison between the two.
- Line 350: The supplementary material is very long. Be more explicit in where the readers should be looking at.
- Significance of section 4.4.4? What was the hypothesis or goal behind this experiment? What are the implications?
- Section 4.6 is quite interesting but not delved into deep enough. Why does Bard get the highest score in this setup despite it being the worst and GPT-4 being the best in the previous setup? (Figure 2). If the MCQ results are different from the manual analysis of the 1002 questions, what is its value? The way that the MCQ version was created makes it adversarial because it would include choices that look correct, and hence the performance difference, but the results are still confusing. What was the motivation behind this adversarial setup? What were the questions that were left out such that there's only 388 questions in this MCQ setup?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730862427284

thread_type:
official_review

<thread_object>
creator: reviewer_6jcf

<content>
summary:
This paper presents an extensive benchmark study comparing four major language models (LLMs)—ChatGPT, GPT-4, Claude, and Bard—across a diverse dataset of 1,002 questions, covering 27 categories including reasoning, logic, humour, and coding. The authors create the "Wordsmiths dataset," aiming to evaluate the models on a comprehensive set of tasks that mimic diverse human capabilities. Key findings include that GPT-4 outperforms other models with an overall accuracy of 84.1%, while Bard trails at 62.4%. The study identifies ChatGPT and GPT-4 as highly correlated, with Bard displaying a lower correlation with the others, especially in categories like symbolic reasoning and translation.

The evaluation also includes qualitative assessments of biases, humour, and ethical reasoning, where the models generally performed well, indicating relative safety for use in sensitive contexts. Additionally, the authors introduce the "Wordsmiths-MCQ," a multiple-choice version of their dataset, to facilitate future model evaluations.

strengths:
- The paper introduces a comprehensive dataset of 1,002 questions across 27 diverse categories, covering not only technical domains like reasoning, logic, and coding but also nuanced areas such as bias, humour, and ethical reasoning. This broad scope enables a well-rounded evaluation of the language models' strengths and limitations.
- It provides several insightful post-hoc analyses, which add depth to the evaluation by examining correlations among models, response length distribution, and qualitative failures.

weaknesses:
- While this paper presents valuable work, its contributions would have been more impactful had it been published a year earlier. By now, many of its findings are widely recognized, and several of its claims are outdated. For example, the introduction states, "Although informal evaluations of modern LLMs can be found on social media, blogs, and news outlets, a formal and comprehensive comparison among them has yet to be conducted." This statement no longer holds in 2024, as numerous formal studies comparing major LLMs have since been published. A brief search suggests this paper may have been initially shared online over a year ago. In future, the authors might consider submitting this paper to journals instead of conferences like ICLR.
- The paper lacks important experimental details, particularly in its evaluation methodology. It describes a scoring system where "a score of 2 is given when the chatbot provides the correct answer, a score of 1 is assigned for a partially correct response, and a score of 0 indicates a completely incorrect answer," and notes that "in cases where a partial answer is given, human judgment becomes necessary." However, it is unclear if this scoring scale is scientifically validated or if it accurately reflects nuanced performance differences. Moreover, there is insufficient explanation of how human judgments were standardized, raising questions about consistency and potential bias in scoring. These omissions make it difficult to fully assess the reliability and rigour of the evaluation process.

questions:
See my comments above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731049751627

thread_type:
official_review

<thread_object>
creator: reviewer_cmqy

<content>
summary:
This paper benchmarks four large language models (LLMs) – ChatGPT, GPT-4, Claude, and Bard – using the newly introduced "Wordsmiths" dataset, which includes 1002 questions spanning 27 categories such as reasoning, logic, coding, bias, and language understanding. The evaluation was conducted through a thorough manual assessment by expert raters, providing a comparison of model performance across categories and an analysis of inter-model correlations. Additionally, the authors propose a multiple-choice version of the dataset (Wordsmiths-MCQ) for automated evaluations and explore the potential for ensemble models to outperform individual models.

strengths:
1. The paper’s benchmark spans a wide range of categories, offering a broad assessment of LLM capabilities. This is a significant contribution compared to many benchmarks that focus on narrower domains.

2.  The manual evaluation process, conducted by knowledgeable raters, adds depth and provides a more nuanced understanding of model performance than purely automated evaluations.

3. The introduction of the Wordsmiths dataset and its MCQ version could serve as useful resources for the research community.

4. The analysis of model correlations and the exploration of ensemble approaches highlight interesting avenues for further research.

weaknesses:
1. The core methodology, which involves querying models and manually evaluating their responses, lacks significant novelty. While thorough, it does not advance the state-of-the-art in evaluation techniques.

2. The construction of the Wordsmiths dataset, which relies on publicly available content, social media, and questions generated by researchers, raises concerns about possible biases and the representativeness of the dataset. More detail on the selection and filtering processes is needed.

3. Although manual evaluation allows for nuance, it inherently introduces subjectivity, especially in categories like humor and bias. The paper references discussions and majority votes among raters but does not provide sufficient detail on inter-rater agreement, which is crucial for assessing reliability.

4. The analysis mainly reports accuracy across categories without deeply exploring the types of errors models make or why certain failures occur. A more detailed breakdown of error types and their implications would strengthen the paper.

5. The rationale for the Wordsmiths-MCQ dataset, which uses incorrect model answers as distractors, is not well-developed. It is unclear why this approach would be preferable to creating tailored multiple-choice questions.

6. The study’s findings may not generalize well due to the specific versions of the models tested and the relatively small dataset size. Differences in model versions can significantly impact performance.

questions:
1. What was the systematic approach, if any, used to determine the 27 categories?

2. How did the authors ensure consistency across human raters, and what were the inter-rater agreement scores?

3. Can the authors provide more detailed annotation guidelines and scoring criteria for each category?

4. How do the findings generalize, considering the dataset's reliance on easily accessible content and researcher-generated questions?

5. What criteria were used to define “rare” questions, and why was this aspect emphasized?

6. How did the authors handle edge cases in scoring, especially in subjective categories like humor and bias?

7. What are the potential biases introduced by using model-generated incorrect answers as distractors in the MCQ dataset, and why was this chosen over conventional MCQ creation methods?

8. Can the authors provide more in-depth error analysis to better understand model limitations?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a dataset, wordsmiths, that includes >1K questions from 27 categories, that is used to evaluate LLMs
 (GPT, chatGPT, Claude and BARD). The paper includes an analysis of the outcomes from the evaluation of these models. While the idea is interesting and useful, reviewers raised several weaknesses, such as the limited novelty, limited coverage of models (only 4 models), issues related to data contamination, relationship or comparison to other benchmarks and so on. These remain unanswered after the rebuttal period.