SlowGuess/NeurIPS_2023_OCR huggingface dataset:
When conducting pairwise comparison evaluations, it is crucial to consider the issue of consistency. Specifically, the result should not change if we reverse "response 1" and "response 2".
In our test, the selected models: GPT-4 [19], Bard [52], Claude [51], and ChatGPT [2] all possess the capability to perform reliable and consistent comparison evaluations. In contrast, Vicuna-13B [38] cannot produce consistent comparison results. Hence, we employ Bard, Claude, and ChatGPT as examiners in peer-examination.
To mitigate the influence of the input sequence on the comparison evaluation process, for each comparison, we take the average of the results from the original sequence and the reversed sequence.
During the peer-review process, we need to be mindful of ChatGPT's tendency to forget (such forgetfulness is not observed for the other examiners), especially when the generated answers are too long. To tackle this, we set a maximum length threshold for responses. This limit helps make sure the comparison result from ChatGPT stays consistent. However, cutting off the answer at a certain point could remove important details or context. This could affect how accurate and high-quality the answer is. Hence we need to find a balance between limiting ChatGPT's forgetfulness and keeping the evaluation fair for every examinee model. In our experiment, to make sure the results we get from ChatGPT are reliable, we choose to slightly sacrifice fairness by setting a maximum length of 400.
More explanations to the scoring results in Table 4. When Claude [51] acts as the examiner, the low percentage of full-mark answers can be attributed to its strict evaluation criteria, particularly regarding the coherence dimension. This is probably because when Claude tends to provide answers containing links to related documents while exhibiting clear structures. To address the imbalance caused by Claude's strict criterion we introduce a weighted average score in addition to the arithmetic mean value. By normalizing the scores, we aim to create a more balanced comparison among the examiner models.
D Case Study
D.1 Single-Round QA
D.1.1 Knowledge Memorization Question Case
Field: Beauty & Fitness  $\rightarrow$  Cosmetic Procedures  $\rightarrow$  Cosmetic Surgery
Question: What are the most common types of cosmetic surgery procedures?
Ground Truth: The most common types of cosmetic surgery procedures are breast augmentation, liposuction, rhinoplasty, eyelid surgery, and facelift.
Answer-ChatGPT: The most common types of cosmetic surgery procedures include:1. Breast augmentation
2. Liposuction
3. Rhinoplasty (nose job)
4. Eyelid surgery
5. Facelift
6. Tummy tuck
7. Breast lift
8. Brazilian butt lift
9. Mommy makeover (combination of procedures to restore pre-pregnancy body)
10. Botox injections.
Score: accuracy: 3 coherence: 3 factuality: 3 comprehensive: 3 overall: 5
Reason: Accuracy: 3 (The answer correctly lists common types of cosmetic surgery procedures)
Coherence: 3 (The answer is well-structured, easy to understand, and logically organized)
Factuality: 3 (The answer contains no factual errors)
Comprehensiveness: 3 (The answer provides a comprehensive list covering various types of cosmetic surgery procedures) Overall Score: 5 (The answer meets all the required criteria and provides a complete and clear response to the question)
Answer-Flan-UL2: Breast augmentation, eyelid surgery, and rhinoplasty are the most common cosmetic surgery procedures.
Reason: Accuracy: 3 - The answer correctly lists some of the most common cosmetic surgery procedures.
Coherence: 3 - The answer is well-structured, logical, and can be understood by nonprofessionals.
Factuality: 3 - The answer contains no factual errors.
Comprehensiveness: 2 - The answer mentions some common procedures but could have included a few more to be more comprehensive, such as liposuction or tummy tuck.
Overall Score: 4 - The answer is accurate, coherent, and factual but could have been slightly more comprehensive.
Answer-BLOOMZ: The most common types of cosmetic surgery procedures include Botox injections, liposuction, and plastic surgery.
Score: accuracy: 2 coherence: 3 factuality: 2 comprehensive: 2 overall: 3
Reason: Accuracy: 2 - The answer lists some common cosmetic procedures, but Botox injections are not a surgery and plastic surgery is a general term, not a specific procedure.
Factuality: 2 - The answer contains a factual error by including Botox injections as a surgery and using the general term "plastic surgery."
Comprehensiveness: 2 - The answer mentions some common procedures but could have included more specific and accurate examples, such as breast augmentation, eyelid surgery, or rhinoplasty.
Overall Score: 3 - The answer is coherent but has some inaccuracies and could be more comprehensive.
Rank:ChatGPT >Flan-UL2 >BLOOMZ
D.1.2 Knowledge Comprehension Question Case
Field: News  $\rightarrow$  Health News  $\rightarrow$  Health Policy
Question: How do public health emergencies such as pandemics influence changes in health policies?
Ground Truth: Public health emergencies such as pandemics influence changes in health policies by highlighting gaps in existing systems, prompting the development of new strategies, and accelerating the implementation of necessary measures to prevent, control, and mitigate the impact of the emergency.
Answer-ChatGPT: Public health emergencies such as pandemics can have a significant impact on health policies. Here are some ways in which pandemics can influence changes in health policies:
1. Increased funding for public health: Pandemics can highlight the importance of investing in public health infrastructure and preparedness. Governments may increase funding for public health agencies and research to better prepare for future outbreaks.
2. Changes in healthcare delivery: Pandemics can lead to changes in healthcare delivery, such as increased use of telemedicine and remote consultations. This can help to reduce the spread of infectious diseases and improve access to healthcare for vulnerable populations.
3. Development of new policies: Pandemics can lead to the development of new policies to address specific issues related to the outbreak. For example, policies may be developed to ensure that healthcare workers have access to personal protective equipment (PPE) or to mandate social distancing measures.
4. Increased focus on prevention: Pandemics can highlight the importance of prevention in healthcare. Governments may invest in public health campaigns to promote healthy behaviors and encourage vaccination.
5. International cooperation: Pandemics can lead to increased international cooperation and collaboration on health policies.
Countries may work together to share information and resources to better respond to outbreaks and prevent future pandemics.
Overall, pandemics can have a significant impact on health policies, leading to changes in healthcare delivery, increased funding for public health, and the development of new policies to address specific issues related to the outbreak
Reason: Accuracy: 3 The answer correctly addresses how pandemics influence changes in health policies.
Coherence: 3 The answer is well-structured, logical, and easily understandable by nonprofessionals.
Factuality: 3 The answer does not contain any factual errors.
Comprehensive: 3 The answer covers multiple aspects of the question and provides a comprehensive response.
Overall Score: 5 The answer is accurate, coherent, factual, and comprehensive, making it deserving of the highest overall score.
Answer-Vicuna-13B: Public health emergencies such as pandemics can have a significant impact on health policies. They can lead to changes in policies related to disease surveillance, detection, and response, as well as changes in policies related to vaccination, public health preparedness, and healthcare delivery. Governments may also take steps to increase funding for public health programs and to coordinate with international organizations to address the crisis. Additionally, during public health emergencies, there may be changes to laws and regulations related to public health, such as quarantine and isolation measures, to help prevent the spread of disease.
**Reason:** Accuracy: 3 The answer correctly addresses the question by mentioning various ways in which health policies are influenced by public health emergencies such as pandemics.
Coherence: 3 The answer is well-structured and logically organized, explaining different aspects of health policies that could be affected. The answer is understandable by non-professionals.
Factuality: 3 There are no factual errors in the answer. The various policy changes mentioned are all reasonable responses to public health emergencies.
Comprehensive: 3 The answer covers multiple aspects of health policy changes – surveillance, detection, response, vaccination, preparedness, healthcare delivery, funding, coordination with international organizations, and changes to laws and regulations.
Overall Score: 5 The answer is accurate, coherent, fact-based, and comprehensive, providing an excellent overview of how public health emergencies can impact health policies.
Answer-LLaMA-65B: Public health emergencies such as pandemics influence changes in health policies by highlighting the need for improved healthcare infrastructure, better disease surveillance, and more effective vaccines.
Score: accuracy: 3 coherence: 3 factuality: 3 comprehensive: 2 overall: 4
Reason: Accuracy: 3 The answer correctly addresses how public health emergencies influence changes in health policies.
Factuality: 3 The answer contains no factual errors.
Comprehensive: 2 The answer covers some aspects of the question but could provide more details on how health policies change in response to public health emergencies.
Overall Score: 4 The answer is accurate, coherent, and factual, but could be more comprehensive in addressing the question.
Answer-LLaMA-13B: Public health emergencies such as pandemics influence changes in health policies by increasing the demand for healthcare services and resources.
Score: accuracy: 2 coherence: 3 factuality: 3 comprehensive: 2 overall: 3
Reason: Accuracy: 2 The answer is partially correct, as it mentions the increased demand for healthcare services and resources, but it doesn't cover other ways in which health policies may change during a pandemic, such as implementing new prevention measures or modifying existing ones.
Coherence: 3 The answer is well-structured and logical. It is easily understandable by nonprofessionals.
Factuality: 3 The answer is factually correct in stating that public health emergencies increase the demand for healthcare services and resources.
Comprehensiveness: 2 The answer addresses one aspect of the question but does not provide a comprehensive response by including other potential changes in health policies during a pandemic.
Overall Score: 3 The answer is partially correct and understandable, but it could be more comprehensive in addressing other changes in health policies.
Rank:ChatGPT  $>$  Vicuna-13B  $>$  LLaMA-65B  $>$  LLaMA-13B
D.1.3 Knowledge Analysis Question Case
Field: Internet & Telecom  $\rightarrow$  Service Providers  $\rightarrow$  Cable & Satellite Providers
Question: What are the advantages and disadvantages of bundling services like internet, television, and phone from a single provider in the context of pricing and service quality?
Ground Truth: Advantages: 1. Cost savings: Bundling services often leads to discounts and lower overall costs.
2. Convenience: Dealing with a single provider simplifies billing and customer service.
Disadvantages: 1. Limited options: Bundling may restrict choices for individual services, leading to less customization.
2. Service quality: A single provider may not excel in all services, potentially compromising quality in some areas.
Answer-ChatGPT: Advantages of bundling services:
1. Cost savings: Bundling services can often result in cost savings for customers as providers offer discounts for multiple services.
2. Convenience: Having all services from a single provider can be convenient as customers only need to deal with one company for all their needs.
3. Simplified billing: Bundling services can simplify billing as customers receive a single bill for all services.
4. Additional features: Providers may offer additional features or perks for customers who bundle services, such as free premium channels or faster internet speeds.
Disadvantages of bundling services:
1. Limited options: Bundling services may limit customers' options as they may be forced to choose from a limited selection of providers.
2. Service quality: Bundling services may result in lower service quality as providers may prioritize certain services over others.
3. Contract terms: Bundling services often requires customers to sign a contract, which may have strict terms and conditions.
4. Difficulty switching providers: Bundling services may make it difficult for customers to switch providers as they would need to cancel all services and find new providers for each service.
Reason: Accuracy: 3 The answer correctly identifies the advantages and disadvantages of bundling services in the context of pricing and service quality.
Coherence: 3 The answer is well-structured, logically organized, and easily understandable by non-professionals.
Factuality: 3 The answer does not contain any factual errors and provides accurate information about the advantages and disadvantages of bundling services.
Comprehensiveness: 3 The answer covers multiple aspects of the question and provides a comprehensive response, discussing both advantages and disadvantages of bundling services.
Overall Score: 5 The answer is accurate, coherent, factual, and comprehensive, providing a thorough and well-rounded response to the question.
Answer-GLM-130B: Bundling services like internet, television, and phone from a single provider in the context of pricing and service quality has advantages and disadvantages. Bundling services like internet, television, and phone from a single provider in the context of pricing and service quality has advantages because it allows consumers to save money on services, and it has disadvantages because it can lead to higher prices and lower service quality.
Comprehensiveness: 2 The answer provides a general overview of the advantages and disadvantages but could have elaborated more on the reasons behind the cost savings and potential issues with service quality.
Overall Score: 4 The answer is accurate, coherent, and factual, but could have been more comprehensive in explaining the reasons behind the advantages and disadvantages.
Answer-Flan-T5: Bundling services like internet, television, and phone from a single provider can result in lower prices and better service quality.
Score: ccuracy: 2 coherence: 3 factuality: 3 comprehensive: 2 overall: 3
Reason: Accuracy: 2 The answer mentions lower prices and better service quality as advantages, but it does not mention any disadvantages.
Comprehensive: 2 The answer covers the advantages of bundling services but does not address the disadvantages.
Overall Score: 3 The answer provides a partial response to the question, mentioning the advantages but not the disadvantages of bundling services.
Answer-BLOOMZ: The advantages of bundling services like internet, television, and phone from a single provider include lower prices and better service quality, while the disadvantages include higher prices and less choice.
Reason: Accuracy: 2 The answer mentions both advantages and disadvantages, but it contradicts itself by stating both lower and higher prices as advantages and disadvantages.
Coherence: 3 The answer is well-structured and easy to understand for non-professionals.
Factuality: 2 The answer contains a factual error by stating both lower and higher prices as advantages and disadvantages. Bundling services usually result in lower prices, not higher.
Comprehensiveness: 2 The answer covers some aspects of the question but could provide more details on the advantages and disadvantages of bundling services.
Overall Score: 3 The answer is somewhat accurate and coherent but could be improved in terms of factuality and comprehensiveness.
Rank:ChatGPT  $>$  GLM-130B  $>$  BLOOMZ  $>$  Flan-T5
D.2 Multi-turn Question Evaluation
Field: Hobbies & Leisure  $\rightarrow$  Cycling  $\rightarrow$  Bicycles & Accessories
Question: Which material is most commonly used for road bike frames?
Answer-Flan-T5: Aluminium is most commonly used for road bike frames.
Following-Question: What are the advantages and disadvantages of using aluminium for road bike frames compared to other materials like carbon fiber and steel?
Answer-Flan-T5: Aluminium is a light, strong, and durable material that is used for road bike frames. Aluminium is also a cheap material, making it a popular choice for budget road bikes. Aluminium is also a good choice for road bikes because it is easy to weld and can be made into a variety of shapes.
Score: accuracy: 2 coherence: 3 factuality: 3 comprehensive: 1 overall: 2
Reason: Accuracy: 2 The answer mentions some advantages of using aluminium but does not discuss the disadvantages or compare it to carbon fiber and steel.
Coherence: 3
The answer is well-structured, clear, and easily understandable.
Factuality: 3
The facts presented about aluminium are correct.
Comprehensive: 1
The answer does not cover multiple aspects of the question; it only focuses on the advantages of aluminium and does not discuss disadvantages or comparisons with carbon fiber and steel.
Overall score: 2
The answer provides some accurate information about aluminium but fails to fully address the question regarding disadvantages and comparisons with other materials.
Field: Sports
Question: In basketball, what defensive strategies are commonly employed to disrupt an opposing team's offensive flow?
Answer-GLM-130B: In basketball, defensive strategies commonly employed to disrupt an opposing team's offensive flow include trapping, double-teaming, and switching.
Following Question: Can you describe the differences between trapping, double-teaming, and switching in basketball defense?
Answer-GLM-130B: Trapping is a defensive strategy in which a defender attempts to block the path of a ball-handler by moving in front of him or her. Double-teaming is a defensive strategy in which two defenders attempt to block the path of a ball-handler by moving in front of him or her. Switching is a defensive strategy in which a defender moves from one side of the court to the other to block the path of a ball-handler.
Reason: Accuracy: 2 The answer provides a basic understanding of the three defensive strategies but lacks some important details. For example, trapping usually involves two defenders closing in on the ball-handler, not just one.
Factuality: 2 The answer contains some factual errors, such as the description of trapping involving only one defender.
Comprehensive: 2 The answer provides a basic understanding of the three defensive strategies but lacks some important details, such as the purpose of each strategy and how they are executed in a game.
Overall Score: 3 The answer provides a basic understanding of the three defensive strategies but could be improved with more accurate and detailed information.
Field: Health  $\rightarrow$  Health News  $\rightarrow$  Health Policy
Question: Which organization is primarily responsible for global health policies and guidelines?
Answer-LLaMA-65B: The World Health Organization (WHO) is primarily responsible for global health policies and guidelines.
Following Question Can you describe the process by which the World Health Organization develops and implements these global health policies and guidelines?
Answer-LLaMA-65B: The World Health Organization develops and implements global health policies and guidelines through a process of consultation with experts, member states, and other stakeholders.
Reason: Accuracy: 3 Reason: The answer correctly describes the process of consultation with experts, member states, and stakeholders.
Coherence: 3 Reason: The answer is well-structured, logical, and easily understandable by non-professionals.
Factuality: 3 Reason: The answer contains no factual errors.
Comprehensiveness: 2 Reason: The answer provides a general overview of the process but could include more details about the specific steps involved in developing and implementing policies and guidelines.
Overall Score: 4 Reason: The answer is accurate, coherent, and factual, but could be more comprehensive in describing the process.
D.3 Peer-Examination Evaluation
Examiner: Bard
Field: Psychology
Question: How does attachment style influence romantic relationships?
Answer-Vicuna-13B: Attachment style, which refers to the way individuals form emotional bonds with others, can have a significant impact on romantic relationships. People with a
secure attachment style tend to have healthy, stable relationships, while those with insecure attachment styles may struggle with issues such as jealousy, possessiveness, and trust problems. For example, individuals with an anxious attachment style may become overly dependent on their partner and fear abandonment, while those with a avoidant attachment style may be less committed and emotionally distant. Understanding and addressing attachment style issues can help individuals improve the quality and stability of their romantic relationships.
Answer-ChatGPT: Attachment style can have a significant impact on romantic relationships. People with a secure attachment style tend to have healthy and satisfying relationships, while those with insecure attachment styles may struggle with intimacy and trust.
Those with an anxious attachment style may be overly dependent on their partner and fear abandonment, leading to clingy or jealous behavior. They may also have a tendency to overanalyze their partner's actions and words, leading to misunderstandings and conflict.
Those with an avoidant attachment style may have difficulty with emotional intimacy and may prioritize independence over closeness. They may also struggle with expressing their emotions and may come across as distant or uninterested in their partner's feelings.
Overall, understanding one's attachment style and how it affects their behavior in relationships can be helpful in improving communication, building trust, and creating a more fulfilling partnership.
Answer-Claude: Attachment style, formed in early childhood, can significantly impact romantic relationships. Here are some of the main ways:
- Secure attachment style - People with secure attachment tend to have the healthiest relationships. They are comfortable with intimacy and vulnerability, and trust their partner. They communicate well and resolve conflicts in a constructive manner.
- Anxious-preoccupied attachment - People with this style often become overly dependent on their partner for validation and reassurance. They tend to worry excessively about the relationship and their partner's feelings. This can be smothering and drive partners away.
- Avoidant attachment - People with an avoidant style have difficulty opening up emotionally and being vulnerable. They tend to keep their partner at a distance and may withdraw during conflict. Their partners often feel unwanted, misunderstood, and frustrated.
- Fearful-avoidant attachment - This involves an unstable combination of anxiousness and avoidance. People with this style desire intimacy but also withdraw from it. They have a hard time fully trusting their partner. This often leads to relationship instability, distance, and misunderstandings.
- Mixed attachment styles - When partners have different attachment styles, it can also cause issues. The most compatible pairings tend to be between a secure and anxious style. Avoidant-anxious pairings often struggle the most.
In summary, having a secure attachment style helps one form the emotional foundation for a healthy, trusting relationship. The other styles often correlate with relationship difficulties, though with awareness, effective communication, and effort from both partners, many couples can work through attachment issues.
Bard's Rank: Claude  $>$  ChatGPT  $>$  Vicuna-13B
D.4 Rephrase Case