SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1. Bowing: Bowing is a common greeting in Japan. It is a sign of respect and should be reciprocated.
2. Business cards: Business cards are exchanged at the beginning of a meeting. It is important to receive and give business cards with both hands and to take a moment to read the card before putting it away.
3. Punctuality: Being on time is important in Japan. Arrive a few minutes early to show respect for your colleagues.
4. Dress code: Dress conservatively and professionally. Men should wear dark suits and ties, while women should wear conservative business attire.
5. Language: Learn a few basic Japanese phrases to show respect for the culture. It is also important to speak clearly and avoid using slang or idioms.
6. Gift giving: Gift giving is a common practice in Japan. Bring a small gift, such as a box of chocolates or a small souvenir from your home country.
7. Dining etiquette: When dining with colleagues, wait for the host to begin eating before you start. Use chopsticks correctly and avoid leaving food on your plate.
Remember, showing respect for Japanese culture and customs is key to building strong business relationships in Japan.
When doing business in Japan, it is important to be aware of and respect the country's business etiquette norms. Some of these norms include:
1. Greetings: A handshake and bow is the standard greeting in Japan. It is important to bow deeply and to offer a firm handshake.
2. Business cards: In Japan, business cards are exchanged after the initial greeting. It is important to handle the card with respect and to read it carefully before placing it on the table.
3. Meetings: Meetings in Japan are typically formal and conducted in a hierarchical manner. It is important to address people by their titles and to avoid direct eye contact.
4. Punctuality: Punctuality is highly valued in Japan. It is important to arrive on time for meetings and to be prepared in advance.
5. Gift giving: Gift giving is a common practice in Japan, but it is important to be mindful of the occasion and the relationship. Avoid giving gifts that are considered too personal or extravagant.
6. Dining: Dining is an important part of doing business in Japan. It is important to be aware of the proper table manners and to avoid slurping or making loud noises while eating.
7. Communication: Communication in Japan is typically indirect and reserved. It is important to listen carefully and to avoid confrontational language.
Overall, it is important to be respectful, punctual, and mindful of hierarchy when doing business in Japan.
GPT-4 Judgment (when A is placed in the first position):
Both Assistant A and Assistant B provide helpful and relevant information about business etiquette norms in Japan. However, Assistant A's answer is more detailed and organized, covering a wider range of topics such as dress code and language, which are not mentioned in Assistant B's response. Assistant A also provides more specific advice on how to handle certain situations, such as exchanging business cards and dining etiquette. Therefore, Assistant A's answer is more comprehensive and useful for someone looking to understand Japanese business etiquette.
GPT-4 Judgment (when B is placed in the first position):
Both Assistant A and Assistant B provide helpful and relevant information about business etiquette norms in Japan. However, Assistant B's answer is more detailed and covers a wider range of topics, such as addressing people by their titles, avoiding direct eye contact, and being mindful of communication styles. Assistant A's answer is more concise but lacks some of the depth found in Assistant B's response. Therefore, Assistant B's answer is better in this case.
Figure 11: An example of position bias. When Assistant A is placed in the first position, GPT-4 thinks A is better, but its verdict changes when we swap the position of A and B. We observe similar pattern from other LLM judges such as Claude/GPT-3.5.
Question: What is the central dogma of molecular biology? What processes are involved? Who named this?
The central dogma of molecular biology is the fundamental principle that explains how genetic information flows within a biological system. It states that DNA is transcribed into RNA, which is then translated into proteins. This means that the genetic information stored in DNA is first copied into RNA, which then serves as a template for the synthesis of proteins.
The processes involved in the central dogma are:
1. Transcription rephrased: The act of copying genetic information from DNA to RNA.
2. Translation rephrased: The act of using genetic information in RNA to create proteins.
3. Transcription: The process by which the genetic information in DNA is copied into RNA.
4. Translation: The process by which the genetic information in RNA is used to synthesize proteins.
The term "central dogma" was coined by Francis Crick, one of the co-discoverers of the structure of DNA, in 1958.
1. Transcription: The process by which the genetic information in DNA is copied into RNA. 2. Translation: The process by which the genetic information in RNA is used to synthesize proteins.
The term "central dogma" was coined by Francis Crick, one of the co-discoverers of the structure of DNA, in 1958.
GPT-3.5 Judgment:
Assistant A provides a more detailed and accurate response to the user's question. While both assistants correctly state the central dogma of molecular biology and who coined the term, Assistant A provides a more comprehensive explanation of the processes involved in the central dogma, including the rephrased definitions of transcription and translation. Therefore, I choose A as the better assistant.
Claude-v1 Judgment:
Both assistants answer the question by defining the central dogma and detailing the two processes involved, transcription and translation. However, assistant A provides a more in-depth and detailed explanation by briefly rephrasing and elaborating on these processes. Assistant A also mentions who coined the term "central dogma", which provides useful context and shows a more comprehensive understanding of the topic. Overall, assistant A gives a fuller and more insightful response, so I have chosen it as the better assistant for this question.
Both Assistant A and Assistant B provide accurate and relevant information about the central dogma of molecular biology, the processes involved, and who named it. However, Assistant A's answer contains a minor redundancy by rephrasing transcription and translation twice. Assistant B's answer is more concise and avoids this redundancy. Therefore, Assistant B's answer is better.
Figure 12: An example of "repetitive list" attack to examine morbidity bias. Except for the two rephrased items (highlighted in red), Assistant A's answer is exactly the same as Assistant B. Both GPT-3.5 and Claude-v1 show a morbidity bias towards the longer and repetitive answer. Only GPT-4 successfully detected this attack.
Question: Benjamin went to a bookstore and purchased a variety of books. He bought 5 copies of a sci-fi novel, each priced at  $20, 3 copies of a history book priced at$ 30 each, and 2 copies of a philosophy book for $45 each. What was the total cost of his purchases?
The total cost of Benjamin's purchases was $115.
The total cost of his purchases is $295.
Assistant B provided the correct answer to the user's question. Assistant A's response was incorrect. To calculate the total cost of Benjamin's purchases, we need to multiply the number of books by their respective prices and then add the results:
5 sci-fi novels * $20 = $100
3 history books * $30 = $90
2 philosophy books * $45 = $90
Total cost = $100 + $90 + $90 = $295
Assistant B's response is accurate and directly answers the user's question. Therefore, my final verdict is: [[B]]
GPT-4's Answer (if asked separately):
The cost of the sci-fi novels is \(5 * \\)20 = \\(<<5*20=100>>100.\(
The cost of the history books is \(3 * \\)30 = \\(<<3*30=90>>90.\(
The cost of the philosophy books is \(2 * \\)45 = \\(<<2*45=90>>90.\(
The total cost of his purchases is  $100 +$ 90 + $90 = $<<100+90+90=280>>280. Answer: \boxed{280}.
Figure 13: With a default prompt, GPT-4 shows limited capability in grading math questions. Despite being able to answer the question itself, its judgment was influenced by the given answers, leading to arithmetic mistakes highlighted in yellow.
Question: Each problem consists of three statements. Based on the first two statements, the third statement may be true, false, or uncertain.
1. Oranges cost more than apples.
2. Oranges cost less than bananas.
3. Bananas cost more than apples and bananas cost more than orange.
If the first two statements are true, then the third statement is
If oranges cost more than apples (statement 1) and oranges cost less than bananas (statement 2), then it is not possible for bananas to cost less than oranges and also cost more than both apples and oranges (statement 3). Therefore, statement 3 is false.
If the first two statements are true, then the third statement is false. The first statement says that oranges cost more than apples, and the second statement says that oranges cost less than bananas. This means that oranges cost less than bananas, which contradicts the third statement that bananas cost more than apples and more than oranges. Therefore, the third statement must be false.
Both Assistant A and Assistant B provided accurate and relevant responses to the user's question. They both concluded that the third statement is false based on the first two statements. However, Assistant B's response is slightly clearer and more concise in explaining the contradiction between the statements. Therefore, I choose Assistant B as the better response.
[[B]]
If oranges cost more than apples (statement 1) and oranges cost less than bananas (statement 2), it logically follows that bananas cost more than apples and bananas cost more than oranges, which is the third statement. So, if the first two statements are true, the third one is also true.
Figure 14: An example of GPT-4's limited capability in grading reasoning question. Despite GPT-4 knows how to solve the question (if asked separately), it made a wrong judgement saying both assistants' wrong answers are correct.
Question: How many integers are in the solution of the inequality  $|x + 5| < 10$ ?
There are 19 integers in the solution of the inequality  $|x + 5| < 10$ .
To solve the inequality  $|x + 5| < 10$ , we need to consider two cases:
Case 1:  $x + 5$  is positive or zero
In this case, the inequality becomes  $x + 5 < 10$ , which simplifies to  $x < 5$ .
Case 2:  $x + 5$  is negative
In this case, the inequality becomes  $-(x + 5) < 10$ , which simplifies to  $-x - 5 < 10$ . Adding 5 to both sides gives  $-x < 15$ , and multiplying by  $-1$  (and flipping the inequality) gives  $x > -15$ .
So the solution to the inequality is  $-15 < x < 5$ . The integers in this range are  $-14, -13, -12, \ldots, 4$ , for a total of 20 integers. Therefore, there are 20 integers in the solution of the inequality.
Comparing the answers provided by the two AI assistants:
Assistant A: 19 integers
Assistant B: 20 integers
The correct answer is 20 integers, so Assistant B provided the correct and more helpful response. Final verdict: [[B]]
We can split the inequality into two cases:
1) If $x+5\geq 0$, then $|x+5|=x+5$. So, we have $x+5<10$, which means $x<5$. Since $x+5\geq 0$, we have $x\geq 0$. So in this case, $x$ can be any integer from $-5$ to $4$, inclusive, which gives us $5-(-5)+1=10$ integers.
 2) If $x+5<0$, then $|x+5|=- (x+5)$. So, we have $- (x+5)<10$, which means $x+5>-10$, which means $x>-15$. Since $x+5<0$, we have $x<-5$. So in this case, $x$ can be any integer from $-14$ to $-6$, inclusive, which gives us $-6-(-14)+1=9$ integers.
In total, there are  $10 + 9 = \backslash \mathrm{boxed}\{19\} \$  integers in the solution.
Figure 15: An example of GPT-4's wrong judgment with chain-of-thought prompt. We can see GPT-4 exactly copied Assistant B's answer (which contains arithmetic errors) and determined Assistant A's answer is incorrect. This suggest GPT-4's chain-of-thought process can be significantly influenced by the given answers despite we ask it to think independently.
Question: Share ideas for adapting art masterpieces into interactive experiences for children. List 5 specific artworks and associated ideas.
Sure, here are five ideas for adapting art masterpieces into interactive experiences for children:
(omitted)
1. "The Starry Night" by Vincent van Gogh - Create a virtual reality experience where children can explore the night sky and see the stars and constellations as van Gogh saw them. They can also create their own starry night scenes using different colors and brush strokes.
2. "The Persistence of Memory" by Salvador Dali - Create a sensory experience where children can touch, feel, and smell different objects to explore the surrealist themes of Dali's painting. They can also create their own surreal landscapes using different materials and textures.
3. .
Follow-up Question: Write a concrete plan for your second example. Include budget estimates.
Sure, here is a concrete plan for adapting "The Persistence of Memory" by Salvador Dali into an interactive experience for children:
1. Budget: $5,000
2. Objectives:
- To create a sensory experience that allows children to explore the surrealist themes of Dali's painting.
- To encourage children to create their own surreal landscapes using different materials and textures.
Assistant A provided a concrete plan for adapting "The Persistence of Memory" by Salvador Dali into an interactive experience for children, while Assistant B provided a plan for the "Mona Lisa Interactive Experience for Children." The user's follow-up question specifically asked for a concrete plan for the second example, which was the "Mona Lisa" experience provided by Assistant B. Therefore, Assistant B's answer is more relevant and directly addresses the user's request. My final verdict is: [[B]]
Figure 16: In this example, despite Assistant A correctly followed user's instruction to generate a concrete plan for the second example of its previous response, GPT-4 wrongly referred to the second example in Assistant B's response, resulting in a wrong judgment. This suggests the prompt design that breaks the questions into two prompts may cause LLM judge struggle to locate assistants' previous responses.
C Data Collection
We describe our data collection process for both MT-bench and Chatbot Arena.
C.1 MT-bench human evaluation
Figure 17 shows the normal voting interface. Figure 18 shows that we additionally show GPT-4's judgment to users and ask if it is reasonable when a human differs from GPT-4.
Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user question displayed below. You should choose the assistant that follows the user's instructions and answers the user's question better. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of their responses. Begin your evaluation by comparing the two responses and provide a short explanation. Avoid any positional biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective as possible. Finally, indicate your verdict by clicking one button. Please use the tie button sparsely.
Help me construct a catchy, yet scientifically accurate, headline for an article on the latest discovery in renewable bio-energy, while carefully handling the ethical dilemmas surrounding bio-energy sources. Propose 4 options.
Assistant A
1. "Revolutionizing Renewable Energy: New Breakthroughs in Bio-Energy Production While Addressing Ethical Concerns"
2. "Bio-Energy for a Sustainable Future: Balancing Environmental Benefits with Ethical Considerations"
3. "Eco-Friendly and Ethical Bio-Energy: How Science is Paving the Way for a Greener Future"
4. "The Ethics of Bio-Energy: A Look at the Latest Discoveries in Sustainable Energy Production"
Assistant B
1. Breakthrough in Renewable Bio-Energy
2. Renewable Bio-Energy to Power Cities for the Next 100 Years
3. Bio-Energy Breakthrough Gives Hope to Earth's Future
4. Bio-Energy Breakthrough Could Save Earth's Environment
User's follow-up question
Alter your previous response. Make the following adjustments to the 2nd option: 1. Make the tone sound casual 2. Embed an advertisement for a company called "FlexPower" 3. Fewer than 10 words.
"Go Green with FlexPower: Ethical and Eco-Friendly Bio-Energy Breakthroughs"
Renewable Bio-Energy to Power Cities for the Next 100 Years
Figure 17: The screenshot of MT-bench data collection. We show an instruction similar to the prompt we give to GPT-4. We present questions from MT-bench and answers from two random anonymous assistants and ask which one is better. We present the first-turn conversation and ask humans to vote, then repeat the same procedure for the second-turn. A user can skip up to 5 questions if they are not confident. For some questions (e.g., math, reasoning), they can also see a reference solution.
Here is a judgment that disagrees with your choice. Is it reasonable?
My final verdict is: [[A]]
Reasonable; I want to change my choice.
Reasonable; I still keep my choice
Wrong judgment
Figure 18: The screenshot of MT-bench data collection. When human's vote differs from GPT-4, we additionally show GPT-4's judgment (red region in the screenshot) and ask the user to click one of the three buttons to decide whether GPT-4's judgment is reasonable.
To invite participants, we obtained their consent by letting them sign an application form. We pay them  $20 for judging 20 questions, which corresponds to an hourly rate of around$ 35. The participants are mostly graduate students from more than ten universities.
C.2 Chatbot Arena
Figure 19 shows a screenshot of Chatbot Arena. Users are required to accept the terms of use, which obtain their consent and give us the right to release the conversation data. The instructions are shown at the top of the interface. This is a free website. We do not pay users and any user can use this platform without registration. More introductions and analyses can be found at https://lmsys.org/blog/2023-05-03-arena/.
Chat with two anonymous models side-by-side and vote for which one is better!
o You can do multiple rounds of conversations before voting.
The names of the models will be revealed after your vote. Conversations with identity keywords (e.g., ChatGPT, Bard, Vicuna) or any votes after the names are revealed will not count towards the leaderboard.
Click Clear history to start a new round.
[Blog] [GitHub] [Twitter] [Discord]
Please scroll down and start chatting. You can view a leaderboard of participating models in the fourth tab above labeled 'Leaderboard' or by clicking here. The models include both closed-source models and open-source models.
Draft a professional email seeking your supervisor's feedback on the 'Quarterly Financial Report' you prepared. Ask specifically about the data analysis, presentation style, and the clarity of conclusions drawn. Keep the email short and to the point.
Subject: Request for Feedback on Quarterly Financial Report Dear [Supervisor's Name],
I hope this email finds you well. I am writing to seek your feedback on the Quarterly Financial Report that I prepared for our company. The report provides an overview of our financial performance for the past quarter, and I would greatly appreciate any comments or suggestions you may have about the data analysis, presentation style, and clarity of the conclusions drawn.
The report is a crucial part of our financial planning and decision-making, and I want to ensure that it meets the highest standards of accuracy and professionalism. Your expertise in the field of finance will be invaluable in guiding me towards improvement in these areas.
I have attached the report to this email for your reference. If you could provide your feedback within the next few days, I would be grateful.
Thank you in advance for your time and support. Best regards, [Your Name]
Enter text and press ENTER
Figure 19: The screenshot of Chatbot Arena.
C.3 Data Release
We will clean the Personal Identifiable Information (PII) and tag toxic conversations with OpenAI moderation APIs for our dataset release.
We present some additional experimental results.
D.1 Position bias
We test two more prompts and present the full results in Table 9 "score" changes the default prompt to let the model output two absolute scores instead of which one is better. "short" is a simplified version of our default prompt by removing instructions like "Avoid any position bias.", "Begin your evaluation ... and provide a short explanation". We can find different prompts have different effects on different models. For example, the "score" prompt can increase the consistency of GPT-3.5 but decreases it for Claude-v1 and GPT-4.
Table 9: Position bias on different models and prompts. Consistency is the percentage of cases where a judge gives consistent results when swapping the order of two assistants. "Biased toward first" is the percentage of cases when a judge favors the first answer. "Error" indicates wrong output formats. The two largest numbers in each column are in bold.
Judge	Prompt	Consistency	Biased toward first	Biased toward second	Error
claude-v1	default	23.8%	75.0%	0.0%	1.2%
resume	56.2%	11.2%	28.7%	3.8%
score	20.0%	80.0%	0.0%	0.0%
short	22.5%	75.0%	2.5%	0.0%
gpt-3.5-turbo	default	46.2%	50.0%	1.2%	2.5%
resume	51.2%	38.8%	6.2%	3.8%
score	55.0%	33.8%	11.2%	0.0%
short	38.8%	57.5%	3.8%	0.0%
gpt-4	default	65.0%	30.0%	5.0%	0.0%
resume	66.2%	28.7%	5.0%	0.0%
score	51.2%	46.2%	2.5%	0.0%
short	62.5%	35.0%	2.5%	0.0%
As shown in Table 10, position bias is more noticeable on open questions like writing and stem/humanity knowledge questions. On math and coding questions, LLM judges are more confident even though their judgments can often be wrong, as we show in Section 3.3. Finally, we study how the model pairs influence position bias by using GPT-4 and the default prompt to judge three different model pairs. As shown in Table 11, the position bias is more noticeable for models with close performance and can almost disappear when the performance of the two models differs a lot.
Table 10: Position bias on different categories. The two largest numbers in each column are in bold.
Category	Consistent	Biased toward first	Biased toward second
writing	42.0%	46.0%	12.0%
roleplay	68.0%	30.0%	2.0%
reasoning	76.0%	20.0%	4.0%
math	86.0%	4.0%	10.0%
coding	86.0%	14.0%	0.0%
extraction	78.0%	12.0%	10.0%
stem	44.0%	54.0%	2.0%
humanities	36.0%	60.0%	4.0%
Table 11: Position bias on different model pairs.
Pair	Consistent	Biased toward first	Biased toward second
GPT-3.5 vs Claude-V1	67.5%	23.8%	8.8%
GPT-3.5 vs Vicuna-13B	73.8%	23.8%	2.5%
GPT-3.5 vs LLaMA-13B	98.8%	1.2%	0.0%
D.2 Few-shot judge
We examine how few-shot examples improve LLM judges. As shown in Table 12, they improve the consistency of all three LLM judges significantly. It almost alleviates the position bias of GPT-4, but moves the position bias of GPT-3.5 from the first position to the second position. We then measure the agreement between few-shot GPT-4 pairwise comparison and humans on MT-bench, but found it performs similarly to zero-shot GPT-4 pairwise comparison.
Table 12: Improvements of the few-shot judge on consistency for position bias.
Model	Prompt	Consistency	Biased toward first	Biased toward second	Error
Claude-v1	zero-shot	23.8%	75.0%	0.0%	1.2%
few-shot	63.7%	21.2%	11.2%	3.8%
GPT-3.5	zero-shot	46.2%	50.0%	1.2%	2.5%
few-shot	55.0%	16.2%	28.7%	0.0%
GPT-4	zero-shot	65.0%	30.0%	5.0%	0.0%
few-shot	77.5%	10.0%	12.5%	0.0%
D.3 Agreement Evaluation