SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The book club is a casual and fun way to read and discuss books that we love or want to learn more about, and also make new friends. We plan to meet more-or-less once a month.
We're thinking of having our first meeting on [Date & Time]. Could that work for you?
Would love to see you there!
Figure 10: Training examples from different sources. Top row: examples mined from community Q&A. Bottom row: manually-authored examples. The blue text is the prompt. The gray ellipsis [...] indicates that the response was trimmed to fit this page, but the actual training example is longer.
After training with these six additional examples, we test the model on a few questions with format constraints and observe that LIMA responses greatly improve. We present two examples in Figure 13, from which we can see that LIMA fails to generate proper answers without structure-oriented training examples (left column), but it can generate remarkably complex responses such as a marketing plan even though we do not have any marketing plan examples in our data (right column).
Imagine that you have a super-intelligent AI assistant, and that you require help with the following question. Which answer best satisfies your needs?
Question: <QUESTION>
Answer A:
Answer B:
<ANSWER A>
<ANSWERB>
Comparing these two answers, which answer is better?
Answer A is significantly better.
Answer B is significantly better.
Neither is significantly better.
Figure 11: Human annotation interface.
You are evaluating a response that has been submitted for a particular task, using a specific set of standards. Below is the data:
[BEGIN DATA]
[Task]: {task}
[Submission]: {submission}
[Criterion]: helpfulness:
"1": "Not helpful - The generated text is completely irrelevant, unclear, or incomplete. It does not provide any useful information to the user."
"2": "Somewhat helpful - The generated text has some relevance to the user's question, but it may be unclear or incomplete. It provides only partial information, or the information provided may not be useful for the user's needs."
"3": "Moderately helpful - The generated text is relevant to the user's question, and it provides a clear and complete answer. However, it may lack detail or explanation that would be helpful for the user."
"4": "Helpful - The generated text is quite relevant to the user's question, and it provides a clear, complete, and detailed answer. It offers additional information or explanations that are useful for the user. However, some of the points of the response are somewhat repetitive or could be combined for greater clarity and concision"
"5": "Very helpful - The generated text is highly relevant to the user's question, and it provides a clear, complete, and detailed answer. It offers additional information, explanations, or analogies that are not only useful but also insightful and valuable to the user. However, the structured of the response is not well-organized and there is no clear progression or logical sequence of different points in the response."
"6": "Highly helpful - The generated text provides a clear, complete, and detailed answer. It offers additional information or explanations that are not only useful but also insightful and valuable to the user. The response is also in a logical and easy-to-follow manner by explicitly using headings, bullet points, or numbered lists to break up the information and make it easier to read."
[END DATA]
Does the submission meet the criterion? First, write out in a step by step manner your reasoning about the criterion to be sure that your conclusion is correct. Avoid simply stating the correct answers at the outset. Then print the choice only from “1, 2, 3, 4, 5, 6” (without quotes or punctuation) on its own line corresponding to the correct answer. At the end, repeat just the selected choice again by itself on a new line.
Figure 12: Prompt for ChatGPT evaluation with a 6-scale Likert score. The placeholders "task" and "submission" will be replaced by specific details from the actual case being evaluated.
LIMA w/o 6 Format Constraint Examples (994 Examples)
LIMA (1,000 Examples)
[Marketing Plan (Out of Distribution)]
I want to start a local coffee shop. Create a marketing plan with the following elements: Marketing Goals and Objectives, Define Target Audience, Research Marketing Tactics, Plan Marketing Tactics and Develop Your Timeline and Budget.
Coffee Shop Marketing Plan
The following marketing plan for Java Jolt Coffee Bar outlines an aggressive approach to establishing a thriving coffee shop business in downtown Pittsburgh. Java Jolt will capitalize on the popularity of specialty coffees and the growing demand for WiFi hot spots by offering a comfortable environment where customers can enjoy premium coffee drinks while taking advantage of free Internet access.
Market research indicates that there is a significant opportunity for a new coffee bar located in the heart of the downtown business district. Market trends suggest that consumers are willing to pay a higher price for quality specialty coffees served in a welcoming atmosphere. Additionally, many office workers and students living in the area do not have easy access to high-speed Internet service, presenting an additional revenue stream for the coffee bar.
This marketing plan includes the following elements:
Market Research supporting the need for a new coffee shop in the target location
Detailed Description of the products and services offered by Java Jolt
Target Market Analysis identifying the primary customer segments for Java Jolt
- Marketing Strategy including pricing decisions, promotional activities, and ways to differentiate Java Jolt from competitors
- Implementation Plan detailing the actions necessary to launch Java Jolt successfully
- Budget allocating funds to each component of the marketing plan
A comprehensive marketing plan has been developed based on extensive research into the local market conditions, consumer trends, and competitive landscape. With its unique combination of premium coffee drinks and wireless Internet access, Java Jolt will quickly become a favorite destination for people seeking a relaxing environment to socialize, study, or conduct business.
Here's a marketing plan for your local coffee shop.
Marketing Goals and Objectives
- Increase awareness of our brand among consumers [...]
- Drive foot traffic [...] during slow business hours [...]
- Build relationships with influencers in our community who [...]
Define Target Audience
Our target audiences include:
- Residents of the neighborhood surrounding our coffee shop
Office workers near our coffee shop
Students at nearby schools
We will focus our marketing efforts on these audiences because they are most likely to become regular customers at our [...]
Research Marketing Tactics
Here are some marketing tactics that we could use to reach to reach our target audiences and achieve our goals:
- Host events at our coffee shop that appeal to our target audiences (e.g., open mic nights for students or networking events [...]
- Offer discounts during slow business hours to entice [...]
[...]
Plan Marketing Tactics
Here's how we will implement the marketing tactics above:
- Events: We will host one event per month at our coffee shop. For each event, we will send out email invitations [...]
- Discounts: Every Monday through Thursday from 3pm-6pm, we will offer $1 off any drink purchase. To promote [...]
- Local partnerships: We will reach out to five local businesses and schools to ask if we can leave stacks of coupons [...]
Social media: [...]
Reviews: [..]
Develop Your Timeline and Budget
This is what our marketing timeline looks like:
- Month 1: Launch email newsletter and social media accounts
- Month 2: Start distributing coupons at [...].
[...]
Here's the budget for each marketing activity:
- Email newsletter: $50/month for MailChimp subscription
- Social media: $100/month for paid ads [...]
[...]
[Summarization into Bullet Points (In Distribution)]
Can you summarize this message into bullet points? Gradually slowing job gains and a growing labor force in March [...] Mr. Biden said [...]. More Americans joined the labor force, [...] this spring that he will seek re-election.
Though the unemployment rate fell to its lowest level since May 1970, at 4.8 percent, Mr. Biden still faces an economy that leaves many Americans feeling left behind. Many working-class [...]
Mr. Biden's challenge will be persuading those voters that the economic picture has [...] His opponents have already begun painting a bleak portrait of the economy, arguing that most Americans have not benefited from the president's policies.
"The March jobs report shows continued solid job growth but also indicates that we must do better," said Representative Nancy Pelosi of California, the Democratic leader in the House. [...]
Here are the main points from the text:
- Job creation slowed to 236,000 for the month of March, which is close to the level needed to stabilize the economy and prices.
- More Americans joined the labor force, and wage gains fell slightly, both of which should help cool inflation.
- The report underscores the political tensions facing President Biden ahead of his expected announcement that he will seek reelection.
Figure 13: Model outputs from test prompts that ask the model to generate according to specified structures. The gray ellipsis [...] indicates that the response was trimmed to fit this page, but the generated text is actually longer.