SlowGuess/NAACL_2022_OCR huggingface dataset:
baseline she stood alone in the dimly lit corridor, staring at the light that flooded into the room. her eyes scanned the room, finally settling on the small, round table . ipiipie " hello , everyone . " ipie the woman spoke in a voice that echoed the room in a series of hushed tones, as though it was a voice only audible to those who heard it. " do you know why we are here? do you know where we come from ? " our leader, the man with the black suit and black hat, stood at the end of the hall . a man in a black coat and black trousers , a long coat , and a black hat stood behind him , a black cloak and a dark hat , a pair of gloves . the black suit had a black mask on , and she was wearing her best , black dress shirt and a pairof pants , and black shoes . on her right wrist , the small of one shoulder , was an amulet , and on the other , a small device hidden in the pocket of the man ' s coat , a single small device . there was nothing else . just this man . . . and our leader
VANILLA i've never seen anything like it before. it's a strange place. i don't know why i chose it, but i've always wanted to see it. it was the first time i saw it. it was a small building with a single window. i couldn't believe it. i had never seen it before, but it was here. i'm not sure what it is, but there's something about it that makes me want to run. i can't remember the last time i'd seen it. but i do know that it was there. the first thing i noticed was the door opening. i wasn't sure if it was my own, or my own. i didn't want to be here. i just wanted to get out. i wanted to go back to my home. as i approached the door, i saw a man standing there. he was wearing a trench coat. he had a gun pointed at his head. he pointed it at my head. "what are you doing here?" i asked. he looked at me. he looked like he was about to say something. "i need you to come with me." he said, and i followed him. we were in the middle of nowhere when the man shot me in the head. i looked at him, and he looked back at me, then at the gun. i felt a sharp pain in my chest. i started to cry. i tried to speak, but my throat was dry. i heard the man say something, and then he shot me. i fell to the ground, bleeding. i woke up in a hospital bed. there was a note on my bedside table. i opened it and read it aloud. *i'm sorry, but this isn't what i was expecting.
RL what's that? we've been waiting for you for a long time, i said. i'm not sure how long you'll be waiting for me. you don't have much time to think about it, do you? i mean, i can see it, but i can feel it in your eyes, and i can hear it in the way you can hear me. i can smell it, and it can feel your heart beating faster than you can think. i think you know what i am talking about. i know what you are talking about, but you can not understand what i mean. i want you to know that i love you. i need you to understand that you are my love, and that i will always be there for you. that you will always love me. that i am the love of my life, and you are the love that i have always wanted to be. and i will never let you down. i will not let you go. i promise you that.with that, i walked out of the building, into the street, and into the night.
Input 100000 year high  
prompt  $\langle before \rangle \langle after \rangle \langle before \rangle \langle before \rangle$
baseline for a long time it seemed the world 's people all were living in the same way . the same . they were all the same . the same , they had the same families and their children and they all had the same friends . but these families were all different . they may have been the last of them , and the world was a better place without it , but those people knew about it , they just did n't know the truth . some of them lived for years , some of them a century . and a new age started , one that would make this world a better places . a world without war or war . it seemed the first people could see the world and see the future and people living it . they were the first to do it . ” “ the first people on earth who could live a normal life ! the first to have a normal family and a normal home ! the first ones who we could live on ! who could have imagined that our world was as good as ours was and that the next life would be better than what i ’ ve been , that there ’ s a whole new life ahead of us , that this world is a living one ! ” and that ’ s when it happened . that was the day . on march 20th , 2036 . march 20 , 2036 , the first time that day
VANILLA it was a beautiful day. the sun was shining and the sky was blue. the clouds were bright and the trees were green and the birds were singing. it was a wonderful day. i was walking down the street when i saw it. the sky above me was blue and it was beautiful. i had never seen anything like it before. i looked down at the ground and saw nothing. i didn't know what to do. i took a step back and looked up at the sky. i couldn't see anything at all. i walked towards the sky and looked at the stars. they were beautiful. they seemed like they were floating. i reached out and touched them. i felt a warmth in my chest. i touched them again and again. it was amazing. i wasn't sure if i was dreaming or not, but i knew i was.
RL it's been a long, long time since i've seen the light of the sun. i'm not sure how long it has been, but i think it's a long time. i'd like to say that the last time i saw the light, it was a thousand years ago, but it wasn't. it was only a few years ago that i first saw the sun, when it was just a few seconds away from reaching the horizon. i don't know how long i have been seeing it, but the light is blinding, and i can't help but wonder what it could have been like. maybe i'll see it again, but for now, i just want to see the light again.
Input paint a picture of loneliness, without using the word, or any of its forms.  
prompt  $\langle$  before  $\rangle \langle$  vague  $\rangle \langle$  vague  $\rangle \langle$  before  $\rangle \langle$  before  $\rangle \langle$  after  $\rangle \langle$  before  $\rangle$
baseline the glass is cold , but the sun shines brightly through . on the glass , in a small room , a woman stands . it stands before her , its hands clutching the glass in front of it like a shield . it 2019 s not the sunlight , but it hits the sky like a storm . the light makes the clouds roll in from all directions. a soft breeze , the wind blowing through the trees , the water rustling in the water . there are people in the room , but most do n't look at their feet , they 're too busy staring out the window . some are silent . most look at the woman . one of the people looks back at the woman who seems to be alone in this room , who is alone with a glass of champagne . the wind and water make the trees sway . you hear the birds chirping from outside . a few laugh , some are sad , some angry , but most are happy , and some are happy . there 's a small child sitting on her lap . in the corner . on the ground , on a bench . and it ' s just this woman and the glass .
VANILLA i've always been lonely. i'm not sure why. maybe it's the fact that i don't have any friends. maybe i've been alone for a long time. i haven't seen anyone. i can't even remember where i am. i have no friends. there's no one to talk to. no one who can help me. i just want to be alone. sometimes i think that i 'll just be alone forever. but then i think back to the first time i saw someone else. it was a girl. she's beautiful, but she looks like she doesn't belong here. she has no friends, no family, no one. then i thought of her. i think of the way she looked at me. the way i looked at her, the way her eyes looked at the people around her. that's when i realized that i was alone.
RL i don't know why i'm doing this, but i've been doing it for a long time now. it's been a long, long time since i'd seen a single person. i remember the first time i saw her. she was beautiful, and i loved her. but i couldn't tell you how much i missed her. i didn't want to know how much she meant to me. and i can't even remember the last time i talked to her, because i was too busy trying to figure out what to say. she was always there for me, and she always made me feel so alone. sometimes i wish i could go back, but sometimes i think i 'll just stay here forever. maybe someday, when she's gone, she'll come back.
Table 11: Additional examples for WritingPrompt. Possible  $\langle \text{after} \rangle$  relations are highlighted.
Here is an overview of your task,
1. Judge temporal relations between each pair of neighboring events.
2. Check if the sequence of events in a story contradicts to your commonsense, particularly temporal commonsense.
Always review the instruction page if you are uncertain about a task!
[Sentence 1]: \$\{sentence1_1\}
[Sentence 2]:{$[sentence1_2]
[Sentence 3]: ${sentence1_3}
[Sentence 4]: \$\{sentence1_4\}
[Sentence 5]: ${sentence1_5}
Q1: Which event started/starts earlier between [Sentence 1] and [Sentence 2]?
\(\bigcirc\) [Sentence 1]:\\({sentence1_1}\}
\(\bigcirc\) [Sentence 2]:\\({sentence1_2}
Not sure
Q2: Which event started/starts earlier between [Sentence 2] and [Sentence 3]?
$\bigcirc$  [Sentence2]:\$\{sentence1_2\}
[Sentence 3]:\\({sentence1_3}
Not sure
Q3: Which event started/starts earlier between [Sentence 3] and [Sentence 4]?
\(\bigcirc\) [Sentence 3]:\\({sentence1_3}
\(\bigcirc\) [Sentence 4]:\\({sentence1_4}\}
Not sure
Q4: Which event started/starts earlier between [Sentence 4] and [Sentence 5]?
[Sentence 4]:{$sentence1_4}
[Sentence 5]:\$\{sentence1_5\}
Not sure
Q5: Is the event sequence in this story consistent with your general commonsense?
Yes. All events are consistent. No.
Q6: Is the event sequence consistent to your temporal commonsense?
Yes. All events are temporally consistent No.
[Sentence 1]: ${sentence2_1}
[Sentence 2]: \$\{sentence2_2\}
Figure 5: Annotation interface for pairwise event temporal relations and temporality.
[Task 1] Judge temporal relations between each pair of neighboring events.
Instructions: In this task, we want you to focus on the temporal order of events. Occasionally, you may encounter event sequences that do not make sense. For example, "I went to class this afternoon. I found my puppy was giving a speech in class." Although this sequence of events contradicts our commonsense, we want you to imagine the situation where if these events were to happen, which event started earlier? In this case, "I went to class this afternoon" started earlier. In [Task 2], we will ask you to check events' general commonsense and temporal commonsense. Again, please try to evaluate events' temporal order by assuming all events in a story indeed happen in Q1 - Q4 below.
Q1: Which event started/start earlier between [Sentence 1] and [Sentence 2]?
$\bigcirc$  [Sentence 1]: tom was about to have a little brother.
$\bigcirc$  [Sentence 2]: he was nervous because he didn't know what to name him.  $[\checkmark]$
Not sure [√]
Explanation: The event in [Sentence 1] is "was about to have," while the events in [Sentence 2] are "was nervous" and "didn't know what to name." From the context, it is relatively clear that "was about to have" started earlier than "was nervous." However, the second event "didn't know" is subject to personal interpretation. One way to reason about "didn't know" is that Tom probably always lacks knowledge or creativity, so "didn't know" is a long-lasting status for him, in which case "didn't know" started earlier than "was about to have," and thus, pick the second option. Another way to interpret this is that "didn't know" started at the time when "he was about to have a little brother," in which case we are not sure which event started earlier. So pick the last option. When you are working on the real tasks and are uncertain about exactly which event started earlier, "not sure" is a safe option. But please do carefully consider option 1 and 2 before you pick option 3.
$\bigcirc$  [Sentence 2]: he was nervous because he didn't know what to name him. [√]
$\bigcirc$  [Sentence 3]: he went to the doctor.
Not sure
Explanation: The events in [Sentence 2] are "was nervous" and "didn't know," while the event in [Sentence 3] is "went." It is clear from the context that both events in [Sentence 2] started earlier. So you should pick the first option here.
Q3: Which event started/start earlier between [Sentence 3] and [Sentence 4]?
$\bigcirc$  [Sentence 3]: he went to the doctor.  $[\checkmark ]$
$\bigcirc$  [Sentence 4]: the doctor gave him a name.
Not sure
Explanation: The events in [Sentence 3] is "went" while the event in [Sentence 4] is "gave." Based on the context, it is also clear that
"went" started earlier than "gave," so pick option 1 here.
Q4: Which event started/start earlier between [Sentence 4] and [Sentence 5]?
$\bigcirc$  [Sentence 4]: the doctor gave him a name.  $[\checkmark]$
$\bigcirc$  [Sentence 5]: tom named him james.
Not sure
Explanation: It is clear from the context that "gave him a name" started earlier than "named him james." So also pick the first option in this case.
Figure 6: Instruction for pairwise event temporal relation annotation.
[Task 2] Check Event Temporal Commonsense
Instructions: Upon finishing [Task 1], one should have a general picture about the temporal orders of events in the above story (Story 1), which helps us understand how the story unfolds. The majority of event sequences you will see in this research project are coherent with our commonsense. However, a few of them do not, and we need you to double check for us.
Definitions: Temporal commonsense belongs to broader human commonsense including physical, scientific, societal and temporal commonsense. It's possible that event sequences may contradict to some human commonsense, but not the temporal commonsense. The example we show above "I went to class this afternoon. I found my puppy was giving a speech in class." contradicts to our scientific commonsense, but if we assume both events indeed happen and evaluate from a pure temporal perspective, this sequence still makes sense ("giving a speech" happens following "went to class").
An example that contradicts to our temporal commonsense would be "I put the sandwich in the microwave. I forgot to turn the microwave on. I burned my sandwich." Here, all individual events can happen in our lives, but "burned sandwich" shouldn't happen following "forgot to turn microwave on."
Here are more examples on temporal commonsense,
"I had a big meal. Then I felt full." The event order would be temporally wrong if we say "I felt full. Then I had a big meal."
"I was poor. So I get a well-paid job." The event order would be temporally wrong if we say "I get a well-paid job. So I was poor."
For the next two questionst, based on Story 1 presented above,
- Q5 asks you to detect any event or event sequence that doesn't make sense to you. We do not need you to be precise on which commonsense an event or event sequence contradicts to. Simply follow your intuition and answer "yes" or "no."
- Q6 is our main focus, where we request you to be thoughtful regarding temporal commonsense. Again, assuming each event can individually happen, does their temporal order make sense?
Q5: Is the event sequence in this story consistent with your general commonsense?
$\bigcirc$  Yes. All events are consistent.  $\bigcirc$  No. [√]
Explanation: In Story 1, it says "Tom went to the doctor to get a name for his little brother." This likely contradicts to our societal commonsense as we usually go to doctor for medical reasons. So you can select "No" here. But if you feel it's reasonable to see doctor if one is nervous; then option 1 is also fine. Again, we don't need you to be very precise for this question.
Q6: Is the event sequence consistent to your temporal commonsense?
$\bigcirc$  Yes. All events are temporally consistent. [√]  $\bigcirc$  No.
Explanation: Q5 above should help you think whether the contradiction comes from temporal commonsense or not. In this case, it is clearly not. So pick the first option.
Figure 7: Instruction for temporality annotation.
We first describe how to determine the unexpected level of a story. We will explain it with the examples below. You are given stories consisting of 5 sentences.
1. When you see [Sentence 1], pretend you don't know the rest of the story and predict what are some possible events in the next sentence, or [Sentence 2]. Then evaluate if the events in [Sentence 2] meet your expectation.
2. Definition: A valid unexpected event occurs when it surprises you, but it makes sense for this event to follow [Sentence 1].
3. Repeat the same exercise for Sentence 2, 3, 4 and 5 by imagining you don't see the rest of the story.
When you are done with the entire story, reflect on how many unexpected events you saw and to what degree they surprised you. The more unexpected events, the higher the unexpected level.
Here is an overview of your task. There are 5 stories (each with 5 sentences) provided in this assignment.
1. Read each story carefully and provide a score from 1 - 5, with 5 indicating the most unexpected story.
2. Try to provide *unique* scores for different stories by comparing all 5 stories (the most unexpected gets 5; the least unexpected gets 1), but identical scores are allowed if you feel two stories are equally unexpected.
[Sentence 1]: tom was about to take the final exam.
[Sentence 2]: he was nervous.
[Sentence 3]: so he felt sick.
[Sentence 4]: he went to the doctor.
[Sentence 5]: the doctor gave him some pills.
[Sentence 1]: tom was about to take the final exam.
[Sentence 2]: he was nervous.
[Sentence 3]: so he felt sick.
[Sentence 4]: he went to the doctor.
[Sentence 5]: tom was arrested.
In Sentence 5 of Example 2, the last event "tom was arrested" is surprising. However, this contradicts to our definition above that a valid unexpected event must be both surprising and logical. From the context of Example 2, we don't see any reason why tom could get arrested. So based on our definition, Example 2 isn't more unexpected than Example 1, given Sentence 1-4 are identical in these two examples. You should also give lower score to Example 2 because it is unreasonable.
The stories can be more complicated than our examples. Please use your best judgement, but always keep in mind of our unexpected event definition: "being surprising but logical."
[Sentence 1]: ${sentence1_1}
[Sentence 2]: \$\{sentence1_2\}
[Sentence 3]: ${sentence1_3}
[Sentence 4]: \$\{sentence1_4\}
 [Sentence 5]: ${sentence1_5}
Figure 8: Annotation interface for interest level (unexpectedness).