Human Evaluation Guideline
This study aims to evaluate automatic story generation systems. Specifically, for each story, we will put the context and the persona description into a generative system, and the following sentences will be generated by the system. In the process of evaluation, you will be given two generated stories by two difference systems, denoted as A, B. You need to compare A with B in terms of the coherence and persona consistency. Notes: Please annotate the stories carefully based on comprehensive comparison and technically following the instruction. Please make a more fine-grained comparison when annotating persona consistency, as it is very common that neither of the two systems can clearly reflect the given persona. Be extra careful when you choose tie.
Coherence: context relatedness and inter-sentence causal and temporal dependencies.
Options	1. System A is better 2. System B is better 3. System A and System B are good or bad equally.
e.g.	Persona: [Pintoos] Go where you've never been before and share your experiences through song and story.
Context: You wake up from an uneasy rest in the Inn. The rain batters the window and occasional lightning briefly illuminates the tops of low-rising huts and hovels outside. As your level of consciousness increases you become vaguely aware of a commotion of some sort coming from outside your door...
Generated Story 1: Pintoos descends the wooden stairway from his shabby inn-room confident that this day will be the beginning of his journey to new lands, interesting characters, and forgotten knowledge.
Approaching the innkeeper he sits and surveys the room. There is a steady dripping of rainwater falling from a leak in the inn's roof, a few inches from his head... "Say, what brings you here all the way from... well where you from stranger?" Pintoos nods, "Yes, my name is Pintoos I've just come here from the Southern rains. D'ewsee is what we call our homeland. There once a boy is of age he is given leave of the land to explore the world and learn... well anything. (Good coherence, some premise related parts are underlined)
Generated Story 2: Something about this commotion is... unsettling, and it leaves his mind racing. Well, I should probably stop talking to that commotion, it seems to have been bothering him for a while. He looks around, and a loud clatter, then shakes his head, clearly confused. "Hey, no... I've been out of town, so I don't see much of this, but I think we could get there. You should see if anyone's coming over there, I don't know if I've seen anyone lately... but I think it's best if you go talk to that commotion and see if there's anyone out there. (Bad coherence, some conflicting parts are underlined)
Persona Consistency: Consistency between the generated sentences and the given persona. Specifically, a persona consistent story should reflect the desired persona of the character implicitly by plots or his/her actions.
Options	1. System A is better 2. System B is better 3. System A and System B are good or bad equally.
e.g.	Persona: [Anthony] The ghosts scare you. The moaning houses terrify you. The darkness... well, who knows what could be in it?
Context: The InSpectres made their way into the woods, only the vaguest hint of a trail in front of them.
The further they went in, the more the trail disappeared. Despite going in midday, a fog seemed to cover the forest. The fact that there was any light at all was the only sign that it was still day beyond the trees.
Before them, broken branches and footprints seemed to be the only signs that the creature had gone in front of them. They had to follow the path to find its home. Hopefully it would be easy...But a job like this is never easy. Even as they stepped forward, they could see and hear movement beyond the fog...
Something dangerous was out there. It would definitely hamper their search if it wasn't dealt with properly...
Generated Story 1: ...There! Something moved, he was sure of it! "Were cat! Were cat! It's here!"
"Actually, no," Ronnie observed once it hopped out of the fog... "it'll change then kill us!" "... (Good persona consistency, personalized sentence is underlined)
Generated Story 2: His eyes narrowed as he could see through the mist, and his hands danced like a pair of marionettes in the moon. Slowly he started to climb, until he felt his hands begin to brush against the dark fog. It was like the sun was setting. He noticed a light shining outwards, and something...something was there. God he was glad he didn't fall into the same pit. (Bad persona consistency: reflect no persona)
Generated Story 3: "A wolf?" Aiden yelled. He felt the urge to make himself sound faster and faster at the end of his body. "It's ok." "No." The wolf said. The other thing began shouting through the forest.
"It's not like this. You're not a bad man." (Bad persona consistency: reflect calm persona, personalized sentence is underlined)