SlowGuess/NAACL_2022_OCR huggingface dataset:
Jian Guan, Ziqi Liu, Minlie Huang*
The CoAI group, DCST; Institute for Artificial Intelligence; State Key Lab of
Intelligent Technology and Systems; Beijing National Research Center for
Information Science and Technology; Tsinghua University, Beijing 100084, China.
{j-guan19, liuzq19}@mails.tsinghua.edu.cn,
aihuang@tsinghua.edu.cn
Teaching morals is one of the most important purposes of storytelling. An essential ability for understanding and writing moral stories is bridging story plots and implied morals. Its challenges mainly lie in: (1) grasping knowledge about abstract concepts in morals, (2) capturing inter-event discourse relations in stories, and (3) aligning value preferences of stories and morals concerning good or bad behavior. In this paper, we propose two understanding tasks and two generation tasks to assess these abilities of machines. We present STORAL, a new dataset of Chinese and English human-written moral stories. We show the difficulty of the proposed tasks by testing various models with automatic and manual evaluation on STORAL. Furthermore, we present a retrieval-augmented algorithm that effectively exploits related concepts or events in training sets as additional guidance to improve performance on these tasks.
Stories play an essential role in one's moral development (Vitz, 1990). For example, individuals usually learn morals from life experiences or literature such as fables and tell their morals by representing their lived experience in a narrative form (Tappan and Brown, 1989). Accordingly, it is a crucial ability for humans to bridge abstract morals and concrete events in stories. However, this ability has not yet been investigated for machines.
There have been many tasks proposed for evaluating story understanding and generation, including story ending selection (Mostafazadeh et al., 2016) and story generation from short prompts (Fan et al., 2018). Unlike these tasks, which focus on reasoning plots from context, we emphasize the ability to associate plots with implied morals. As exemplified in Table 1, the challenges mainly lie in (1)
Stories: Four cows lived in a forest near a meadow. They were good friends and did everything together. They grazed together and stayed together, because of which no tigers or lions were able to kill them for food.
But one day, the friends fought and each cow went to graze in a different direction. A tiger and a lion saw this and decided that it was the perfect opportunity to kill the cows. They hid in the bushes and surprised the cows and killed them all, one by one.
Morals: Unity is strength.
Table 1: An example in STORAL
grasping knowledge about abstract concepts (e.g., "unity," "strength") and relations among them (e.g., "is") in morals; (2) capturing inter-event discourse relations in stories (e.g., the contrast between endings of the "cows" when they are "united" and "divided"); and (3) aligning value preferences (Jiang et al., 2021) of stories and morals (e.g., the story implies support for "unity", not opposition, which agrees with "is strength" in the moral). To test these abilities of machines, we propose two understanding tasks and two generation tasks. Both understanding tasks require selecting the correct moral from several candidates given a story. And they have respective candidate sets for testing machines in two aspects, including concept understanding (MOCPT for short) and preference alignment (MOPREF for short). The generation tasks require concluding the moral of a story (ST2MO for short), and conversely generating a coherent story to convey a moral (MO2ST for short).
Furthermore, we collected a new dataset named STORAL composed of 4k Chinese and 2k English human-written stories paired with morals through human annotation to address the above challenges. We call the Chinese dataset STORAL-ZH and the English dataset STORAL-EN, respectively. And we construct datasets for the proposed tasks based on STORAL. Our focus of morals is on the social set of standards for good or bad behavior and character, or the quality of being right, honest or accept-
Human Language Technologies, pages 5069 - 5087
able (Ianinska and Garcia-Zamor, 2006). We conduct extensive experiments on the proposed tasks. Furthermore, we present a retrieval-augmented algorithm to improve model performance by retrieving related concepts or events from training sets as additional guidance. However, the experiment results demonstrate that existing models still fall short of understanding and generating moral stories, which requires a better modeling of discourse and commonsense relations among concrete events and abstract concepts<sup>1</sup>.
Story Datasets ROCStories (Mostafazadeh et al., 2016) and WritingPrompts (Fan et al., 2018) are two frequently used story datasets in related studies. The former consists of artificial five-sentence stories regarding everyday events, while the latter contains fictional stories of 1k words paired with short prompts. Besides, some recent works collected extra-long stories such as roleplayerguild (Louis and Sutton, 2018), PG-19 (Rae et al., 2020), and STORIUM (Akoury et al., 2020). Guan et al. (2022) proposed a collection of Chinese stories. These stories usually aim to narrate a coherent event sequence but not convince readers of any morals.
Story Understanding and Generation There have been many tasks proposed for evaluating story understanding and generation. Firstly, some works tested the machinery commonsense reasoning ability regarding inter-event causal and temporal relations through story ending selection (Mostafazadeh et al., 2016), story ending generation (Guan et al., 2019) and story completion (Wang and Wan, 2019). Secondly, a series of studies focused on the coherence of story generation (Fan et al., 2018; Yao et al., 2019; Guan et al., 2020). Another line of works concentrated on controllability to impose specified attributes into story generation. These attributes involved outlines (Rashkin et al., 2020), emotional trajectories (Brahman and Chaturvedi, 2020) and story styles (Kong et al., 2021). Our tasks investigate not only the above aspects but also the ability to understand abstract concepts and reason value preferences of stories.
A task similar to ST2MO is text summarization (Finlayson, 2012) since both tasks require generating a short text to condense crucial information
of a long text. But summarization requires reorganizing a few words of the original text instead of concluding a character-independent moral. For example, a plausible summary of the story in Table 1 is "Four cows were killed by two tigers and a lion" (generated by  $\mathrm{BART}_{\mathrm{Large}}$  (Lewis et al., 2020) finetuned on a summarization dataset XSUM (Narayan et al., 2018)), which includes specific characters and events of the original story. Moreover, MO2ST is similar to persuasive essay generation (Stab and Gurevych, 2017), which also requires conveying a viewpoint in generated texts. However, persuasive essays usually convince readers by directly presenting arguments but not narrating a story.
Morals Haidt and Joseph (2004) provided a theoretical framework named Moral Foundations Theory (MFT) to summarize five basic moral foundations such as "Care/Harm," "Fairness/Cheating," etc. Based on the theory, recent studies have explored to classify the moral foundations of partisan news (Fulgoni et al., 2016), tweets (Johnson and Goldwasser, 2018; Hoover et al., 2020), and crowd-sourced texts (Pavan et al., 2020). And Volkova et al. (2017) proposed identifying suspicious news based on the features of moral foundations. However, we focus on morals which are free-form texts far beyond the scope of the five categories in MFT. In addition, recent studies proposed multiple datasets for machine ethics research such as SBIC (Sap et al., 2020), Social Chemistry (Forbes et al., 2020), Moral Stories (Emelin et al., 2020), ETHICS (Hendrycks et al., 2021) and Scuples (Lourie et al., 2021). But these datasets focus more on how machines behave ethically in some scenario, while STORAL emphasizes the ability to conclude the moral implied by a story. Moreover, most cases in these datasets consist of short texts of descriptive ethical behavior, typically in the form of one sentence. In contrast, STORAL provided longer and more context-specific stories for moral understanding.
3 STORAL Dataset
We collected STORAL from multiple web pages of moral stories. All stories are allowed to use and redistribute for research and have been reviewed by the website editors as stated on the pages. We show the full list of links to these pages in Section A.1. After de-duplication, we collected 19,197 Chinese and 2,598 English raw texts. Then we adopted human annotation for decoupling the story and moral
1All data and evaluation scripts are available at https: //github.com/thu-coai/MoralStory.
Figure 1: The pipeline of human annotation for constructing STORAL (Left) and our constraints (Right).
in each raw text. Due to resource limitations, we only constructed 4,209 Chinese and 1,779 English story-moral pairs. We will first show the details of human annotation, then present the topic analysis and statistics of STORAL, and finally describe the details of dataset construction for the proposed tasks.
3.1 Human Annotation
To narrow down our focus, we define a story as a series of coherent events involving several interrelated characters, and implies support or opposition of some behavior. Such a definition constrains the story to exhibit a moral without any explicit arguments. And we define a moral as a judgment to describe what the story implies concerning good or bad behavior. Note that we do not require morals in STORAL to be always reflective of normatively virtuous behavior. We emphasize that the morals should align with the story. Then, a key issue is how to extract the story and moral from a raw text. We observe that there are no markers such as "The story tells us" to separate the story and moral in most cases. The moral may be tightly weaved into the plot (e.g., included in a dialogue). Therefore, we adopted human annotation for this extraction task. We hired a commercial team to annotate STORAL-ZH. All annotators are native Chinese speakers and well trained for our task. For STORAL-EN, we hired three graduates with good English language proficiency. We did not use AMT since it is inconvenient to train online annotators. Figure 1 shows the annotation pipeline.
We first ask annotators to judge whether the raw text contains a story and moral and whether they meet our constraints shown in Figure 1. We show the examples given to the annotators to inform them of our requirements for stories and morals in Section A.2. If the constraints are not met, we then ask annotators to refine the story and moral.
In the refinement stage, annotators have to clean up the data with following heuristics: (1) refusing examples which may violate general ethical principles (e.g., discrimination); (2) deleting noisy words (e.g., links, codes); (3) refining the stories and morals to be coherent and formal. And to ensure the quality of collected data, annotators may refuse to refine the example if it requires much creative writing. Finally, we review the annotation results and provide detailed feedback to the annotators before approving their submissions. We show an annotation example in Table 2.
Raw Text: A man whowWw.xxx.c0Mlived a long time ago believed that he could read the future in the stars. He called himself an Astrologer, and spent his time at night gazing at the sky. One evening he was walking along the open road outside the village. His eyes were fixed on the stars. He thought he saw there that the end of the world was at hand, when all at once, down he went into a hole full of mud and water. There he stood up to his ears, in the muddy water, and madly clawing at the slippery sides of the hole in his effort to climb out. His cries for help soon brought the villagers running. As they pulled him out of the mud, one of them said: "You pretend to read the future in the stars, and yet you fail to see what is at your feet! This may teach you to pay more attention to what is right in front of you, and let the future take care of itself." "what use is it?" said another, "to read the stars, when you can't see what's right here on the earth?"
Story: A man who lived a long time ago believed that he could read ... As they pulled him out of the mud, one of them said: "You pretend to read the future in the stars, and yet you fail to see what is at your feet!"
Moral: Pay more attention to what is right in front of you, and let the future take care of itself.
Table 2: An example for extracting the story and moral from a raw text. We highlight the words which should be revised in the raw text in italic. And the moral in the raw text is bold. To save space, we replace some events with “…” in the story.
3.2 Topic Analysis
To provide insight into the taxonomy of morals within STORAL, we adopt LDA (Blei et al., 2003) for topic modeling of morals. Let  $B$  denote the number of topics and  $V$  denote the vocabulary size. Based on the variational parameter for topic word distribution  $\beta \in \mathbb{R}^{B \times V}$ , we determine  $B$  as the minimum value that makes the following formula
Topic Words	Examples
懂得(understand),也是(also),了解(know),方法(method),收 获(gain),保护(protect),大脑(brain),才能(able),付出(pay), 进步(progress)	在犯错的时候我们要懂得看全局,要了解全局才能对事情有定义。(When making mistakes, we must understand the overall situation. And we are able to have a definition of things only when knowing the overall situation.)
不要(not),一定要(must),危险(danger),时候(when), 对待(treat),安全(safety),千万(any way),好好(well),学 会(learn),遇到(encounter)	生活中也要牢记“安全”这两字,在“安全”两字面前切不可存有侥幸心理, 把安全当成儿戏。(Keep in mind the word “safety” in your life, and do not take any chances to treat safety as a joke.)
事情(thing),才能(able),做好(do well),优秀(excellent), 应该(should),做到(achieve),自信(self-confident),有 所(somewhat),无法(unable),可能(may)	做好自己该做的事情,做自己的主人。(Do what you should do and be your own mas- ter.)
时候(when),其实(actually),很多(many),发现(discover), 希望(wish),发生(happen),生活(life),已经(already),伤 害(hurt),可能(may)	人要善于自己发现自己,而不是老等着别人来发现我们。(We should be good at discovering ourselves instead of waiting for others to do.)
遇到(encounter),问题(question),困难(difficulty),解 决(solve)思考(think),帮助/help),时候(when),应 该(should),给予(give),头脑(brain)	乐于助人,是一种朴实的中国传统美德。每个人都有遇到困难的时候,最需要的是别人给予的帮助。(Being helpful is a Chinese traditional virtue. When someone encounters difficulties, what he needs most is help from others.)
good,always,come,believe,first,honesty,speak,world,around, act	1. Always be honest. Honesty is always rewarded. 2. A liar will not be believed, even when he speaks the truth.
help,also,good,need,hope,lose,carry,feel,say,self	1. One should not be carried away by what others say. Don’t be fooled by those who wants to take advantage of you. 2. Self help is the best help. Heaven helps those who help themselves.
friend,act,wisely,moment,think,place,time,choose,great, ability	1. Little friends may prove great friends. 2. One should not panic in difficult times and think wisely.
love,care,parent,respect,always,value,take,mean,give,one	1. You reap what you sow. Regardless of your relationship with your parents, you’ll miss them when they’re gone from your life. Always respect, care for and love them. 2. Be content with your lot; one cannot be first in everything.
look,see,bad,make,turn,strong,strength,choice,give,deserve	1. The strong and the weak cannot keep company. 2. It is easy to despise what you cannot get.
Table 3: Topic words and examples for STORAL-ZH (top) and STORAL-EN (bottom). We underline the topic words that occurs in the examples.
holds true for any  $b \in \{1, 2, \dots, B\}$ :
$$
s _ {b} = \frac {\sum_ {v \in \mathcal {V} _ {b} ^ {(k)}} \beta_ {b v}}{\sum_ {v = 1} ^ {V} \beta_ {b v}} \geq h,
$$
$$
\mathcal {V} _ {b} ^ {(k)} = \operatorname {a r g m a x} _ {\mathcal {V} _ {b} ^ {* (k)}} \sum_ {v \in \mathcal {V} _ {b} ^ {* (k)}} \beta_ {b v},
$$
where  $\beta_{bv}$  is the element at the  $b$ -th row and  $v$ -th column of  $\beta$ ,  $k \in \{1,2,\dots,V\}$  is the size of the top- $k$  vocabulary  $\mathcal{V}_b^{(k)}$ , and  $h \in [0,1]$  is a predefined threshold.  $s_b$  is used to measure the specificity of the  $b$ -th topic. Intuitively, the larger  $s_b$ , the more specific the topic. We set  $k$  to 20 and  $h$  to 0.5. Finally, we derive 40/24 topics for STORAL-ZH/STORAL-EN, respectively. And the minimum proportion of examples of one topic is  $1.6\% /3.2\%$  for STORAL-ZH/STORAL-EN, respectively.
Table 3 shows the topic words in  $\nu^{(10)}$  of each topic and two morals assigned to each topic with the highest probabilities for the five topics with the largest specificity scores. The topics cover diverse situations ranging from facing others ("honesty," "help"), parents ("love"), ourselves ("self-help," "self-discovery") to facing difficulties ("think") and
danger ("safety"). And examples of the same topic present related semantics to some extent, such as "being honest" and "not believing liars" for the first topic in STORAL-EN. We also show the analysis of high-frequency words of stories and morals in Section A.3 and discussion about the commonsense and discourse relations in stories in Section A.4.
3.3 Dataset Statistics of STORAL
Table 4 shows the statistics of STORAL. We regard the unlabeled data which contain entangled stories and morals as an in-domain resource for research on unsupervised or semi-supervised learning for the proposed tasks. And the data are also suitable for learning to generate morals stories where the morals are weaved naturally into the story plots.
3.4 Task-Specific Dataset Construction
Based on STORAL, we build task-specific datasets for our understanding tasks (MOCPT and MOPREF) and generation tasks (ST2MO and MO2ST). We randomly split the labeled data in STORAL-ZH and STORAL-EN for training/validation/testing by 8:1:1 and 3:1:1, respectively. Table 5 shows the task
Datasets	Labeled Data	Unlabeled Data
# Examples	Stories	Morals	# Examples	# Word	# Sent	Vocab
# Word	# Sent	Vocab	# Word	# Sent	Vocab
STORAL-ZH	4,209	321.75	17.62	63,493	25.09	1.35	10,522	14,988	487.00	26.12	147,805
STORAL-EN	1,779	302.33	17.71	15,873	19.77	1.45	3,384	819	614.55	38.05	20,853
Table 4: STORAL statistics. We use  $\mathrm{Jieba}^2/\mathrm{NLTK}$  (Loper and Bird, 2002) for word tokenization of STORAL-ZH/STORAL-EN. #Word/#Sent is the average number of words/sentences. Vocab is the vocabulary size.
Tasks	Abilities	Inputs & Outputs	STORAL-ZH |Train|Vall|Test|	STORAL-EN |Train|Vall|Test|
MOCPT	Concept Understanding	Given a story and five candidate morals, choosing the correct moral.	3,368 / 420 / 421	1,068 / 355 / 356
MOPREF	Preference Alignment	Given a story and two candidate morals, choosing the correct moral.	3,276 / 410 / 411	988 / 344 / 339
ST2MO	Moral Generation	Given a story, generating a moral which is character-independent and generally applicable.	3,368 / 420 / 421	1,068 / 355 / 356
MO2ST	Story Generation	Given a moral and a story beginning and outline, generating a story which has a coherent plot and convinces readers of the moral.	3,368 / 420 / 421	1,068 / 355 / 356
Table 5: Description of the proposed tasks about the abilities they investigate, inputs and outputs, and the data sizes.
descriptions and data sizes.
MOCPT It requires selecting the correct moral from five candidates given a story. We constructed the dataset by taking the original moral as the correct candidate and four negatively sampled morals as incorrect candidates for each example. To avoid more than one plausible candidate, we ensured that the negative morals are assigned to different topics from the original one by the LDA model (Section 3.2). In this way, MOCPT can effectively test the ability to distinguish different concepts.
MOPREF It requires selecting the correct moral from two candidates. Its difference from MOCPT is that we created the incorrect candidate by substituting one random token in the original moral to its antonym. For example, the moral "unity is strength" can be transformed to "unity is weakness". We perform the transformation using a rule-based method (Ribeiro et al., 2020). Because there exist examples where no words have antonyms, the number of examples for MOPREF are a little fewer than MOCPT. MOPREF will serve for testing the ability to capture the value preference of stories.
ST2MO It requires generating the moral of a given story. We regard the original story as input and the original moral as target output.
MO2ST It requires generating a story to convey a given moral. Unfortunately, automatic evaluation for open-ended story generation is still highly challenging due to the notorious one-to-many issue (Zhao et al., 2017): There may be multiple
plausible stories with the same moral. For example, the moral in Table 1 can also be conveyed by another story: “bees unite to build their beehives.” Such openness makes automatic metrics unreliable for quality evaluation (Guan and Huang, 2020).
To alleviate this issue, we extract the first sentence and an outline from a target story, and pair them with the moral as input for generating the story. We follow Rashkin et al. (2020) to extract a set of at most eight phrases from a story through RAKE (Rose et al., 2010) as the outline. We set the maximum number of words in each phrase to eight. We also filtered those phrases that are substrings of others. For example, the outline for the story in Table 1 is {"lions," "friends fought," "good friends," "grazed," "perfect opportunity)}. Finally, for STORAL-ZH/STORAL-EN, the average number of phrases for each example is 7.5/6.8 and the average number of words in each phrase is 2.87/2.44, respectively.
4 Retrieval Augmentation
A critical challenge for tackling the proposed tasks is the sparsity of morals and events makes it difficult to learn relations between them. Prior studies have shown that retrieval improves performance towards infrequent data points across various tasks such as open-domain question answering (Chen et al., 2017) and text classification (Lin et al., 2021). We present a retrieval-augmented algorithm that exploits the moral-event relations in training sets. We illustrate our model for the MOPREF task in Fig-
Figure 2: Model overview for the MOPREF task.
ure 2. Our models for other tasks are similar.
For both MOCPT and MOPREF, we encode the story and candidates using an input encoder, and then predict a probability distribution over the candidates by normalizing the dot-product scores between the representations of the story and each candidate. We optimize the model by minimizing the cross-entropy loss. We insert special tokens [S] and [C] before the story and each candidate, respectively, and take the corresponding hidden states as their representations. Furthermore, we propose to retrieve related concepts from the training set using the input story. We encode the story using a query encoder, then take the output as the query to retrieve  $m$  most related stories based on a story index, i.e., a set of dense vectors as the representations of stories in the training set. We adopt BERT (Devlin et al., 2019) followed by a mean-pooling layer to build the query encoder and story index, which are frozen in the training stage. Finally, we extract the nouns, verbs, adjectives and adverbs from the morals of the top- $m$  stories and lemmatize them as the retrieved concepts. We feed the concepts together with the original input to the input encoder. For example, the retrieved concepts for the story in Table 1 include "support" and "strength", which may serve as additional guidance for models' prediction.
The retrieval-augmented algorithm can easily adapt to the generation tasks. For ST2MO, we take the input story paired with the retrieved concepts into the encoder and then generate the output using the decoder. And for MO2ST, we use the input moral as the query to retrieve top- $m$  stories, and regard their outlines as the retrieved additional information to guide the subsequent story generation.
We evaluated the following baselines for the understanding tasks: BERT (Devlin et al., 2019),