SlowGuess/NAACL_2022_OCR huggingface dataset:
RoBERTa (Liu et al., 2019) and T5 (Raffel et al., 2020). When evaluating T5, we feed the input to both the encoder and decoder of T5 and optimize the model using the cross-entropy loss. To investigate potential biases of the proposed datasets, we added a baseline called BERT w/o story, which is fine-tuned to make prediction without taking the story as input. For the generation tasks, we evaluated ConvS2S (Gehring et al., 2017), Fusion (Fan et al., 2018), GPT2 (Radford et al., 2019) and T5, which are trained or fine-tuned with the standard language modeling objective. Moreover, we also evaluate a task-specific model PlotMachines (PM for short) (Rashkin et al., 2020), which is proposed for tackling outline-conditioned generation by tracking the dynamic plot states. We use GPT2 as the backbone model of PM.
We also design models to test the adaption of the unlabeled data of STORAL to the proposed tasks. Specifically, we first post-train RoBERTa and T5 on the unlabeled data with their original pretraining objectives, respectively (i.e., masked language model and text infilling) and then fine-tune them on the labeled data for the downstream tasks (Gururangan et al., 2020). We call the baselines RoBERTa-Post and T5-Post. We perform our retrieval-augmented algorithm based on the post-trained models, called RA-RoBERTa and RA-T5, respectively.
We implement the pretrained models based on the codes and pretrained checkpoints of Hugging-Face's Transformers (Wolf et al., 2020). We use LongLM<sub>base</sub> (Guan et al., 2022) as the T5 model for experiments on STORAL-ZH, and set all pretrained models to the base version due to limited computational resources. As for the hyperparameters, we set the batch size to 16, the maximum sequence length to 1,024, the learning rate to 3e-5,  $m$  to 10 for our retrieval-augmented model. We generate outputs using top- $k$  sampling (Fan et al., 2018) with  $k = 40$  and a softmax temperature of 0.7 (Goodfellow et al., 2016). We show more details in Section B.1.
5.3 Automatic Evaluation
Evaluation Metrics We adopt accuracy to evaluate the understanding tasks. For generation tasks, we do not use perplexity since perplexity scores are not comparable among models with different vocabularies. We adopt the following metrics for automatic evaluation: (1) BLEU  $(\mathbf{B} - n)$ : It is used
to measure  $n$ -gram overlaps ( $n = 1,2$ ) between generated and ground-truth texts (Papineni et al., 2002). (2) BERTScore-F1 (BS): It is used to measure the semantic similarity between generated and ground-truth texts (Zhang et al., 2019). (3) Repetition (R-n): It calculates the ratio of texts that repeat at least one  $n$ -gram in all generated texts (Shao et al., 2019). (4) Distinct (D-n): It measures the diversity using the percentage of distinct  $n$ -grams to all  $n$ -grams in generated texts (Li et al., 2016). For both R-n and D-n, we set  $n = 2$  for ST2MO and  $n = 4$  for MO2ST considering the much shorter length of morals than stories. Besides, we also report the average number of generated words (Len).
We also adopt the following metrics for automatic evaluation of MO2ST: (1) Coverage (Cov): It computes Rouge-L recall (Lin, 2004) between generated stories and phrases in the corresponding outlines. A higher score means the generated stories cover more phrases in the given outlines. (2) Order (Ord): It measures the disparity between the positional orders of given phrases in the ground truth and generated story using the percentage of inversions in the generated story (Guan et al., 2022). An inversion is a position pair of two phrases that is out of the ground-truth order. Higher order scores mean that the stories arrange the outline more reasonably. In Section B.2, we also construct a learnable automatic metric to measure the faithfulness between morals and stories.
Results Table 6 and 7 show the results on the understanding and generation tasks, respectively. To get the human performance on MOCPT and MO-PREF, we sampled 100 examples from the test set and recruited three annotators with good Chinese or English language proficiency to complete these tasks. We made final decisions among the annotators through major voting. The annotation results show an almost perfect agreement with Fleiss's  $\kappa > 0.85$  (Fleiss and Joseph, 1971).
We summarize the results on the understanding tasks as follows: (1) The MOPREF datasets suffer from innate biases as indicated by the high accuracy of BERT w/o story. Such biases may result from the noise introduced by the automatic construction technique, i.e., antonym substitution. And models may learn patterns of good behavior (e.g., "unity" is good and "disunity" is bad in general) and make predictions easily without depending on stories. However, MOPREF is still meaningful as an evaluation task since BERT can achieve much
Models	#P	MOCPT	MOPREF
ZH	EN	ZH	EN
Random	N/A	20.19	20.22	50.12	50.00
BERT w/o Story	110M	23.52	22.47	71.81	72.57
BERT	110M	59.62	51.97	82.97	79.35
RoBERTa	110M	62.71	54.78	89.54	81.12
RoBERTa-Post	110M	64.61	51.40	87.59	81.42
T5	220M	69.60	58.99	82.00	76.99
T5-Post	220M	70.07	62.64	81.75	77.29
RA-RoBERTa	110M	65.08	60.96	90.02	81.71
RA-T5	220M	72.68*	67.42**	82.97	82.60
Human	N/A	95.00	96.00	98.00	99.00
Table 6: Accuracy  $(\%)$  for MOCPT and MOPREF. # P is the number of parameters. The best performance is highlighted in bold and the second best is underlined. The scores marked with \* and \*\* of RA model mean it outperforms the best baseline significantly with p-value  $<  0.1$  and p-value  $<  0.05$  (sign test), respectively.
better accuracy when taking stories as input. And we experiment using manually constructed examples for evaluating preference alignment in the appendix. (2) T5 performs better than RoBERTa on MOCPT but worse on MOPREF, indicating T5 may not be good at capturing value preferences. (3) Post-training on the unlabeled data (i.e., RoBERTa-Post and T5-Post) does not always bring improvement on both tasks, suggesting that it is necessary to develop a better way to exploit these data in future work. (4) Retrieving additional concepts improves models' performance effectively, particularly for the MOCPT task on STORAL-EN. However, there is still a big gap between our models and human performance.
As for the generation tasks, we draw the following conclusions: (1) Almost all pretrained models achieve better lexical and semantic similarity with ground-truth texts than non-pretrained models, as indicated by higher BLEU and BERTScore values. (2) Non-pretrained models have less repetition than pretrained ones, and repeat even less than the ground-truth texts when generating morals. It may be because non-pretrained models generate shorter sequences than pretrained models despite the same decoding algorithm, which also accounts for the higher distinct scores of the non-pretrained models on the MO2ST task. (3) When generating stories, T5-Post can cover more input phrases and arrange them in a correct order than other baselines, as indicated by higher coverage and order scores. (4) Retrieval augmentation can improve the generation similarity with the ground-truth texts on both tasks and improve the coverage and order scores on ST2MO significantly compared with T5-Post.
Models	#P	Dataset: STORAL-ZH	Dataset: STORAL-EN
B-1↑	B-2↑	BS↑	R-2↓	D-2↑	Len	B-1↑	B-2↑	BS↑	R-2↓	D-2↑	Len
ConvS2S	50M	14.31	1.86	56.71	26.60	43.67	19.31	9.69	0.93	82.57	6.46	47.35	11.75
Fusion	100M	14.78	2.23	56.90	27.55	41.21	21.96	9.87	0.82	82.68	6.18	43.59	13.15
GPT2	124M	14.54	2.16	60.75	35.39	48.22	20.72	10.98	1.24	79.39	20.22	60.36	16.19
T5	220M	18.19	3.60	61.61	76.48	44.84	29.06	13.31	2.26	85.89	33.15	58.73	19.39
T5-Post	220M	17.98	3.91	61.52	69.12	51.97	29.14	13.83	2.11	85.85	34.83	57.12	18.49
RA-T5	220M	18.32	3.64	61.93**	70.78	48.14	29.44	14.59	2.61	86.16**	31.46	60.61	18.54
Truth Morals	N/A	N/A	N/A	N/A	29.22	73.70	25.09	N/A	N/A	N/A	16.85	73.95	20.41
Models	Dataset: STORAL-ZH	Dataset: STORAL-EN
B-1↑	B-2↑	BS↑	R-4↓	D-4↑	Cov↑	Ord↑	Len	B-1↑	B-2↑	BS↑	R-4↓	D-4↑	Cov↑	Ord↑	Len
ConvS2S	15.57	6.43	60.00	75.30	78.41	21.61	33.03	150	16.25	6.38	79.27	61.85	80.29	6.46	41.88	122
Fusion	15.53	6.45	60.06	74.11	80.51	22.86	33.33	148	17.17	6.82	79.52	61.24	75.79	7.27	43.07	137
GPT2	14.91	6.48	63.32	91.45	58.67	48.57	51.58	282	25.83	12.91	83.25	84.27	74.63	45.18	59.95	247
PM	15.82	7.04	63.58	90.97	57.33	50.51	52.35	280	26.34	13.92	81.63	80.90	72.64	47.07	60.31	264
T5	17.74	9.44	65.89	91.69	61.76	58.18	56.11	166	30.56	16.75	79.89	90.17	77.53	74.21	63.45	283
T5-Post	18.42	9.77	65.63	94.54	58.13	60.11	56.96	176	32.36	18.04	83.80	94.10	77.27	76.09	64.33	281
RA-T5	23.36**	12.98**	67.37**	95.72	59.49	69.24**	60.44**	241	32.46*	18.31**	84.07**	92.42	76.74**	80.21**	66.10**	253
Truth	N/A	N/A	N/A	55.34	96.06	100.00	100.00	324	N/A	N/A	N/A	58.71	95.09	100.00	100.00	281
Table 7: Automatic evaluation results for ST2MO (Top) and MO2ST (Bottom).  $\downarrow / \uparrow$  means the lower/higher the better. All scores except Len are multiplied by 100. The best result is in bold and the second best is underlined. The scores marked with * and ** of RA-T5 mean it outperforms the best baseline significantly with p-value  $< 0.1$  and p-value  $< 0.05$  (sign test), respectively.
Data	Task	Model	Flu (κ)	Cohc (κ)	Faith (κ)
STORAL-ZH	ST2MO	Fusion	0.24 (0.31)	0.22 (0.37)	0.08 (0.72)
T5	0.75 (0.40)	0.61 (0.38)	0.31 (0.32)
RA-T5	0.85 (0.65)	0.63 (0.26)	0.36 (0.27)
Truth	1.00 (1.00)	0.99 (0.96)	0.86 (0.57)
MO2ST	Fusion	0.25 (0.39)	0.11 (0.61)	0.02 (0.93)
T5	0.38 (0.40)	0.24 (0.37)	0.05 (0.81)
RA-T5	0.45 (0.29)	0.34 (0.25)	0.11 (0.72)
Truth	0.98 (0.93)	1.00 (1.00)	0.96 (0.84)
STORAL-EN	ST2MO	Fusion	0.32 (0.39)	0.26 (0.35)	0.24 (0.41)
T5	0.76 (0.35)	0.74 (0.27)	0.55 (0.33)
RA-T5	0.81 (0.51)	0.79 (0.40)	0.67 (0.37)
Truth	0.94 (0.80)	0.94 (0.77)	0.88 (0.56)
MO2ST	Fusion	0.47 (0.43)	0.40 (0.47)	0.37 (0.45)
T5	0.56 (0.35)	0.48 (0.37)	0.49 (0.39)
RA-T5	0.58 (0.28)	0.51 (0.31)	0.57 (0.31)
Truth	0.95 (0.69)	0.98 (0.69)	0.93 (0.53)
Table 8: Manual evaluation results for ST2MO and MO2ST. Flu, Cohe and Faith mean fluency, coherence and moral faithfulness, respectively. The best results are highlighted in bold. All results show fair or moderate inter-annotator agreement measured by Fleiss'  $\kappa$  (Fleiss and Joseph, 1971).
5.4 Manual Evaluation
On the generation tasks, we conducted a Likert-scale based manual evaluation to measure the gap between existing models and humans. For STOAL-ZH, we hired three graduate students (native Chinese speakers) as annotators. We conducted evaluation on STORAL-EN using Amazon Mechanical Turk (AMT). For both tasks, we randomly sampled 100 examples from the test set, and obtained 300 generated texts from Fusion, T5 and RA-T5. For each text we require three annotators to rate its qual
ity along with the input using a binary score in three following aspects: (1) linguistic fluency: correctness in grammaticality; (2) coherence: reasonable relations between sentences regarding relatedness, causality and temporal orders; and (3) moral faithfulness: exhibition of a faithful moral to the input. Three aspects are independently evaluated. We decided the final score of a text through majority voting. The annotation instruction is shown in Section B.3.
Table 8 shows the manual evaluation results. We show  $p$ -values of the results in Section B.4. For ST2MO, T5 achieves a substantial improvement compared with Fusion ( $p < 0.01$ ), and our model further outperforms T5. The superiority becomes less significant for MO2ST. However, the big gap between these models and humans, particularly in terms of faithfulness, proves both tasks challenging for existing models. Furthermore, we evaluate whether machines can capture the value preference of a story using manually constructed examples. And we show error analysis and case study for the proposed tasks in Section C. We believe that explicit modeling of the relations among events and abstract concepts will further promote progress on these tasks, which we regard as future work.
We present STORAL, a collection of Chinese and English moral stories. To test the ability to bridge
concrete events and abstract morals, we propose new understanding and generation tasks based on STORAL, including selecting the correct moral from several candidates with different topics or opposite value preferences, concluding the moral of a story and generating a story to convey a moral. Extensive experiments prove these tasks still to be challenging for existing models. We propose a retrieval-augmented algorithm to improve performance by retrieving related concepts or events from training sets. Although it is possible to further increase the dataset size, we expect to make meaningful progress by developing better representations of commonsense and discourse relations among events and abstract concepts in future work.
This work was supported by the National Science Foundation for Distinguished Young Scholars (with No. 62125604) and the NSFC projects (Key project with No. 61936010 and regular project with No. 61876096). This work was also supported by the Guoqiang Institute of Tsinghua University, with Grant No. 2019GQG1 and 2020GQG0005. This work was also sponsored by Tsinghua-Toyota Joint Research Fund. We would also like to thank the anonymous reviewers for their invaluable suggestions and feedback.
8 Ethics Statements and Broader Impact
We collected STORAL from public web resources. All stories are under licenses that allow use and redistribution for research purposes. We asked commercial annotation teams to extract stories and morals from the crawled raw texts. We required the annotators to refuse the examples which violate general ethical principles (e.g., showing discrimination for someone, containing disrespectful content, or encouraging to disturb public order, etc.). Totally, we paid more than  $7 (CNY 45) per hour on average for annotating each example in STORAL, which was far beyond the minimum hourly wage in China (CNY 21). Furthermore, we resorted to AMT for manual evaluation of generated and human-written texts for two proposed generation tasks. We hired three annotators and paid each annotator$ 0.2 on average for annotating each example.
In this paper, we emphasize the ability to model relations between concrete events and abstract morals, which is also helpful for various scenar-
ios such as reading comprehension (e.g., drawing authors' viewpoints from narratives) and essay writing (e.g., writing essays to convince readers of some arguments by presenting examples or anecdotes). STORAL provides a good start point for exploring these directions.
Nader Akoury, Shufan Wang, Josh Whiting, Stephen Hood, Nanyun Peng, and Mohit Iyyer. 2020. STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6470–6484, Online. Association for Computational Linguistics.
David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. the Journal of machine Learning research, 3:993-1022.
Faeze Brahman and Snigdha Chaturvedi. 2020. Modeling protagonist emotions for emotion-aware storytelling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5277-5294, Online. Association for Computational Linguistics.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870-1879, Vancouver, Canada. Association for Computational Linguistics.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2020. Revisiting pretrained models for Chinese natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 657-668, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Denis Emelin, Ronan Le Bras, Jena D Hwang, Maxwell Forbes, and Yejin Choi. 2020. Moral stories: Situated reasoning about norms, intents, actions, and their consequences. arXiv preprint arXiv:2012.15738.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers), pages 889-898.
Mark Mark Alan Finlayson. 2012. Learning narrative structure from annotated folktales. Ph.D. thesis, Massachusetts Institute of Technology.
Fleiss and L. Joseph. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378-382.
Maxwell Forbes, Jena D. Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. 2020. Social chemistry 101: Learning to reason about social and moral norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 653-670, Online. Association for Computational Linguistics.
Dean Fulgoni, Jordan Carpenter, Lyle Ungar, and Daniel Preoticiuc-Pietro. 2016. An empirical exploration of moral foundations theory in partisan news sources. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 3730-3736.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In International Conference on Machine Learning, pages 1243-1252. PMLR.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep learning. MIT press.
Jian Guan, Zhuoer Feng, Yamei Chen, Ruilin He, Xiaoxi Mao, Changjie Fan, and Minlie Huang. 2022. LOT: A Story-Centric Benchmark for Evaluating Chinese Long Text Understanding and Generation. Transactions of the Association for Computational Linguistics, 10:434-451.
Jian Guan, Fei Huang, Zhihao Zhao, Xiaoyan Zhu, and Minlie Huang. 2020. A knowledge-enhanced pretraining model for commonsense story generation. Transactions of the Association for Computational Linguistics, 8:93-108.
Jian Guan and Minlie Huang. 2020. _UNION: an un-referenced metric for evaluating open-ended story generation_. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_, EMNLP 2020, Online, November 16-20, 2020, pages 9157-9166. Association for Computational Linguistics.
Jian Guan, Yansen Wang, and Minlie Huang. 2019. Story ending generation with incremental encoding and commonsense knowledge. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6473-6480.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360.
Jonathan Haidt and Craig Joseph. 2004. Intuitive ethics: How innately prepared intuitions generate culturally variable virtues. *Daedalus*, 133(4):55-66.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021. Aligning {ai} with shared human values. In International Conference on Learning Representations.
Joe Hoover, Gwenyth Portillo-Wightman, Leigh Yeh, Shreya Havaldar, Aida Mostafazadeh Davani, Ying Lin, Brendan Kennedy, Mohammad Atari, Zahra Kamel, Madelyn Mendlen, Gabriela Moreno, Christina Park, Tingyee E. Chang, Jenna Chin, Christian Leong, Jun Yen Leung, Arineh Mirinjian, and Morteza Dehghani. 2020. Moral foundations twitter corpus: A collection of 35k tweets annotated for moral sentiment. Social Psychological and Personality Science, 11(8):1057-1071.
Silvana Ianinska and Jean-Claude Garcia-Zamor. 2006. Morals, ethics, and integrity: How codes of conduct contribute to ethical adult education practice. *Public Organization Review*, 6(1):3-20.
Liwei Jiang, Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Maxwell Forbes, Jon Borchardt, Jenny Liang, Oren Etzioni, Maarten Sap, and Yejin Choi. 2021. Delphi: Towards machine ethics and norms. arXiv preprint arXiv:2110.07574.
Kristen Johnson and Dan Goldwasser. 2018. Classification of moral foundations in microblog political discourse. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 720-730.
Xiangzhe Kong, Jialiang Huang, Ziquan Tung, Jian Guan, and Minlie Huang. 2021. Stylized story generation with style-guided planning. In *Findings of the Association for Computational Linguistics: ACLIJCNLP* 2021, pages 2430-2436, Online. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7871-7880. Association for Computational Linguistics.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and William B Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North
American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Yuxiao Lin, Yuxian Meng, Xiaofei Sun, Qinghong Han, Kun Kuang, Jiwei Li, and Fei Wu. 2021. Bertgcn: Transductive text classification by combining gcn and bert. arXiv preprint arXiv:2105.05727.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Edward Loper and Steven Bird. 2002. Nltk: The natural language toolkit. In Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics, pages 63-70.
Annie Louis and Charles Sutton. 2018. Deep dungeons and dragons: Learning character-action interactions from role-playing game transcripts. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 708-713.
Nicholas Lourie, Ronan Le Bras, and Yejin Choi. 2021. Scruples: A corpus of community ethical judgments on 32,000 real-life anecdotes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13470-13479.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of NAACL-HLT, pages 839-849.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318.
Matheus C Pavan, Vitor G Dos Santos, Alex GJ Lan, Joao Martins, Wesley R Santos, Caio Deutsch, Pablo B Costa, Fernando C Hsieh, and Ivandre Paraboni. 2020. Morality classification in natural language text. IEEE Transactions on Affective Computing.