SlowGuess/NAACL_2021_OCR huggingface dataset:
The effectiveness of large pretrained language models has been proven in NLP downstream tasks (Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019; Beltagy et al., 2020). RoBERTa introduced by Liu et al. (2019) is one of these models achieving impressive performances on text classification. We employ RoBERTa for our plausibility classification task. We start from pretrained RoBERTa parameters and fine-tune them on the constructed evaluation dataset to predict plausibility scores.
One of the main limitations of RoBERTa is its length requirement of at most 512 tokens. Recently, this limitation was addressed by considering a sparser set of attention mechanisms such as locality-sensitive hashing and sliding window attentions, which reduce the computation complexity
from  $\mathcal{O}(n^2)$  to  $\mathcal{O}(n\log n)$  and  $\mathcal{O}(n)$  respectively (Kitaev et al., 2020; Beltagy et al., 2020). In this work, we broaden the scope of the text plausibility evaluation to cover not only short but also long texts with more than 512 tokens. To this end, we examine and evaluate the quality of long texts using Longformer (Beltagy et al., 2020) that has linear complexity in terms of the number of tokens in a text. We fine-tune the pretrained Longformer for long text plausibility evaluation.
We benchmark both fine-tuned classifiers on the manipulated data with the two following baselines. UNION. Recently, Guan and Huang (2020) proposed an automatic evaluation metric by training a BERT model (Devlin et al., 2019) with an auxiliary reconstruction objective which helps to recover the perturbation from a negative sample. The proposed model is trained on negative implausible texts constructed by adopting repetition, substitution, reordering, and negation sampling techniques. This model and its proposed approach for data construction were compared with previously proposed methods and shown to be more efficient.
SENTAVG. We complete our investigation by selecting SENTAVG (Lai and Tetreault, 2018) as another baseline model for the plausibility evaluation task. SENTAVG leverages LSTM to get sentence representation from their words GloVe embeddings. All the sentences vectors are averaged to form the representation for the whole text and this vector is passed to a hidden layer. A softmax layer at the end computes the probability distribution of texts over positive and negative labels.
We investigate the effectiveness of our proposed approach versus heuristic negative sampling techniques by focusing on the evaluation of open-domain story generation models in two datasets with short and long stories. We show the generalizability of metrics trained on our proposed plot manipulation data. We also separately assess the impact of each manipulation technique on the metric accuracy.
We conduct our experiments on two English stories datasets that are significantly different in terms of length and topic; ROCStories (shortly ROC) and
Dataset	Train/Valid/Test
HEUR_SENT_SUB	47.1k/5.9k/5.9k
HEUR_KEY_SUB	47.1k/5.9k/5.9k
UNION_DATA	47.1k/5.9k/5.9k
MANPLTS	47.1k/5.9k/5.9k
AF_MANPLTS	94.2k/11.8k/11.8k
Writing Prompt (briefly WP) datasets including on average 49.4 and 734.5 tokens in each story.
ROCStories. ROCStories is a resource of five-sentence commonsense stories collected via crowdsourcing (Mostafazadeh et al., 2016) covering a logically linked set of daily events. We follow the approach proposed by Yao et al. (2019) to extract story plots (storylines) for the stories and manipulate them to guide conditional language models to generate negative samples.
Writing Prompt. Writing Prompt dataset contains abstract high-level prompts and their corresponding long human-written stories from an online forum (Fan et al., 2018). To apply the plot manipulation technique for implausible text construction, we follow the procedure proposed by Fan et al. (2019) to extract the plots with verb and argument type role labeling tags.
Data Preparation. We split the stories from both datasets into two subsets for training generation and evaluation models, respectively. We use 70 percent of stories in ROC (ROC_LM) and WP (WP_LM) for fine-tuning GPT2 (Radford et al., 2019) language model with batch size of 4. After 3 epochs of fine-tuning, the perplexity on the validation set of ROC and WP datasets are 8.28 and 25.04, respectively.
The remaining 30 percent of stories from ROC (ROC_Eval) and WP (WP_Eval) are used for training and evaluating the evaluation models. All stories in the original dataset represent plausible texts. We apply approaches from Section 3 to augment negative samples. Table 2 and Table 3 summarize the resulting datasets for ROC and WP. In HEUR_SENT_SUB, we extract all stories with at least 2 sentences and replace  $50\%$  of their sentences with random ones. For HEUR_KEY_SUB, we do random substitution of  $15\%$  of keywords with their corresponding antonyms extracted from ConceptNet and ignore stories without substitutable keywords. The UNION_Data is resulted by following rules from Guan and Huang (2020) and is applied
Table 2: Plausibility evaluation datasets for ROC stories using different negative sampling techniques.
Dataset	Train/Valid/Test
HEUR_SENT_SUB	163.1k/9.3k/9.1k
HEUR_KEY_SUB	162.7k/9.3k/9.0k
UNION_DATA	161.8k/9.2k/9.0k
MANPLTS	84.5k/4.7k/4.7k
AF_MANPLTS	107.2k/35.7k/35.7k
Table 3: Plausibility evaluation datasets for WP stories using different negative sampling techniques.
Data	Texts	Annotators	Kappa
ROC	300	27	0.61
WP	300	75	0.56
Table 4: Statistics and inter-annotator agreement of AMT annotations for plausibility metrics evaluation.
to stories with at least four sentences.
To create MANPLTS dataset, we first fine-tune the BART model (Lewis et al., 2019) with a batch size of 8 for three epochs on pairs of ground-truth plots and stories from ROC_LM and WP_LM data with the resulting perplexity of 3.44 and 6.79 for the validation sets. Afterward,  $15\%$  of plots are employed and two up to four proposed manipulation techniques in Section 3.2 are randomly selected and applied. We leverage the fine-tuned BART model and use the top-50 sampling technique with a temperature of 0.8. We specify the maximum length of 200 for ROC dataset and 1024 for WP dataset to generate implausible texts on manipulated plots.
In the AF_MANPLTS dataset, we apply the adversarial filtering technique on top of six generated implausible stories using the fine-tuned BART model conditioned on the manipulated plots. The output contains each human-written story and its three most challenging implausible samples.
5.2 Human Annotations
The performance of automatic evaluation metrics is assessed based on their correlations with human judgments. To this end, we gather human evaluations and examine the Spearman  $(\rho)$  and Kendall  $(\tau)$  correlations with metrics predicted scores (Newman et al., 2010; Lai and Tetreault, 2018; Guan and Huang, 2020). Spearman and Kendall are beneficial in estimating monotonic associations for not normally distributed and ranked scores.
We collect human judgments through Amazon Mechanical Turk (AMT) experiments. We randomly choose 150 human-written stories from ROC_Eval and WP_Eval test sets and 150 machine-generated texts by the fine-tuned GPT2 models. Five distinct participants are asked to rate each story on a scale of 0 to 5 (from not at all plausible
2We fine-tune GPT2 language model using https://github.com/huggingface/transformers.
to completely plausible). We prepare an attention check test to guarantee the accuracy of human annotations and recollect evaluations for users who do not pass the test. The average score of the five annotators is treated as the final human score for each text. We normalize human scores to be in the same range of 0-1 as the model's output scores are. Table 4 shows the statistics and agreements in the conducted experiments.
We conduct a comprehensive set of experiments to examine and show the importance of training data in the plausibility evaluation task. We train both evaluation and language models on a machine with a GeForce RTX 2080 Ti GPU.
In our experiments, we have SENTAVG as the baseline model. We compare SENTAVG across more powerful classifiers - RoBERTa for ROC stories and Longformer for WP stories (FT_LM). We fine-tune pretrained RoBERTa-base model with the learning rate of 2e-5 and batch size 8 for three epochs and process the ROC stories with a maximum of 128 tokens. To evaluate WP with lengthy stories, we fine-tune pretrained Longformer-base model with the learning rate of 2e-5 and batch size 3 by encoding texts with at most 1024 tokens for three epochs.
We complete the models' comparisons by incorporating the recently proposed UNION model (Guan and Huang, 2020) to our experiments. We retrain it on the ROC_Eval and WP_Eval sets with the same hyper-parameters stated in their paper.
Table 5 depicts the quantitative results of correlation analysis between human and automatic evaluation metrics. For almost all constructed datasets for evaluation, the RoBERTa and Longformer in the case of short and long stories surpass the baseline models that show the impact of large transformer-based models in this evaluation task. The models trained on heuristically generated implausible samples by random sentence/keyword substitutions show the lowest correlations. The main reason for such weakness is the huge dissimilarity of heuristically generated training data and machine-generated test data, which has a significant negative impact on the model's performance. The positive
Dataset	Model	ROC	WP
ρ	τ	ρ	τ
HEUR_SENT_SUB	SENTAVG	0.04	0.03	-0.13	-0.10
FT_LM	0.10	0.07	0.12	0.10
HEUR_KEY_SUB	SENTAVG	-0.04	-0.03	-0.26	-0.18
FT_LM	0.31	0.22	0.08	0.06
UNION_DATA	SENTAVG	0.11	0.08	-0.22	-0.15
FT_LM	0.46	0.34	0.49	0.32
UNION	0.22	0.15	0.19	0.15
MANPLTS	SENTAVG	0.24	0.16	0.22	0.20
FT_LM	0.50	0.37	0.71	0.48
AF_MANPLTS	SENTAVG	0.22	0.16	0.25	0.23
FT_LM	0.56	0.41	0.74	0.52
Table 5: Higher correlations of plausibility evaluation models trained on manipulated plots and adversarially filtered negative samples with human judgments versus heuristically constructed negative samples. Ft_LM represents fine-tuned RoBERTa and Longformer models for ROC and WP datasets, respectively.
impact of UNION_Data is visible in Table 5. It demonstrates that the construction of implausible stories based on a more complete set of heuristic alterations yields better training data but still has its own shortcomings. This could be due to fact that text-level manipulations introduce artifacts that break the naturalness of the texts and have quite different styles compared to machine-generated implausible texts.
The superiority of RoBERTA and Longformer models trained on MANPLTS and AF_MANPLTS datasets show the effectiveness of our proposed plot manipulation technique in enhancing the similarity between the training and test data. Adversarial filtering technique further helps to increase the quality of negative samples and generate better implausible machine-generated texts, which consequently improves the accuracy of evaluation. By applying hypothesis testing to compare the metrics correlations with human scores (Diedenhofen and Musch, 2015), we verify that these improvements are statistically significant  $(p < .05)$ . We also note that the correlations between plot manipulation-based metrics and human evaluation are much higher in WP dataset. This could result from the limited ability of the current generative models to generate plausible long stories, thus making them easily distinguishable both by humans and automated metrics.
One of the desirable features of automated evaluation metrics for story generation is their generalizability or robustness to different datasets (Sellam et al., 2020; Guan and Huang, 2020). The dataset
3We fine-tune RoBERTa and Longformer models using https://github.com/huggingface/transformers.
Dataset	ROC → WP	WP → ROC
ρ	τ	ρ	τ
UNION_DATA	0.17	0.15	0.12	0.07
MANPLTS	0.57	0.39	0.23	0.16
AF_MANPLTS	0.60	0.42	0.26	0.18
Table 6: Correlation of plausibility metrics with human judgements. Arrow shows the train and test data used for examining metrics robustness.
Dataset	ρ	τ
MANPLTS-REORDER	0.65	0.42
MANPLTS-CONTINSER	0.69	0.45
MANPLTS-REPEATINSER	0.67	0.43
MANPLTS-RANDSUB	0.68	0.45
Table 7: Correlations of Longformer model fine-tuned on plot-level manipulated datasets with one specific excluded technique.
shifting robustness shows the metric's success in accurately evaluating texts in different datasets. We examine the robustness of metrics by leveraging ROC and WP as two distributionally different types of stories datasets. We train models on various training data constructed from negative sampling techniques in ROC dataset and test them on human scores collected through AMT experiments conducted on WP dataset  $(\mathbf{ROC}\rightarrow$  WP) and vice versa  $(\mathbf{WP}\rightarrow \mathbf{ROC})$ . In Table 6, we show the robustness of fine-tuned language models trained on the last three datasets of Table 5 as the best performing models in comparison to models trained on sentence and keyword substitutions. According to Table 6, the correlation drops due to the quite different structure of two datasets. RoBERTa/Longformer models fine-tuned on AF_MANPLTS in ROC/WP datasets and subsequently tested on WP/ROC dataset have the highest correlations with human judgments and can be generalized well on two datasets. The data shifting from ROC to WP better preserves the performance of metrics rather than the counterpart shifting. The reason for correlation decline of models trained on WP and tested on ROC could be the format of implausible texts in WP that could not be found in ROC data since the stories are shorter in this data and the reason for implausibility is fewer.
5.4.1 Ablation Study
The positive impact of plot-level manipulations in precisely evaluating the plausibility can be assessed with regard to the four different manipu
lation techniques. We conduct an ablation study on WP dataset to examine each manipulation technique's impact separately. We construct different training data each time by excluding one of the manipulation techniques and generating a new set of negative samples. Then we fine-tune Longformer on all these training datasets with different negative samples and compute the correlation of the finetuned Longformer as the evaluation metric with human judgments.
The lower correlations shown in Table 7 in comparison to Table 5 illustrate the harms that the elimination of each of the proposed approaches from the construction of training data could cause. This attests to the effectiveness of all proposed manipulation techniques in the generation of higher quality training data and subsequently resulting in more accurate evaluation metrics.
As this table demonstrates, the correlation drops the most by abulating the reordering and repeating plots, which shows that they are the major problems in generating long texts by language models and have the most significant role in constructing high-quality implausible samples and consequently accurate evaluation metrics.
Automatic plausibility evaluation models that are trained on heuristically generated data show low correlation with human judgement. We address this issue by creating a better quality set of implausible texts. In contrast to existing methods that modify text at token level, our approach introduces incoherence sources at a more abstract plot level, which helps to guide the generative model conditioned on those manipulated plots to generate negative samples that are more similar to machine-generated incoherent texts. We further improve the data quality by applying adversarial filtering to select more challenging and refined negative samples. Our experiments demonstrate that negative examples generated according to the proposed method result in more realistic implausible texts and consequently lead to more accurate evaluation metrics that have higher correlation with human judgement.
All co-authors of this work totally understand and agree with ACM Code of Ethics and its importance in expressing the conscience of the profession. We ensure this work is compatible with the provided
code, specifically in the terms of providing nonoffensive dataset construction.
1) training data construction In our approach, we use BART model conditioned on manipulated story plots to construct implausible samples that better reflect the implausibility in generation models. The main concern that arises here is the probability of generating abusive language samples from manipulated plots. Indeed, these plots origin from human-written stories without abusive languages provided by (Mostafazadeh et al., 2016; Fan et al., 2018) where users are not allowed to write profanity and inappropriate content. Accordingly, our manipulated version of plots and the BART model conditioned on them generate samples unlikely to contain strong biases or abusive content. It is noteworthy to mention that even the source plots are relatively benign, the process of altering them would have the possibility of creating objectionable texts. Other potential attack could be the dual-usage of metrics by augmenting offensive language texts as plausible samples. This would harm the underlying tasks such as story generation models to be encouraged to generate inappropriate stories. Such attacks can be identified and dissolved by security trended studies which are out of this work's scope.
1) testing data collection We collect human judgments by conducting Amazon Mechanical Turk (AMT) experiments that are leveraged to compare the accuracy of trained metrics in terms of their correlations with human scores. The conducted AMT does not disrupt user privacy as we do not contain their personal information. This fades the possibility of any gender bias problems and IRB approval needs. Annotators were asked to rate the coherence of stories in each HIT page of AMT in the range of 0 up to 5. We fairly compensated annotators. The average time of annotating each HIT in AMT was 25 minutes (including three stories for evaluation and their explanations), and according to the per hour wage of $13, we fairly paid them $6 per HIT.
This work targets the NLP open-domain generation community. Our metrics establish the main basis to achieve higher-quality generations by automatically assess the outputs and save time, cost, and human efforts. We don't anticipate specific failure modes in our work since the provided approach's success has been investigated through a comprehensive set of comparisons with other existing metrics.
This work is supported by the CwC program under the Contract W911NF-15-1-0543 with the US Defense Advanced Research Projects Agency (DARPA). We would like to thank the anonymous reviewers for their helpful comments and the members of PLUSlab from USC/UCLA, Shushan Arakelyan, and Ninareh Mehrabi for their constructive feedback.
Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. Comet: Commonsense transformers for automatic knowledge graph construction. In Association for Computational Linguistics (ACL).
Faeze Brahman, Alexandru Petrusca, and Snigdha Chaturvedi. 2020. Cue me in: Content-inducing approaches to interactive story generation. In Asia-Pacific Chapter of the Association for Computational Linguistics (AACL).
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics (NAACL-HLT).
Birk Diedenhofen and Jochen Musch. 2015. cocor: A comprehensive solution for the statistical comparison of correlations. PloS one.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833.
Angela Fan, Mike Lewis, and Yann Dauphin. 2019. Strategies for structuring story generation. In Association for Computational Linguistics (ACL).
Seraphina Goldfarb-Tarrant, Tuhin Chakrabarty, Ralph Weischedel, and Nanyun Peng. 2020. Content planning for neural story generation with aristotelian rescoring. In Empirical Methods in Natural Language Processing (EMNLP).
Seraphina Goldfarb-Tarrant, Haining Feng, and Nanyun Peng. 2019. Plan, write, and revise: an interactive system for open-domain story generation. In 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT 2019), Demonstrations Track, volume 4, pages 89-97.
Jian Guan and Minlie Huang. 2020. Union: An unref- erenced metric for evaluating open-ended story generation. In Empirical Methods in Natural Language Processing (EMNLP).
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In ICLR.
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. 2018. Learning to write with cooperative discriminators. In Association for Computational Linguistics (ACL).
Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In ICLR.
Alice Lai and Joel Tetreault. 2018. Discourse coherence in the wild: A dataset, evaluation and methods. arXiv preprint arXiv:1805.04993.
Mirella Lapata and Regina Barzilay. 2005. Automatic evaluation of text coherence: Models and representations. In *IJCAI*, volume 5, pages 1085–1090.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Association for Computational Linguistics (ACL).
Jiwei Li and Dan Jurafsky. 2016. Neural net models for open-domain discourse coherence. arXiv preprint arXiv:1606.01545.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Eleni Miltsakaki and Karen Kukich. 2004. Evaluation of text coherence for electronic essay scoring systems. Natural Language Engineering, 10(1):25-55.
N Mostafazadeh, N Chambers, X He, D Parikh, D Batra, L Vanderwende, P Kohli, and J Allen. 2016. A corpus and cloze evaluation framework for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT 2016), San Diego, CA, USA. Association for Computational Linguistics.
David Newman, Joy Han Lau, Karl Grieser, and Timothy Baldwin. 2010. Automatic evaluation of topic coherence. In Human language technologies: The