SlowGuess/NAACL_2021_OCR huggingface dataset:
region 1: We share the widespread outrage at its attitude to history, in particular World War II, but also its policies on enlargement, on immigration, on race and its attitude to the European Union itself. We were also outraged, region 2: however
region 3: , at the tolerance of the left
<sup>6</sup>Europarl segments are either very long sentences formed by several clauses or by 2-3 sentences clustered together, as a product of the sentence alignment process.
for the tyranny, the terror and the excesses of the former USSR.
b. condition name: manipulated
region 1: We share the widespread outrage at its attitude to history, in particular World War II, but also its policies on enlargement, on immigration, on race and its attitude to the European Union itself. We were also outraged, region 2: since
region 3: , at the tolerance of the left for the tyranny, the terror and the excesses of the former USSR.
Some connectives may have the same sense depending on the specific context in which they appear (Stede, 2012; Webber et al., 2003), for instance both since and while may bear a temporal interpretation. On that account, we expect that a replacement with a different connective bearing a different sense leads to region 3 being more surprising than a different connective able to have the same sense.
Results. Not all relations captured by the connectives are equally difficult, producing high variability in the scores, as shown in Table 4. While temporal senses seem to be relatively unproblematic (scores about 0.85 on average, GPT-2), 'contrast', 'concession' and in particular 'causal' senses are more difficult to distinguish (since_causal and as_causal have averages of 0.66 and 0.52 respectively).
The results for as present an interesting contrast. This connective can also be used as a preposition. When the connectives with this particular sense are replaced, the models do not have any trouble recognising the original from the manipulated sentence, as suggested by the systematic high scores obtained, between 0.96 and 0.99. In most other senses, however, scores plummet as low as 0.28. We observe a similar pattern for yet when used as an adverb in the DIALOGPT model.
4.6 Speaker Commitment Test Suite
Hypothesis: While it is possible for different speakers to have different opinions, speakers should not contradict themselves. This test suite targets the notion of speaker commitment in dialogue models. The test suite is created automatically based on the DialogueNLI corpus (Welleck et al., 2019), which contains pairs of utterances annotated as contradiction, entailment or neutral. The sentence pairs are
extracted from the PERSONA-CHAT corpus introduced in Section 4.1. The sentences can either be part of the conversation or the persona descriptions. We extract the contradicting sentence pairs from the human verified test set, and create two conditions for each utterance pair, as illustrated below:
(8) a. condition name: speaker change region 1: since the beginning of the year, i am a nurse. [SEP] region 2: i am a kindergarten teacher.
b. condition name: same speaker region 1: since the beginning of the year, i am a nurse. region 2: i am a kindergarten teacher.
In the first condition, we simulate a speaker change by introducing a [SEP] token (which is converted to the tokenizer's separator token internally) in the dialogue history, whereas in the second condition the continuation is uttered by the same speaker as the context.
A model that is encoding some notion of speaker commitment should find the second utterance more surprising if no speaker change occurred.
As non-dialogue language models do not encode the notion of speaker change, this test suite only yields relevant results for dialogue models.
Results. DIALOGPT shows a tendency towards finding contradictions within the same speaker more surprising. A manual inspection of the data revealed that even though we use the human verified test set, there are quite some instances where the implications are not as clear, for example in the following two sentence pairs:
(9) a. "my nurse skills come in handy when i volunteer."
"I am a kindergarten teacher."
b. "i love art and want to be a famous artist."
This highlights the importance of quality over quantity. In future work, we will inspect this phenomenon more closely and combine the selection of items with human evaluation, to gain a better understanding of how the notion of speaker commitment is and can be encoded in neural dialogue models.
GPT-2
Connective used in manipulation	DIALOGPT
Connective used in manipulation
CONNECTIVE SENSE
although_concession	although	as	however	since	though	while	yet	although	as	however	since	though	while	yet
-	0.92	0.92	0.857	0.84	0.86	0.90	-	0.88	0.83	0.82	0.76	0.76	0.89
although Contrast	-	1.00	0.86	0.86	0.43	1.00	0.86	-	1.00	0.86	1.00	0.93	0.79	1.00
as_causal	0.44	-	0.80	0.28	0.64	0.72	0.80	0.64	-	0.68	0.40	0.84	0.80	0.88
as_comparison	0.96	-	0.95	0.96	0.94	0.97	0.97	0.86	-	0.89	0.86	0.93	0.87	0.92
as_concession	0.33	-	0.67	0.67	0.33	1.00	1.00	0.67	-	0.67	0.67	1.00	1.00	0.67
as_PREPOSITION	0.99	-	0.99	0.98	0.99	0.99	0.99	0.96	-	0.97	0.97	0.97	0.97	0.98
as_temporal	0.95	-	0.95	0.86	1.00	0.81	0.95	0.86	-	1.00	0.86	1.00	0.76	1.00
however_concession	0.70	0.90	-	0.86	0.63	0.80	0.64	0.58	0.79	-	0.79	0.71	0.71	0.53
however_concession	0.67	0.89	-	0.89	0.33	0.89	0.67	0.67	0.89	-	0.78	0.56	0.67	0.56
since_causal	0.61	0.78	0.83	-	0.72	0.79	0.93	0.66	0.82	0.74	-	0.83	0.79	0.89
since_temporal-causal	1.00	0.83	1.00	-	1.00	1.00	1.00	0.67	1.00	1.00	-	0.83	0.83	1.00
since_temporal	0.96	0.97	0.96	-	0.94	0.97	0.98	0.95	0.97	0.95	-	0.95	0.92	0.95
though_concession	0.43	0.82	0.79	0.87	-	0.78	0.90	0.37	0.87	0.76	0.82	-	0.78	0.79
though_concession	0.56	0.84	0.84	0.88	-	0.88	0.80	0.41	0.75	0.59	0.77	-	0.72	0.83
while_concession	0.46	1.00	1.00	0.96	0.78	-	0.98	0.57	0.96	0.87	0.89	0.76	-	0.93
while_concession	0.78	0.93	0.93	0.85	0.81	-	0.81	0.81	0.93	0.81	0.85	0.96	-	0.81
while_temporal-causal	0.90	0.80	0.90	0.70	0.80	-	0.80	0.80	0.80	1.00	0.70	0.90	-	1.00
while_temporal-contrast	0.73	0.90	0.90	0.73	0.77	-	0.81	0.67	0.81	0.81	0.85	0.88	-	0.81
while_temporal	0.57	0.57	0.71	0.86	0.71	-	0.86	0.86	0.71	0.86	0.86	1.00	-	1.00
yet_ADV	0.95	0.98	0.82	0.93	0.93	0.98	-	0.92	0.96	0.85	0.92	0.94	0.97	-
yet_concession	0.92	0.95	0.95	0.92	0.97	0.95	-	0.59	0.90	0.72	0.79	0.74	0.85	-
yet Contrast	0.92	0.88	0.88	0.92	0.92	0.79	-	0.67	0.96	0.75	0.88	0.79	0.88	-
Table 4: CD scores on explicit connectives test suite. The first column list all the connective senses from Disco-Annotation. Scores below 0.50 are boldfaced, while the PREPOSITION and ADverb senses are highlighted in yellow.
contradiction
DIALOGPT	0.59
#items	4041
Table 5: CD score for speaker commitment test suite.
We revisit the targeted evaluation paradigm and create test suites focusing on specific coherence phenomena. Each test suite contains minimal pairs of sequences that illustrate a specific component of coherence.
We evaluate two transformer models for language and dialogue modelling based on the token level surprisal scores they assign to the coherent and incoherent versions. Extending the existing SyntaxGym toolkit, we evaluate GPT-2 and DIALOGPT on our newly designed test suites on entity re-mention, explicit discourse connectives and speaker commitment in dialogue. Existing test sets are also integrated easily, which we demonstrate for sentence order detection, Story Cloze and Winograd Schema resolution tasks. Our results support previous work suggesting that the notion of coherence encoded in neural language models is more nuanced than the sentence order discrimination task
can reflect.
The mixed results we get, with some manipulations (e.g. the different sense connective substitutions) easily being spotted by the tested models and others (e.g. how to re-mention entities, or speaker contradictions) posing to be more difficult, point to the value of such targeted evaluation, which eventually might help in pointing towards where the introduction of different inductive biases could increase a model's performance.
In this study, we focus on the English language. However, our approach is not inherently designed for English alone. While  $1\mathrm{m} - 200$  only contains English language models at the moment, other language models can be added easily. The shuffling perturbations can be applied to any corpus. Our other test suites are based on available annotated corpora, which require some familiarity with the language, but can in principle be applied in a similar fashion to resources in other languages, such as the Potsdam Commentary Corpus (Bourgonje and Stede, 2020) for German connectives, for example. We leave a multilingual extension of our framework for future work.
Our next efforts will focus on adding more language and dialogue models to determine the impact of different model architectures and sizes. Building additional test suites in order to capture a more thor-
ough notion of coherence is also among our priorities. Last, we plan to collect human judgements to evaluate our coherence manipulations more closely and to create an upper bound for what we can expect from neural models.
We thank Johann Seltmann and Jon Gauthier for their help with augmenting lm-zoo and syntaxgym. We also thank the anonymous reviewers for their valuable feedback. This work was funded by the German Research Foundation (Deutsche Forschungsgemeinschaft, DFG) - Project ID 317633480 - SFB 1287.
Mira Ariel. 2004. Accessibility marking: Discourse functions, discourse profiles, and processing cues. *Discourse Processes*, 37(2):91-116.
Regina Barzilay and Mirella Lapata. 2008. Modeling Local Coherence: An Entity-Based Approach. Computational Linguistics, 34(1):1-34.
Peter Bourgonje and Manfred Stede. 2020. The Potsdam commentary corpus 2.2: Extending annotations for shallow discourse parsing. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 1061-1066, Marseille, France. European Language Resources Association.
Mingda Chen, Zewei Chu, and Kevin Gimpel. 2019. Evaluation benchmarks and learning criteria for discourse-aware sentence representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 649-662, Hong Kong, China. Association for Computational Linguistics.
Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. 2020. SyntaxGym: An online platform for targeted evaluation of language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 70-76, Online. Association for Computational Linguistics.
Thomas Givón. 1983. *Topic Continuity in Discourse: A Quantitative Cross-Language Study*. John Benjamin, Amsterdam.
Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41-58. Brill.
Barbara Grosz, Aravind K. Joshi, and Scott Weinstein. 1995. Centering: A framework for modelling the local coherence of discourse. Computational Linguistics, 21(2):203-225.
Jerry R Hobbs. 1979. Coherence and Coreference. Cognitive Science, 3(1):67-90.
Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725-1744, Online. Association for Computational Linguistics.
Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of the 10th Machine Translation Summit, MT Summit X, pages 79-86, Phuket, Thailand.
Alice Lai and Joel Tetreault. 2018. Discourse coherence in the wild: A dataset, evaluation and methods. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pages 214-223, Melbourne, Australia. Association for Computational Linguistics.
A. Lascarides and N. Asher. 2009. Agreement, Disputes and Commitments in Dialogue. Journal of Semantics, 26(2):109-158.
William C. Mann and Sandra A. Thompson. 1987. Rhetorical structure theory: A theory of text organization. In Livia Polanyi, editor, The Structure of Discourse. Ablex Publishing Corporation, Norwood, N.J.
Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192-1202, Brussels, Belgium. Association for Computational Linguistics.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428-3448, Florence, Italy. Association for Computational Linguistics.
Shikib Mehri and Maxine Eskenazi. 2020. Unsupervised evaluation of interactive dialog with DialogGPT. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 225-235, 1st virtual meeting. Association for Computational Linguistics.
Mohsen Mesgar, Sebastian Bucker, and Iryna Gurevych. 2020. Dialogue coherence assessment without explicit dialogue act labels. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1439-1450, Online. Association for Computational Linguistics.
Elham Mohammadi, Timothe Beiko, and Leila Kosseim. 2020. On the creation of a corpus for coherence evaluation of discursive units. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 1067-1072, Marseille, France. European Language Resources Association.
Han Cheol Moon, Tasnim Mohiuddin, Shafiq Joty, and Chi Xu. 2019. A unified neural coherence model. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2262-2272, Hong Kong, China. Association for Computational Linguistics.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839-849, San Diego, California. Association for Computational Linguistics.
Leila Pishdad, Federico Fancellu, Ran Zhang, and AfSaneh Fazly. 2020. How coherent are neural models of coherence? In Proceedings of the 28th International Conference on Computational Linguistics, pages 6126-6138, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Andrei Popescu-Belis, Thomas Meyer, Jeevanthi Liyanapathirana, Bruno Cartoni, and Sandrine Zufferey. 2012. Discourse-level Annotation over European for Machine Translation: Connectives and Pronouns. In Proceedings of the eighth international conference on Language Resources and Evaluation (LREC), Istanbul, Turkey.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report, OpenAI.
Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D. Manning. 2019. Do massively pretrained language models make better storytellers? In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 843-861, Hong Kong, China. Association for Computational Linguistics.
Manfred Stede. 2012. Disourse Processing. Morgan and Claypool Publishers, Toronto.
Alexandre Tamborrino, Nicola Pellicanò, Baptiste Pannier, Pascal Voitot, and Louise Naudin. 2020. Pretraining is (almost) all you need: An application to commonsense reasoning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3878-3887, Online. Association for Computational Linguistics.
Trieu H. Trinh and Quoc V. Le. 2019. A Simple Method for Commonsense Reasoning. arXiv:1806.02847 [cs].
Olga Uryupina, Ron Artstein, Antonella Bristot, Federica Cavicchio, Francesca Delogu, Kepa Rodriguez,
and Massimo Poesio. 2020. Annotating a broad range of anaphoric phenomena, in multiple genres: the ARRAU corpus. Natural Language Engineering, 26:95-128.
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377-392.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641.
Bonnie Webber, Matthew Stone, Aravind Joshi, and Alistair Knott. 2003. Anaphora and discourse structure. Computational Linguistics, 29(4):545-587.
Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019. Dialogue natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3731-3741, Florence, Italy. Association for Computational Linguistics.
Peng Xu, Hamidreza Saghir, Jin Sung Kang, Teng Long, Avishek Joey Bose, Yanshuai Cao, and Jackie Chi Kit Cheung. 2019. A cross-domain transferable neural coherence model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 678-687, Florence, Italy. Association for Computational Linguistics.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204-2213, Melbourne, Australia. Association for Computational Linguistics.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. DIALOGPT: Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 270-278, Online. Association for Computational Linguistics.