SlowGuess/ACL_2020_OCR huggingface dataset:
In DiCoh, the size of the hidden states in LSTMs of the utterance module is 128 and of the dialogue module is 256. The parameters of this model are optimized using the Adam optimizer where its parameters have default values except the learning rate which is initiated with 0.0005. A dropout layer with  $p = 0.1$  is applied to the utterance vectors. We
Model	DailyDialog	SwitchBoard
UO	UI	UR	EUO	UO	UI	UR	EUO
Random	50.10	49.97	49.97	49.92	49.98	50.02	49.99	50.13
CoSim	57.20	50.88	65.18	66.86	82.84	55.63	50.87	74.48
ASeq	68.21	57.41	61.89	62.73	99.70	73.94	63.48	99.20
EAGrid	71.72	60.93	68.49	67.18	99.65	73.70	75.61	99.83
S-DiCoh	94.23 ± .74	83.33 ± .81	81.89 ± .26	86.38 ± .29	95.51 ± .61	80.60 ± 1.12	53.61 ± .35	88.83 ± .35
M-DiCoh	95.92 ± .12	88.20 ± .36	83.02 ± .50	88.55 ± .39	99.41 ± .11	85.04 ± 1.14	58.67 ± 1.79	97.08 ± .20
Table 3: The accuracy (%) of the examined models on the test set of each experiment defined on DailyDialog and SwitchBoard.
train the model for 20 epochs on DailyDialog and 10 epochs on SwitchBoard and evaluate it at the end of each epoch on the validation set. The best performing model on the validation set is used for the final evaluation on the test set. Parameters  $\gamma_{1}$  and  $\gamma_{2}$  (see Equation 11) are initiated with 2.0 and are updated during training. To have fair comparisons, we train and evaluate all compared models on identical training, validation, and test sets.
Results Table 3 shows the accuracy of the baseline models (top) and our model (bottom) on DailyDialog and SwitchBoard.
We investigate how well our DiCoh model performs in comparison with its baseline peers that do not take DAs into account, i.e., Random and CoSim. We observe that S-DiCoh strongly outperforms these models for all the examined problem-domains on both DailyDialog and SwitchBoard, confirming the validity of our DiCoh model for capturing the semantics of utterances.
In a more challenging comparison, we compare S-DiCoh with ASeq and EAGrid as the baseline models that use DA information. Our S-DiCoh even surpasses these models for all problem-domains on DailyDialog. However, on SwitchBoard, S-DiCoh achieves lower accuracy than these models for all problem-domains except UI. This observation shows that when dialogue utterances are short (like those in SwitchBoard in comparison with those in DailyDialog), DAs are more crucial for coherence assessment. It is worth noting that unlike EAGrid and ASeq, S-DiCoh is completely agnostic to DA information.
When we employ DAP as an auxiliary task to train the DiCoh model in our MTL setup, we observe that M-DiCoh substantially outperforms the Random, CoSim, and S-DiCoh models (which do not use DAs) for all problem-domains on both DailyDialog and SwitchBoard. It concludes that our proposed MTL approach effectively leverages the DAP task to learn informative utterance vectors
for dialogue coherence assessment.
Compared with the ASeq and EAGrid models, which explicitly use gold DA labels during evaluations, our M-DiCoh achieves the highest accuracy for all problem-domains on DailyDialog, showing that our approach for involving DAs yields more informative utterance representations for coherence assessments. However, on SwitchBoard, M-DiCoh increases the accuracy of S-DiCoh up to those of EAGrid for UO and EUO. Surprisingly, it achieves lower accuracy than what EAGrid achieves for UR.
An explanation for why M-DiCoh outperforms ASeq and EAGrid on DailyDialog but not on SwitchBoard might be that the ASeq and EAGrid models explicitly use gold DA labels during evaluation but M-DiCoh does not; and the DA labels in SwitchBoard are about 10 times higher fine-grained than those in DailyDialog (see Table 2). This interpretation becomes more concrete by observing a considerable reduction in the performance of ASeq and EAGrid when they are evaluated on DailyDialog compared with when they are evaluated on SwitchBoard. In contrast, our M-DiCoh, which uses DAs only during training to obtain better utterance vectors, performs almost evenly on both corpora. Since our model does not need DA labels during evaluations, it is more suitable than the examined models for evaluating dialogue coherence in real scenarios.
Finally, to shed some light on which parts of a dialogue receive higher attentions by our M-DiCoh model, we analyze the attention weights it assigns to utterance words. Table 4 illustrates the attention weights for an example dialogue from the training set of the UO problem-domain on DailyDialog, where words with higher attention weights are darker than the those with lower attention weights. We observe that using dialog act prediction as an auxiliary task helps our coherence model to assign high attention weights to the salient words in dialogue utterances. The wh-question, adjectives, and
Figure 3: Comparing EAGrid (black bars) and M-DiCoh (white bars) in cross problem-domain. The labels of figures are the perturbations of the training sets and the labels on x-axes are the perturbations of the test sets.
the verb in questions have higher attention weights; while in other utterances, nouns, e.g. outlet, inexpensive, prices, are more salient. So, our multi-task learning approach yields richer representations of dialog utterances for coherence assessment.
4.5 Cross Problem-domain Evaluation
In a more challenging evaluation setup, we use the model trained on the training set of one problem-domain to evaluate it on the test sets of the other problem-domains. Therefore, the perturbation methods used for constructing the training sets differ from those used for creating the test sets. We compare EAGrid as the state-of-the-art coherence model, and M-DiCoh as our complete model, for cross problem-domain evaluations on DailyDialog.
Results Figure 3 shows the results on the test sets of the problem-domains, where the models are trained on the training set created by the (a) UO, (b) UI, (c) UR, and (d) EUO perturbations. For all perturbations used to construct the training sets, we observe that M-DiCoh outperforms EAGrid for all test perturbations. Interestingly, among all examined perturbations, both M-DiCoh and EAGrid achieve the highest accuracy on UO. We speculate that this perturbation is easy-to-solve as it rearranges all utterances in a dialogue. Cervone et al.
Utterance	DA labels
utt1	hello, where can i buy an inexpensive cashmere sweater?	Question
utt2	maybe you should look around for an outlet.	Directive
utt3	that is a wonderful idea.	Commissive
utt4	outlets have more reasonable prices.	Inform
utt5	thank you for your help.	Inform
utt6	no problem. good luck.	Inform
Table 4: An illustration of attention weights assigned to words in a dialogue from DailyDialog. Different gray shades show different attention weights.
(2018) also show that UR is easier to solve than UI.
We note a low-discrepancy in the accuracy of the M-DiCoh model on the test set of UO when the model is trained on the training sets of the different examined problem-domains. The biggest drop in accuracy (3.2 percentage point) on the UO problem-domain is for when the model is trained on the training set of the UR problem-domain. In contrast, we observe a high-discrepancy in the accuracy of the EAGrid model for the UO problem-domain when the model is trained on the training sets of different problem-domains. The accuracy of EAGrid on the test set of UO drops from  $71.72\%$  (when trained for UO) to  $58.7\%$  (when trained for UR). This is about 13 percentage points drop in accuracy. These results confirm that our M-DiCoh model is more robust than the EAGrid model against different types of perturbation.
4.6 DAP Model Evaluation
Since using DAP as an auxiliary task improves the performance of our coherence model; in this experiment, we investigate the impact of MTL on the performance of the DAP model. We train our DAP model without any coherence supervision signal, S-DAP, with  $\mathcal{L} = \frac{\mathcal{L}_{da}^{diai} + \mathcal{L}_{da}^{diaj}}{2}$  in Equation 11, and compare it with the model that is trained with our MTL, M-DAP.
Results Table 5 shows the F1 metric<sup>2</sup> of these models for our problem-domains on the DailyDialog dataset. This dataset is larger than SwitchBoard, and the frequency of dialogue act labels in this dataset is more balanced than those in SwitchBoard. We use an SVM classifier supplied with Bag-of-Word representations of utterances as a baseline to put our results in context.
Both S-DAP and M-DAP models outperform the SVM-BoW model for all problem-domains, indi
2We use F1 because there are more than two DA labels.
UO	UI	UR	EUO
SVM-BoW	76.11	75.52	74.49	75.73
S-DAP	78.10±.20	79.15±.34	77.99±.35	78.81±.31
M-DAP	77.32±.36	78.49±.33	77.52±.27	78.51±.23
Table 5: The F1 metric of the DAP model for the test sets of the problem-domains on DailyDialog. S-DAP is the model trained without any coherence supervision, and M-DAP is the model trained with MTL.
cating that the employed DAP model is suitable for solving this task. However, we observe that the M-DAP model works on par with the S-DAP model. This observation shows that the information encoded by the coherence model is not useful for solving the dialogue act prediction task. The coherence model captures semantic relations in a dialogue by encoding information about the content of utterances. Dialogue acts, which indicate speakers' intentions of stating utterances in a dialogue, are independent of the content of utterances, therefore information learned by the coherence model does not help the DAP model.
However, as the other experiments in this paper demonstrate, DAs can help to obtain more informative utterance representations to model dialogue coherence. Our multi-task learning approach relieves the need for explicit DA labels for coherence assessments, which is the main goal of this paper.
We propose a novel dialogue coherence model whose utterance encoder layers are shared with a dialogue act prediction model. Unlike previous approaches that utilize these two models in a pipeline, we use them in a multi-task learning scenario where dialogue act prediction is an auxiliary task. Our coherence method outperforms its counterparts for discriminating dialogues from their various perturbations on DailyDialog, and (mostly) performs on par with them on SwitchBoard. Our model (a) benefits from dialogue act prediction task during training to obtain informative utterance vectors, and (b) alleviates the need for gold dialogue act labels during evaluations. These properties holistically make our model suitable for comparing different dialogue agents in terms of coherence and naturalness. For future work, we would like to deeply study the impacts of our perturbations on the coherence of the examined dialogues. We will also investigate to what extent the rankings of dialogues obtained by our model correlate with human-provided rankings.
This work was supported by the German Research Foundation through the German-Israeli Project Cooperation (DIP, grant DA 1600/1-1 and grant GU 798/17-1). We thank Kevin Stowe and Leonardo Filipe Rodrigues Ribeiro for their valuable feedback on earlier drafts of this paper. We also thank anonymous reviewers for their useful suggestions for improving the quality of the paper.
Regina Barzilay and Mirella Lapata. 2005. Modeling local coherence: An entity-based approach. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics, Ann Arbor, Mich., 25-30 June 2005, pages 141-148.
Regina Barzilay and Mirella Lapata. 2008. Modeling local coherence: An entity-based approach. Computational Linguistics, 34(1):1-34.
Jill Burstein, Joel Tetreault, and Slava Andreyev. 2010. Using entity-based features to model coherence in student essays. In Proceedings of Human Language Technologies 2010: The Conference of the North American Chapter of the Association for Computational Linguistics, Los Angeles, Cal., 2-4 June 2010, pages 681-684.
Donna K. Byron and Amanda Stent. 1998. A preliminary model of centering in dialog. In Proceedings of the 17th International Conference on Computational Linguistics and 36th Annual Meeting of the Association for Computational Linguistics, Montreal, Quebec, Canada, 10-14 August 1998, pages 1475-1477.
Alessandra Cervone, Evgeny Stepanov, and Giuseppe Riccardi. 2018. Coherence models for dialogue. In Proceedings of the 19th Annual Conference of the International Speech Communication Association, Hyderabad, 2-6 September 2018, pages 1011-1015.
Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander H. Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W. Black, Alexander I. Rudnicky, Jason Williams, Joelle Pineau, Mikhail Burtsev, and Jason Weston. 2019. The second conversational intelligence challenge (convai2). CoRR, abs/1902.00098.
Nouha Dziri, Ehsan Kamalloo, Kory Mathewson, and Osmar Zaiane. 2019. Evaluating coherence in dialogue systems using entailment. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, Minnesota., 2-7 June 2019, pages 3806-3812.
Youmna Farag and Helen Yannakoudakis. 2019. Multi-task learning for coherence modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Florence, Italy, 28 July - 2 August, 2019, pages 629-639.
Sudeep Gandhi and David Traum. 2008. Evaluation understudy for dialogue coherence models. In Proceedings of the 9th SIGdial Workshop on Discourse and Dialogue, Columbus, Ohio, 19-20 June 2008, pages 172-181.
Sudeep Gandhi and David Traum. 2016. A semi-automated evaluation metric for dialogue model coherence. In 7th International Workshop on Spoken Dialogue Systems, Saariselk'a, Finland, 13–16 January 2016, pages 141–150.
Yang Gao, Christian M. Meyer, Mohsen Mesgar, and Iryna Gurevych. 2019. Reward learning for efficient reinforcement learning in extractive document summarisation. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, Macao, China, 10-16 August 2019, pages 2350-2356.
Marjan Ghazvininejad, Chris Brockett, Ming Wei Chang, Bill Dolan, Jianfeng Gao, Wen Tau Yih, and Michel Galley. 2018. A knowledge-grounded neural conversation model. In Proceedings of the 32nd Conference on the Advancement of Artificial Intelligence, New Orleans, Louisiana, 2-7 February 2018, pages 5110-5117.
Barbara J. Grosz and Candace L. Sidner. 1986. Attention, intentions, and the structure of discourse. Computational Linguistics, 12(3):175-204.
Camille Guinaudeau and Michael Strube. 2013. Graph-based local coherence modeling. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Sofia, Bulgaria, 4-9 August 2013, pages 93-103.
M. A. K. Halliday and Ruqaiya Hasan. 1976. Cohesion in English. London, U.K.: Longman.
Daniel Jurafsky and Elizabeth Shriberg. 1997. Switchboard SWBD-DAMSL shallow-discourse-function annotation coders manual, draft 13. Technical Report 97-02, University of Colorado at Boulder.
Alex Kendall, Yarin Gal, and Roberto Cipolla. 2018. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the International Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, 18-22 June 2018, pages 7482-7491.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Taipei, Taiwan, 27 November - 1 December, 2017, pages 986-995.
Ziming Li, Julia Kiseleva, and Maarten de Rijke. 2019. Dialogue generation: From imitation learning to inverse reinforcement learning. In Proceedings of the 33rd Conference on the Advancement of Artificial Intelligence, Honolulu, Hawaii, 21 January -1 February 2019, pages 6722-6729.
Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas, 1–5 November 2016, pages 2122–2132.
Mohsen Mesgar and Michael Strube. 2014. Normalized entity graph for computing local coherence. In Proceedings of TextGraphs-9: Graph-based Methods for Natural Language Processing, Workshop at EMNLP 2014, Doha, Qatar, 29 October 2014, pages 1-5.
Mohsen Mesgar and Michael Strube. 2018. A neural local coherence model for text quality assessment. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, 31 October - 4 November 2018, pages 4328-4339.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, Doha, Qatar, 25-29 October 2014, pages 1532-1543.
C. Raymond Perrault and James F. Allen. 1978. Speech acts as a basis for understanding dialogue coherence. In Theoretical Issues in Natural Language Processing-2.
Amruta Purandare and Diane J. Litman. 2008. Analyzing dialog coherence using transition patterns in lexical and semantic features. In Proceedings of the 21st International Florida Artificial Intelligence Research Society Conference, Coconut Grove, Florida, 15-17 May 2008, pages 195-200.
Vipul Raheja and Joel Tetreault. 2019. Dialogue act classification with context-aware self-attention. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, Minnesota., 2-7 June 2019, pages 3727-3733.
Nils Reimers and Iryna Gurevych. 2018. Why comparing single performance scores does not allow to draw conclusions about machine learning approaches. CoRR, abs/1803.09578.
Gerard Salton. 1971. The SMART Retrieval System - Experiments in Automatic Document Processing. Englewood Cliffs, N.J.: Prentice Hall.
John Searle. 1969. Speech Acts. Cambridge University Press, Cambridge, U.K.
Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of the 30th Conference on the Advancement of Artificial Intelligence, Phoenix, Arizona, 12-17 February 2016, pages 3776-3783.
Dat Tien Nguyen and Shafiq Joty. 2017. A neural local coherence model. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, Canada, 30 July - 4 August 2017, pages 1320-1330.
Svitlana Vakulenko, Maarten de Rijke, Michael Cochez, Vadim Savenkov, and Axel Polleres. 2018. Measuring semantic coherence of a conversation. In Proceedings of the 17th International Semantic Web Conference, Monterey, Ca., 8-12 October 2018, pages 634-651.
Xinnuo Xu, Ondrej Dusek, Ioannis Konstas, and Verena Rieser. 2018. Better conversations by modeling, filtering, and optimizing for coherence and diversity. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, 31 October - 4 November 2018, pages 3981-3991.
Hainan Zhang, Yanyan Lan, Jiafeng Guo, Jun Xu, and Xueqi Cheng. 2018. Reinforcing coherence for sequence to sequence model in dialogue generation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, Stockholm, Sweden, 13-19 July 2018, pages 4567-4573.
A More Details on EAGrid
EAGrid is a recent model for dialogue coherence with which we compare our models. It mainly extends the entity grid representation for monologue texts. Entity grid is a matrix whose rows represent dialogue utterances and columns encode entities mentioned in dialogue. Each entry in an entity grid is filled by the grammatical role (i.e. subject ("S"), object ("O"), neither of them ("X")) of its corresponding entity in its corresponding utterance if the entity is mentioned in the utterance, otherwise it is filled by "−". EAGrid appends a column for encoding dialogue acts to the entity grid such that the entries associated with this column are filled by the dialog act labels of corresponding utterances. Figure 4 shows the EAGrid representation of the example dialogue presented in the top-part of Table 1 in this paper. The grid is generated using EAGrid's code released by its authors. The proba
CHARLES	ENTITIES	DA labels
CAPTAIN	UNCLE	THIS
utt1	X	-	X	S	inform
utt2	-	-	-	-	question
utt3	-	X	-	-	inform
utt4	-	-	-	-	inform
utt5	-	-	-	-	inform
Figure 4: The EAGrid representations of Dialogue presented in Table.
bilities of entities' grammatical role and dialogue act label transitions of length  $n$  across utterance are used as coherence features<sup>3</sup>. These features are supplied to a SVM to rank dialogues concerning their coherence.
B LSTM
As the LSTM layer used in our model is well-known, we give the details of its definition here:
$$
\begin{array}{l} i _ {t} = \sigma (W _ {i i} e _ {t} + b _ {i i} + W _ {h i} h _ {(t - 1)} + b _ {h i}), \\ f _ {t} = \sigma (W _ {i f} e _ {t} + b _ {i f} + W _ {h f} h _ {(t - 1)} + b _ {h f}), \\ g _ {t} = \tanh \left(W _ {i g} e _ {t} + b _ {i g} + W _ {h g} h _ {(t - 1)} + b _ {h g}\right), \\ o _ {t} = \sigma (W _ {i o} e _ {t} + b _ {i o} + W _ {h o} h _ {(t - 1)} + b _ {h 0}), \\ c _ {t} = f _ {t} * c _ {(t - 1)} + i _ {t} * g _ {t}, \\ h _ {t} = o _ {t} * \tanh  \left(c _ {t}\right), \tag {13} \\ \end{array}
$$
where  $h_t$ , is the hidden and  $c_t$  is the cell state at word  $t$ . The input, forget, cell, and output gates at
word  $t$  are shown by  $i_t$ ,  $f_t$ ,  $g_t$ , and  $o_t$ , respectively.  $\sigma$  is the Sigmoid function, and  $*$  is the Hadamard product. The hidden state is initialized with a zero vector for representing each utterance in dialogue.
C Hyperparameters and Training
To approximate the best values of the hyperparameters, we perform a grid search, in which one parameter is varied while all others are fixed. The search was carried out in the multi-task learning setup on the dataset for the UO problem-domain on DailyDialog. For each variation of hyperparameter values, we train the model on the training set of UO and evaluate it on its validation set. The parameter values that result in the highest respective performance was chosen for evaluation on the test set. The values for the number of epochs and batch size were chosen to trade off the running time and memory consumption of the training. For the experiments on DailyDialog we set the maximum number of epochs to 20 and the batch size to 128, while for SwitchBoard the maximum number of epochs is set to 10 and the batch size to 16. Note that hyperparameter tuning has not been performed for SwitchBoard. Thus for the experiments on SwitchBoard mostly the same hyperparameters as those used for the experiments on DailyDialog are used, with the exception of the batch size and the number of epochs. Table 6 shows the final values for hyperparameters of our models. The optimization is performed by Adam with its default parameter values except for the learning rate. We train the model on the shuffled batches of training data. The model is evaluated on the validation set at each epoch. The model with the best performance on the validation set is chosen for the evaluation on the test set. The training procedure is accelerated by the usage of a Tesla P100 GPU running with CUDA v.10.1, while the model is implemented in the pytorch7 framework version 1.1.0.
parameter	DailyDialog	SwitchBoard
epochs	20	10
batch size	128	16
learning rate	0.0005	0.0005
number of LSTM layers	1	1
hidden layer of LSTMu	128	128
hidden layer of LSTMd	256	256
DA dropout rate	0.1	0.1
Table 6: The values of hyperparameters that result in the best performance on the validation set.
3Following the EAGrid model, we set  $n = 2$