SlowGuess/ACL_2020_OCR huggingface dataset:
Mohsen Mesgar
Sebastian Bucker
Iryna Gurevych
Ubiquitous Knowledge Processing Lab (UKP)
Technische Universität Darmstadt (TUDa)
{mesgar,buecker,gurevych}@ukp.tu-darmstadt.de
Recent dialogue coherence models use the coherence features designed for monologue texts, e.g. nominal entities, to represent utterances and then explicitly augment them with dialogue-relevant features, e.g., dialogue act labels. It indicates two drawbacks, (a) semantics of utterances is limited to entity mentions, and (b) the performance of coherence models strongly relies on the quality of the input dialogue act labels. We address these issues by introducing a novel approach to dialogue coherence assessment. We use dialogue act prediction as an auxiliary task in a multi-task learning scenario to obtain informative utterance representations for coherence assessment. Our approach alleviates the need for explicit dialogue act labels during evaluation. The results of our experiments show that our model substantially (more than 20 accuracy points) outperforms its strong competitors on the DailyDialogue corpus, and performs on par with them on the SwitchBoard corpus for ranking dialogues concerning their coherence. We release our source code<sup>1</sup>.
Considering rapid progresses in developing open-domain dialogue agents (Serban et al., 2016; Ghazvininejad et al., 2018; Dinan et al., 2019; Li et al., 2019), the need for models that compare these agents in various dialogue aspects becomes extremely important (Liu et al., 2016; Dinan et al., 2019). Most available methods for dialogue evaluation rely on word-overlap metrics, e.g. BLEU, and manually collected human feedback. The former does not strongly correlate with human judgments (Liu et al., 2016), and the latter is time-consuming and subjective. A fundamental aspect of dialogue is coherence – what discriminates a high-quality
Figure 1: A high-level view of our multi-task learning approach for dialogue coherence modeling.
dialogue from a random sequence of dialogue utterances (Halliday and Hasan, 1976; Grosz and Sidner, 1986; Byron and Stent, 1998). Dialogue coherence deals with semantic relations between utterances considering their dialogue acts (Perrault and Allen, 1978; Cervone et al., 2018).
A Dialogue Act (henceforth DA) gives a meaning to an utterance in a dialogue at the level of "illocutionary force", and therefore, constitutes the basic unit of communication (Searle, 1969; Raheja and Tetreault, 2019). A DA captures what a speaker's intention is of saying an utterance without regard to the actual content of the utterance. For example, a DA may indicate whether the intention of stating an utterance is to ask a question or to state a piece of information.
Recent approaches to dialogue coherence modeling use the coherence features designed for monologue texts, e.g. entity transitions (Barzilay and Lapata, 2005), and augment them with dialogue-relevant features, e.g., DA labels (Cervone et al., 2018). These DA labels are provided by human annotators or DA prediction models. Such coherence models suffer from the following drawbacks: (a) they curb semantic representations of utterances to entities, which are sparse in dialogue because of short utterance lengths, and (b) their performance relies on the quality of their input DA labels.
<https://github.com/UKPLab/ acl2020-dialogue-coherence-assessment
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1439-1450 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
We propose a novel approach to dialogue coherence assessment by utilizing dialogue act prediction as an auxiliary task for training our coherence model in a multi-task learning (MTL) scenario (Figure 1). Our approach consists of three high-level components: an utterance encoder, a dialogue coherence model (DiCoh), and a Dialogue Act Prediction (DAP) model. The layers of the utterance encoder are shared between the DAP and the DiCoh model. This idea enables our DiCoh model to learn to focus on salient information presented in utterances considering their DAs and to alleviate the need for explicit DA labels during coherence assessment.
We evaluate our MTL-based approach on the DailyDialog (Li et al., 2017) and SwitchBoard (Jurafsky and Shriberg, 1997) English dialogue corpora in several discriminating experiments, where our coherence model, DiCoh, is examined to discriminate a dialogue from its perturbations (see Table 1). We utilize perturbation methods, like utterance ordering and utterance insertion, inherited from coherence evaluation approaches for monologue texts, and also introduce two dialogue-relevant perturbations, named utterance replacement and even utterance ordering.
Our core contributions are: (1) proposing an MTL-based approach for dialogue coherence assessment using DAP as an auxiliary task, yielding more informative utterance representations for coherence assessment; (2) alleviating the need for DA labels for dialogue coherence assessment during evaluations; (3) an empirical evaluation on two benchmark dialogue corpora, showing that our model substantially outperforms the state-of-the-art coherence model on DailyDialog, and performs on par with it on SwitchBoard.
Early approaches to dialogue coherence modeling are built upon available models for monologue, such as the EntityGrid model (Barzilay and Lapata, 2005, 2008). EntityGrid and its extensions (Burstein et al., 2010; Guinaudeau and Strube, 2013; Mesgar and Strube, 2014; Tien Nguyen and Joty, 2017; Farag and Yannakoudakis, 2019) rely on entity transitions, as proxies of semantic connectivity, between utterances. These approaches are agnostic to discourse properties of dialogues (Purandare and Litman, 2008; Gandhi and Traum, 2008; Cervone et al., 2018).
Utterance	DA label
coherent
utt1:	This is my uncle, Charles.	inform
utt2:	He looks strong. What does he do?	question
utt3:	He's a captain.	inform
utt4:	He must be very brave.	inform
utt5:	Exactly!	inform
incoherent
utt1::	This is my uncle, Charles.	inform
utt4:	He must be very brave.	inform
utt3:	He's a captain.	inform
utt2:	He looks strong. What does he do?	question
utt5:	Exactly!	inform
Table 1: An example dialogue from DailyDialog (top) and its perturbation (bottom), which is generated by permuting the utterances said by one of the speakers (shown in boldface), and is less coherent. The right column shows the DA labels associated with utterances.
Inspired by EntityGrid, Gandhi and Traum (2016) define transition patterns among DA labels associated with utterances to measure coherence. Cervone et al. (2018) combine the above ideas by augmenting entity grids with utterance DA labels. This model restricts utterance vectors only to entity mentions, and needs gold DA labels as its inputs for training as well as evaluation. However, obtaining DA labels from human annotators is expensive and using DAP models makes the performance of coherence model dependent on the performance of DAP models.
Recent approaches to dialogue coherence modeling benefit from distributional representations of utterances. Zhang et al. (2018) quantify the coherence of dialogue using the semantic similarity between each utterance and its preceding utterances. This similarity is estimated, for example, by the cosine similarity between an utterance vector and a context vector where those vectors are the average of their pre-trained word embeddings. Vakulenko et al. (2018) measure dialogue coherence based on the consistency of new concepts introduced in a dialogue with background knowledge. Similarly, Dziri et al. (2019) utilize a natural language inference model to assess the content consistency among utterances as an indicator for dialogue coherence. However, these approaches lack dialogue-relevant information to measure coherence.
Our MTL-based approach solves these issues: (i) it benefits from DAs and semantics of utterances to measure dialogue coherence by optimizing utterance vectors for both DAP and coherence assessment, and (ii) it uses DA labels to define an
auxiliary task for training the DiCoh model using MTL, instead of utilizing them in a pipeline. Therefore, it efficiently mitigates the need for explicit DA labels as inputs during coherence assessment.
We represent a dialogue between two speakers as a sequence of utterances,  $\text{dial} = [utt_1, \dots, uttm]$ . We address the problem of designing a coherence model, DiCoh, which assigns a coherence score to  $\text{dial}$ ,  $s_{\text{dial}} = \text{DiCoh}(\text{dial})$ . Given a pair of dialogues  $\phi = (dial_i, dial_j)$ , our DiCoh model ideally assigns  $s_{\text{dial}_i} > s_{\text{dial}_j}$  if and only if dialogue  $dial_i$  is preferred over dialogue  $dial_j$  according to their perceived coherence. Instead of using gold DA labels as inputs to DiCoh, we use them to define an auxiliary task and model, DAP, to enrich utterance vectors for DiCoh in an MTL scenario. Figure 2 shows a low-level illustration of our MTL-based approach.
Utterance encoder We use a word embedding layer, Emb, to transform the words in utterance  $u t t = [w_{1},\dots,w_{n}]$  to a sequence of embedding vectors  $E = [e_1,\dots,e_n]$ , where  $n$  is the number of words in  $u t t$ . The embedding layer can be initialized by any pre-trained embeddings to capture lexical relations. We use a Bidirectional recurrent neural network with Long Short-Term Memory cells, BiLSTM, to map embeddings  $E$  to encode words in their utterance-level context:
$$
E = E m b (u t t),
$$
$$
\begin{array}{l} H _ {u} = B i L S T M (E), \end{array} \tag {1}
$$
where  $H_{u}$  shows the hidden state vectors  $[h_1^u,\dots,h_n^u ]$  returned by BiLSTM. At word  $t$ ,  $h_t^u$  is the concatenation of hidden states of the forward  $\overrightarrow{h_t^u}$  and the backward LSTMs  $\overleftarrow{h_t^u}$ :
$$
h _ {t} ^ {u} = \left[ \overrightarrow {h _ {t} ^ {u}}; \overleftarrow {h _ {t} ^ {u}} \right]. \tag {2}
$$
We apply a self-attention mechanism, Atten, to the hidden state vectors in  $H_{u}$  to obtain the vector representation,  $u$ , of utterance  $u_{t}$ :
$$
u = \operatorname {A t t e n} \left(H _ {u}\right). \tag {3}
$$
Generally, the attention layer, Atten, for an input vector  $x$  is defined as follows:
$$
\beta_ {t} = x _ {t} * W,
$$
$$
\alpha_ {t} = \frac {\exp (\beta_ {t})}{\sum_ {t} \exp (\beta_ {t})}, \tag {4}
$$
$$
o = \sum_ {t} \alpha_ {t} * x _ {t},
$$
where  $W$  is the parameter of this layer, and  $o$  is its weighted output vector. Attention enables the utterance representation layer to encode an utterance by the weighted sum of its word embeddings. It is worth noting that the parameters of the utterance encoder are shared for representing all utterances in a dialogue.
DiCoh model For an input dialogue  $\text{dial} = [utt_1, \dots, uttm]$ , the output of the utterance representation encoder is a sequence of vectors, i.e.,  $[u_1, \dots, u_m]$ . Our coherence assessment model (DiCoh) combines these vectors by a BiLSTM to obtain dialogue-level contextualized representations of utterances. Then, a self-attention (Equation 4) with new parameters computes the weighted average of the contextualized utterance vectors to encode the dialogue:
$$
\left[ h _ {1} ^ {d}, \dots , h _ {m} ^ {d} \right] = B i L S T M \left(\left[ u _ {1}, \dots , u _ {m} \right]\right), \tag {5}
$$
$$
d = A t t e n ([ h _ {1} ^ {d}, \dots , h _ {m} ^ {d} ]).
$$
A linear feed-forward layer,  $FF$ , maps the dialogue vector,  $d$ , to a dialogue coherence score,  $s_{dial}$ :
$$
s _ {d i a l} = F F (d). \tag {6}
$$
DAP model Our DAP model, which is used to solve the auxiliary DAP task, is a softmax layer which maps an utterance vector,  $u$ , to a probability distribution  $p_a$  over DA labels  $A$ :
$$
p _ {a} (u) = \operatorname {s o f t m a x} \left(W _ {| u | \times | A |} * u + b\right), \tag {7}
$$
where  $W_{|u|\times |A|}$  shows the weights of the softmax layer,  $|u|$  is the size of the utterance vector,  $|A|$  is the number of DA labels, and  $b$  is the bias.
As illustrated in Figure 1, our main idea is to benefit from the DAP task for improving the performance of the dialogue coherence model by using them in a multi-task learning scenario. We also assume that each utterance  $u t t_{k}$  is associated with DA label,  $a_{k}$ , during training but not during evaluation.
We define a loss function for each task, and then use their weighted average as the total loss. The DAP loss function for dialogue dial is the average cross-entropy:
$$
\mathcal {L} _ {d a} ^ {d i a l} = - \frac {1}{m} \sum_ {k \in (1, \dots , m)} a _ {k} * \log \left(p _ {a} \left(u _ {k}\right)\right), \tag {8}
$$
Figure 2: A low-level illustration of our MTL-based approach to dialogue coherence assessment. The input is dialogue pair  $p = (dial_i, dial_j)$ . Dashed items represent losses. Models' parameters are shared among dialogues.
where  $m$  is the number of utterances in dialogue, and  $a_{k}$  is the one-hot vector representation of the gold DA label associated with the  $k^{th}$  utterance.  $\log (p_a)$  is the natural log of probabilities over DA labels, which is obtained in Equation 7.
Inspired by preference learning approaches (e.g. the proposed method by Gao et al. (2019) for text summarization) we define the loss function for coherence assessment through pairwise comparisons among dialogues. Given dialogue pair  $\phi = (dial_{i},dial_{j})$  and its preference coherence label,
$$
l ^ {c} = \left\{ \begin{array}{l l} 0 & \text {i f d i a l _ {i} i s p r e f e r e d o v e r d i a l _ {j}}, \\ 1 & \text {o t h e r w i s e}, \end{array} \right. \tag {9}
$$
the coherence loss is:
$$
\mathcal {L} _ {c o h} ^ {\phi} = \max  \{0, 1 - s _ {\phi [ l ^ {c} ]} + s _ {\phi [ 1 - l ^ {c} ]} \}, \tag {10}
$$
where  $[.]$  is the indexing function. More formally,  $s_{\phi [l^c ]}$  and  $s_{\phi [1 - l^c ]}$  are the coherence scores of the coherent and incoherent dialogue in pair  $\phi = (dial_i,dial_j)$ , respectively. Finally, the total loss value is the weighted combination (Kendall
et al., 2018) of the above losses:
$$
\mathcal {L} = \frac {\mathcal {L} _ {c o h} ^ {\phi}}{\gamma_ {1} ^ {2}} + \frac {\left(\mathcal {L} _ {d a} ^ {d i a l _ {i}} + \mathcal {L} _ {d a} ^ {d i a l _ {j}}\right)}{\gamma_ {2} ^ {2}} + \log (\gamma_ {1}) + \log (\gamma_ {2}), \tag {11}
$$
where  $\mathcal{L}_{da}^{dial_i}$  and  $\mathcal{L}_{da}^{dial_j}$  are the losses of DAP for dialogues in pair  $\phi = (dial_i, dial_j)$ ,  $\gamma_1$  and  $\gamma_2$  are trainable parameters to balance the impact of losses. We compute the gradient of  $\mathcal{L}$  to update the parameters of both DiCoh and DAP models.
4.1 Dialogue Corpora
We compare our approach with several previous dialogue coherence models on DailyDialog (Li et al., 2017) and SwitchBoard (Jurafsky and Shriberg, 1997) as two benchmark English dialogue corpora. Table 2 shows some statistics of these corpora.
DailyDialog contains human-written dialogues about daily topics (e.g. ordinary life, relationships, work, etc) collected by crowd-sourcing. Crowd-workers also annotated utterances with generic DA labels from the set {Inform, Question, Directive, Commissive}. Dialogues in this corpus contain a few utterances  $(\approx 8)$  making them more on topic
DailyDialog	SwitchBoard
# dialogues	13,118	1,155
# DA labels	4	42
avg. # utter. per dialogue	7.9	191.9
avg. # words per utter.	14.6	9.26
Table 2: The statistics of the DailyDialog and SwitchBoard corpora.
and less dispersed. However, utterances are long in terms of the number of words  $(\approx 15)$ .
SwitchBoard contains informal English dialogues collected from phone conversations between two mutually unknown human participants. The participants were given only one of 70 possible topics as initial topic to start a conversation but they were free to diverge from that topic during the conversation. So, there is no concrete topic associated with each dialogue in this dataset as it is the case for dialogues in DailyDialog.
DA labels in SwitchBoard are about 10 times more fine-grained than those in DailyDialog. For example, a question utterance in SwitchBoard may have a fine-grained DA label such as Yes-No-Question, Wh-Question, Rhetorical-Questions, etc. The distribution of these acts is however highly unbalanced in SwitchBoard: the most frequent act label makes up for  $36\%$  of the utterances in the corpus, the three most frequent acts together make up for  $68\%$  of the utterances, while most of the remaining act labels just make up for  $1\%$  or less of all the utterances.
On average, dialogues in SwitchBoard contain more utterances than those in DailyDialog (192 vs 8) but utterances in SwitchBoard are shorter than those in DailyDialog (9 vs 15). This means that dialogues in SwitchBoard are more likely to span different topics than the ones in DailyDialog. The utterances in DailyDialog are explicitly cleaned of any noise, like "uh-oh", or interruptions by the other speaker, as it is commonly the case for dialogues in SwitchBoard. While each dialogue turn of dialogues in DailyDialog contains only one utterance, dialogue turns in SwitchBoard may consist of several utterances. That is why we consider each dialogue as a sequence of dialogue utterances.
4.2 Problem-domains
The goal of our experiments is to assess if a coherence model assigns coherence scores to dialogues so that a more coherent dialogue obtains a higher score than a less coherent one. Since di
alogues in the examined corpora, i.e. DailyDialog and SwitchBoard, are not associated with any coherence assessment score, we synthetically define four perturbation methods to destroy the coherence of dialogues in these corpora, and create a set of dialogue pairs for training and testing coherence models.
We borrow Utterance Ordering (UO) and Utterance Insertion (UI) from previous studies on coherence assessment (Barzilay and Lapata, 2005; Cervone et al., 2018) and also introduce Utterance Replacement (UR), and Even Utterance Ordering (EUO) as more challenging and dialogue-relevant perturbation methods. Since each experiment follows a specific perturbation method, henceforth, we refer to these perturbations as problem-domains:
Utterance Ordering (UO) We randomly permute the order of utterances in dialogue. The original dialogue is preferred over the perturbed one.
Utterance Insertion (UI) We remove each utterance of a dialogue and then re-insert it in any possible utterance position in the dialogue. We assume that the original place of the utterance is the best place for the insertion. Therefore, a coherence model ideally discriminates the original dialogue from the perturbed ones, which are obtained by re-inserting the removed utterance in any utterance position except its original one. This problem-domain is more difficult to solve than UO as the distinction between dialogues is in the position of only one utterance.
Utterance Replacement (UR) We randomly replace one of the utterances in a dialogue with another utterance that is also randomly selected from another dialogue. The original dialogue is preferred over the dialogue generated by UR. Unlike the other problem-domains, which perturb the structure of a dialogue, this problem-domain perturbs the coherence of a dialogue at its semantic level.
Even Utterance Ordering (EUO) This problem-domain is similar to UO but here we re-arrange the order of utterances that are said by one speaker and keep the order of the other utterances, which are said by the other speaker, fixed. Therefore, EUO is more challenging and dialogue-relevant than UO. This problem-domain assesses to what extent coherence models capture the coherence among utterances that are said by one of the speakers in a dialogue.
4.3 Problem-domain Datasets
To create dialogue pairs for each problem-domain, we use the splits provided by the DailyDialog corpus; and for SwitchBoard we take  $80\%$  of dialogues for the training,  $10\%$  for the validation and  $10\%$  for the test sets. Following Cervone et al. (2018), for any dialogue in each set we create 20 perturbations where each of which makes two pairs with the original dialogue. Given dialogue diali and its perturbation dialj, we define two dialogue pairs:  $(\text{dial}_i,\text{dial}_j)$  with preference coherence label  $l^c = 0$  and  $(\text{dial}_j,\text{dial}_i)$  with label  $l^c = 1$ .
4.4 In problem-domain Evaluation
In this evaluation, we train, fine-tune, and evaluate our models on the training, validation, and test sets of each problem-domain. Note that these sets are constructed by the same perturbation method.
Compared coherence models We compare the following coherence models in this evaluation: (1) Random: This baseline model randomly ranks dialogues in an input dialogue-pair. (2) CoSim (Zhang et al., 2018; Xu et al., 2018): This model represents utterances by averaging the pre-trained embeddings of their words. Then, the average of the cosine similarities between vectors of adjacent utterances is taken as the coherence score. In this model, utterance vectors are made using content words by eliminating all stop words. (3) ASeq (Gandhe and Traum, 2016): This model relies only DAs transitions and is agnostic to semantic relationships (such as entity transitions) between utterances. Coherence features in this model are the probabilities of n-grams across the sequence of DAs associated with the utterances in dialogue. These features are supplied to a SVM to rank dialogues. (4) EAGrid (Cervone et al., 2018): This is the best performing model presented by Cervone et al. (2018) that benefits from both entity and DA transitions between utterances. It represents semantic relationships across utterances via a grid, whose rows are associated with utterances and all columns represent entities but one that represents DAs. Entities are a set of mentions that are extracted by a co-reference system. Entries at the intersections between entity columns and an utterance row represent the grammatical role of an entity in an utterance. The intersection of the DA column and an utterance shows the DA label of the utterance. Cervone et al. (2018) use grammatical role transitions of entities as well as DA label transitions
across utterances as indicative patterns for coherence. The frequencies of these patterns are taken as coherence features, which are supplied to Support Vector Machines (SVMs) to discriminate dialogues with respect to their coherence. (5) S-DiCoh: This is our coherence model, DiCoh, trained by only the supervision signal for coherence ranking, with the total loss  $\mathcal{L} = \mathcal{L}_{coh}^{\phi}$  (see Equation 11). This model does not benefit from DA information to enrich utterance vectors. (6) M-DiCoh: This is our full model trained by the proposed MTL using the supervision signals for both coherence ranking and DAP. The main advantage of this model is that it learns to focus on salient information of utterances for coherence assessment based on the given DAs for utterances.
We follow former coherence papers (Barzilay and Lapata, 2008; Guinaudeau and Strube, 2013; Mesgar and Strube, 2018; Cervone et al., 2018) and use accuracy as the evaluation metric. In our experiments, this metric equals the frequency of correctly discriminated dialogue pairs in the test set of a problem-domain.
$$
a c c = \frac {\# \text {o f c o r r e c t l y d i s c r i m i n a t e d d i a g o u g e p a i r s}}{\# \text {o f d i a g o u g e p a i r s}}. \tag {12}
$$
To reduce the risk of randomness in our experiments, we run each experiment five times with varying random seeds and report their average (Reimers and Gurevych, 2018).
Settings Each batch consists of 128 and 16 dialogue-pairs for the DailyDialog and SwitchBoard corpora, respectively. Utterances are zero-padded and masked. We use pretrained GloVe embeddings (Pennington et al., 2014) of size 300 wherever word embeddings are required (i.e., in CoSim, S-DiCoh, and M-DiCoh). For the CoSim model, we use the SMART English stop word list (Salton, 1971) to eliminate all stop words. For the ASeq model, we use bi-grams of DA labels to define the coherence features (Cervone et al., 2018). All parameters of the EAGrid model have the same value as the best performing model proposed by Cervone et al. (2018).