SlowGuess/NAACL_2022_OCR huggingface dataset:
Yujie Xing $^{1}$ , Jinglun Cai $^{2*}$ , Nils Barlaug $^{1}$ , Peng Liu $^{1}$ , Jon Atle Gulla $^{1}$
$^{1}$ Department of Computer Science, Norwegian University of Science and Technology
2Amazon AWS AI
{yujie.xing,nils.barlaug,peng.liu,jon.atle.gulla}@ntnu.no
cjinglun@amazon.com
Open-domain conversational systems are assumed to generate equally good responses on multiple domains. Previous work achieved good performance on the single corpus, but training and evaluating on multiple corpora from different domains are less studied. This paper explores methods of generating relevant responses for each of multiple multi-domain corpora. We first examine interleaved learning which intermingles multiple corpora as the baseline. We then investigate two multidomain learning methods, labeled learning and multi-task labeled learning, which encode each corpus through a unique corpus embedding. Furthermore, we propose Domain-specific Frequency (DF), a novel word-level importance weight that measures the relative importance of a word for a specific corpus compared to other corpora. Based on DF, we propose weighted learning, a method that integrates DF to the loss function. We also adopt DF as a new evaluation metric. Extensive experiments show that our methods gain significant improvements on both automatic and human evaluation. We share our code and data for reproducibility. $^{1}$
Recent work has achieved improvements in general performance for open-domain response generation (Vinyals and Le, 2015; Serban et al., 2017; Li et al., 2016; Xu et al., 2018). However, most studies are restricted to single-corpus training and evaluating, while there lacks studies for training and evaluating with multiple corpora from different domains. Single-corpus training has intrinsic limitations. For example, a corpus of everyday chats, e.g., the PersonaChat corpus (Dinan et al., 2019), does not cover technical topics discussed in
Context	What are you going to do on the re-mote system exactly?
PersonaChat	I am going to be a pilot. I am going to fly planes.
4 corpora (con-catenated)	I am going to go to the beach.
Table 1: Irrelevant responses generated from fine-tuned GPT-2. The GPT-2 model is fine-tuned respectively on PersonaChat / concatenated 4 corpora (OpenSubtitles, Twitter, Ubuntu, PersonaChat)
Fine-tune corpus	Test set
OSDB	Twitter	Ubuntu	PersonaChat
PersonaChat	478.8	159.6	264.7	19.6
4 corpora (concatenated)	392.8	110.7	199.2	19.0
Table 2: Imbalanced perplexity performance of fine-tuned GPT-2. The GPT-2 model is fine-tuned on PersonaChat / concatenated 4 corpora (OpenSubtitles, Twitter, Ubuntu, PersonaChat)
Ubuntu chatlogs (Lowe et al., 2015). A conversational system that learns only from PersonaChat or from multiple corpora without an appropriate technique is not likely to generate relevant responses for certain topics (see Table 1). Therefore, it is necessary for an open-domain conversational system to learn from multiple corpora, and to learn with good techniques.
Furthermore, the case of using a single small-scale open-domain corpus has apparent weaknesses. A common way of dealing with a small-scale corpus is through fine-tuning (Li et al., 2016; Akama et al., 2017; Chu et al., 2017). Fine-tuning on a single corpus tends to make the model overfit on that specific corpus while performing worse on other corpora. Table 2 shows the result of a GPT-2 model gaining good performance on PersonaChat while performing poorly on other corpora.
This paper explores how to train and evaluate on multiple corpora from different domains for the open-domain response generation task. We propose
* This work was done prior to the author joining Amazon.
1https://github.com/yujie-xing/
Balancing_Multi_Domain_Corpus_Learning_
for_Open_Domain_Response_Generation
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2104 - 2120
several methods to make a model generate relevant responses for each of the multiple corpora.
Since simply training multiple corpora one by one does not solve the imbalanced performance (as shown in Table 1 and 2), we first investigate interleaved learning, a method that intermingles the training data instead of simply concatenating, which ensures a model learns from all corpora evenly. We use this method as a baseline. Additionally, we explore two multi-domain learning methods: labeled learning and multi-task labeled learning. Labeled learning comes from a control technique in response generation (Li et al., 2016; Johnson et al., 2017; Yang et al., 2017). Previous works focus on controlling persona and style, while our method controls corpus's information with the corpus embedding. Multi-task labeled learning is inspired by works of domain adaption (Luan et al., 2017; Niu and Bansal, 2018; Chu and Wang, 2018), where multiple losses from both the corpus classifier and response generator are minimized. To the best of our knowledge, this paper is the first that uses corpus embeddings on the open-domain response generation task for multiple corpora.
Furthermore, we propose a novel weighted learning with Domain-specific Frequency (DF). DF is a word-level importance weight (Leopold and Kindermann, 2002) that assigns different weights (importance) to the same words from different corpora. In the training process, we weight the loss of a model with DF, so that the model focuses on the most important words for a specific corpus.
For automatic evaluation metrics, we eliminate the stop words and use ROUGE-1 (precision, recall, F1) (Lin, 2004) to measure the relevance of the generated responses. In addition, we adopt DF to see how relevant the generated response of a model is to a specific corpus. We will explain DF as an evaluation metric in Section 4.4. Results show that for overall performance, the best method (weighted learning) improves  $27.4\%$  on precision,  $45.5\%$  on recall, and  $34.1\%$  on F1. Further, it has at least  $20.0\%$  higher DF, stating that it uses more important words from the "correct" corpus. We also conduct an extensive human evaluation on 2400 generated responses. The human evaluation shows a highly significant  $(p < 0.001)$  improvement on all of our proposed methods, especially the weighted learning method.
We summarize our work as follows:
- We explore the problem of training and eval
uating on multiple corpora from different domains for open-domain response generation. The task is to make the conversational models generate relevant responses for each corpus.
- We examine several multi-domain corpora learning methods for their ability to solve the proposed task.
- We propose Domain-specific Frequency (DF) as in weighted learning and as an evaluation metric. DF distinguishes important words for each corpus and helps a model to focus on these important words in the training process.
Open-Domain Response Generation Recent work of open-domain response generation generally follows the work of Ritter et al. (2011) where the task is treated as a machine translation task, and many of them use a Seq2Seq structure (Sutskever et al., 2014) following previous work (Vinyals and Le, 2015; Shang et al., 2015; Sordoni et al., 2015). In recent years, substantial improvements have been made (Serban et al., 2017; Li et al., 2016; Wolf et al., 2019), and embeddings are used to control response generation on extra information such as persona (Li et al., 2016), profiles (Yang et al., 2017), coherence (Xu et al., 2018), emotions (Huang et al., 2018), and dialogue attributes like response-relatedness (See et al., 2019). However, there is a lack of work that uses embeddings to control response generation over multiple corpora. Our work follows the common models of open-domain conversational systems, while we study the problem of multiple corpora of different domains.
Multi-Domain Learning and Domain Adaption Multi-domain learning aims at making a conversational model learn from multiple domains to prevent the performance from degrading due to domain differences (Ben-David et al., 2007). There are two categories of solutions for multi-domain learning (Joshi et al., 2012): (i) capturing domain-specific characteristics in the parameters (Daumé III, 2007); (ii) capturing the relationship among different domains (Saha et al., 2011).
Some work of natural language generation and machine translation is related to multi-domain learning. Luan et al. (2017) and Niu and Bansal (2018) use multi-task learning for domain adaption respectively on speaker-role and politeness. Wen
et al. (2016) and Akama et al. (2017) utilizes fine-tuning as a common way of domain adaption for language generator and style transferer. For machine translation, in order to deal with the mixed-domain parallel corpus, Zeng et al. (2018) adjust the weights of target words in the training objective based on their relevance to different domains. We differ in that we propose DF and we deal with the response generation task. Chu et al. (2017) propose mixed fine-tuning, which adds the out-of-domain pre-training data to the fine-tuning dataset, and they observe an improvement of performance. In this paper, we also mix small-scale fine-tuning datasets with out-of-domain training data, while the data we add is not necessarily used during pretraining. Shi et al. (2015) state that fine-tuning can be done by placing the corpus to be fine-tuned at the end of the entire corpus, which is an extension of curriculum learning proposed by Bengio et al. (2009). We also explore how the order of multiple corpora influences the result, but our focus is on balancing performance. Recently, Smith et al. (2020) investigated blending conversational skills with knowledge and empathy skills, where they mix 3 corpora. They focus on selecting appropriate skills and they propose a blended corpus with labels, while we focus on generating responses that are most relevant to a specific corpus.
3 Base Models
We use two base models: an LSTM Seq2Seq model with attention (Hochreiter and Schmidhuber, 1997; Sutskever et al., 2014; Bahdanau et al., 2015) and a pre-trained GPT-2 model (Radford et al., 2019). The LSTM Seq2Seq model with attention is a common model for conversational systems (Li et al., 2016; See et al., 2019), and the GPT2 model is a state-of-the-art model for the response generation task (Zhang et al., 2020; Zhao et al., 2020).
The basic task of response generation is to predict the next word given the past and current words of the context and response, and to make the generated response as similar to the original response as possible. The task can be described as follows. Probability of response  $Y$  given context  $X$  is predicted as:
$$
P (Y | X) = \prod_ {t = 1} ^ {n} P (y _ {t} | y _ {1}, \dots , y _ {t - 1}, X), \quad (1)
$$
where  $X = x_{1},\ldots ,x_{m}$  and  $Y = y_{1},\ldots ,y_{n}$  is a context-response pair.
3.1 LSTM Seq2Seq Model with Attention
We simplify an LSTM with attention unit as  $LSTM^{*}$  since it is well introduced in previous work (Li et al., 2016). We calculate the hidden vector  $h_t$  at step  $t$  as:
$$
h _ {t} = L S T M ^ {*} \left(h _ {t - 1}, E \left(z _ {t}\right)\right), \tag {2}
$$
where  $h_{t-1} \in \mathbb{R}^{dim}$  is the hidden vector at step  $t - 1$ ,  $dim$  is the dimension of hidden vectors, and  $E(z_t)$  is the word embedding for word  $z_t \in (x_1, \ldots, x_m, y_1, \ldots, y_{n-1})$ . We apply dot multiple in the attention mechanism when calculating the context vector  $c_t$ :
$$
c _ {t} = H \cdot \left(\operatorname {s o f t m a x} \left(H ^ {\top} \cdot h _ {t}\right)\right)
$$
where  $H \in \mathbb{R}^{d \times m}$  is the concatenation of hidden vectors from the encoder.  $c_t$  is input to the next step  $t + 1$  in the decoder. Each token's hidden vector  $h_t$  in the decoder is combined with  $c_t$  through a linear layer and an activation to predict the next token.
3.2 GPT-2
As for GPT-2, we follow the adaption of Wolf et al. (2019). The transformer block of GPT-2 captures the relation of multiple words in one sentence, which largely follows Vaswani et al. (2017). The hidden vector to be input to the transformer block is calculated as:
$$
h _ {0 [ t ]} = E \left(X, Y _ {[ 1: t ]}\right) + \left(E _ {0}, E _ {1}\right) + W _ {p}, \tag {3}
$$
where  $Y_{[1:t]}$  is  $(y_1, \ldots, y_t)$ ,  $E(X, Y_{[1:t]})$  is the subword embedding for context  $X$  and response  $Y_{[1:t]}$ .  $E_0$  and  $E_1$  are dialogue-state embeddings, which tutor the model to distinguish between contexts and responses.  $W_p$  is a pre-trained position embedding. The probability of the subword to generate is then calculated as:
$$
h _ {[ t ]} = \text {t r a n s f o r m e r \_ b l o c k} \left(h _ {0 [ t ]}\right) \tag {4}
$$
$$
P (y) _ {t + 1} = \operatorname {s o f t m a x} \left(E ^ {\top} \left(h _ {[ t ]}\right)\right), \tag {5}
$$
where  $y \in V$ , and  $V$  stands for the sub-word vocabulary. We simplify the structure of transformer block as transformer_block. In the block, a mask is filled in the attention matrix, which bans past words from attending to future words. This ensures that the model follows the traditional language modeling. The hidden vector of  $t_{\mathrm{th}}$  sub-word is used to generate the probability distribution for the vocabulary  $(P(y), y \in V)$  for  $(t + 1)_{\mathrm{th}}$  sub-word.  $E^{\top}$  means that the model uses the sub-word embeddings in calculating sub-word probabilities for generation (Press and Wolf, 2017).
4 Proposed Methods
4.1 Interleaved Learning
Interleaving is a concept in cognitive psychology proven to be efficient for learning (Kornell and Bjork, 2008): intermingling learning material of different topics helps students to gain better learning results than learning the material topic by topic. Previous work from machine learning also shows that training order greatly influences the performance (Bengio et al., 2009). When the training is conducted on a simple concatenation of multiple corpora, the model tends to concentrate on the last corpus (Shi et al., 2015). To address this issue, we propose interleaved learning as an alternative: each time we collect one context-response pair from each of the corpora, and we randomly shuffle them. For example, if there are 3 corpora  $(a_{1}, a_{2}, \ldots)$ ,  $(b_{1}, b_{2}, \ldots)$ ,  $(c_{1}, c_{2}, \ldots)$  where  $a_{i}, b_{i}$  and  $c_{i}$  are context-response pairs, the resulting mixed corpus might be  $(b_{1}, a_{1}, c_{1}, c_{2}, b_{2}, a_{2}, \ldots)$ . Interleaved learning guarantees that the combined corpus is evenly distributed, which helps the model learn from multiple corpora evenly.
4.2 Labeled Learning
We propose our labeled learning as follows: each corpus is assigned a randomly initialized unique embedding, and the conversational model learns these embeddings together with conversations during the training period. We denote these embeddings as "corpus embedding", or  $E_{c}$ . A model captures each corpus's characteristics through the corpus embedding and uses it to control the generated responses. To know which corpus embedding to use, each context is labeled with which corpus it comes from, and these labels are provided to the model both in the training and generation period. We propose an approach for each of our base models for encoding corpus embeddings.
For the LSTM model, following Li et al. (2016), we input the corpus embedding  $E_{c}$  into the first layer of the decoder LSTM at every step, together with the response words. Calculation of a hidden vector  $h_t$  in the decoder LSTM is then adapted to:
$$
h _ {t} = L S T M ^ {*} \left(h _ {t - 1}, E \left(y _ {t}\right), E _ {c}\right). \tag {6}
$$
The structure is illustrated in the dashed red rectangle of Figure 1a.
For the GPT-2 model, our method is based on Wolf et al. (2019). Instead of two kinds of dialogue
state embeddings (context embedding  $E_0$  and response embedding  $E_1$ ), we replace the response embedding with corpus embeddings  $E_c$ . As a result, the model is aware of which corpus the response belongs. Calculation of a hidden vector to be input to the transformer block is adapted to:
$$
h _ {0 [ t ]} = E \left(X, Y _ {[ 1: t ]}\right) + \left(E _ {0}, E _ {c}\right) + W _ {p}. \tag {7}
$$
The structure is illustrated in Figure 1b.
4.3 Multi-Task Labeled Learning
Labeled learning needs corpus labels for both training and generation processes. To avoid providing labels in the generation process, we combine multitask learning with labeled learning on multiple corpora. Here, the conversational model has to predict by itself which corpus a context belongs to, which is expected to result in worse performance, but less information is required. In the encoder, we have a classifier layer that uses the sum of hidden vectors from the encoder  $(\sum H)$  to predict the corpus of a context. The loss of the classifier is calculated as:
$$
\mathcal {L} _ {c} = - \log \left(\operatorname {s o f t m a x} \left(\left(\sum H\right) \cdot W _ {[ c ]}\right)\right), \tag {8}
$$
where  $W_{[c]} \in \mathbb{R}^{dim}$  is the part from the classifier layer for target corpus  $c$ .  $\mathcal{L}_c$  is summed up with the loss from the response generator. The predicted corpus embedding is input into the decoder like labeled learning (see Section 4.2). The simplified structure is illustrated in Figure 1a.
4.4 Document-specific Frequency (DF)
We propose Domain-specific Frequency (DF) to measure how important a word is with respect to a different corpus under a collection of corpora. DF is used for weighted learning and evaluation. It is calculated as follows:
$$
f (w) _ {d} = \operatorname {f r e q} (w) _ {d} - \min  _ {v} \left\{\operatorname {f r e q} (v) _ {d} \right\} \tag {9}
$$
$$
\mathrm {d f} (w) _ {d} = \left\{ \begin{array}{l l} 0 & f (w) _ {d} = 0 \\ \frac {f (w) _ {d}}{\sum_ {d \in D} f (w) _ {d}} & f (w) _ {d} \neq 0 \end{array} \right. \tag {10}
$$
$$
\mathrm {D F} (w) _ {d} = \frac {\mathrm {d f} (w) _ {d}}{\max  _ {v} \left\{\mathrm {d f} (v) _ {d} \right\}}, \tag {11}
$$
where  $\operatorname{freq}(w)_d$  is the relative frequency of a word  $w$  in a corpus  $d$ , and  $D$  represents the set of all corpora. It is easy to see from Equation 10 that  $\mathrm{DF}(w)_d$  represents the importance of word  $w$  for corpus  $d$  compared to other corpora. For a word  $w$  that frequently appears in corpus  $d$  but seldom
(a) Structure of multi-task labeled learning on LSTM model
Figure 1: Adapted models with labeled learning, multi-task labeled learning and weighted learning
(b) Corpus embeddings with sub-word embeddings on GPT-2
(c) Structure of weighted learning
in other corpora (e.g., "upgrade" from Ubuntu corpus),  $\sum_{d\in D}f(w)_d$  is close to  $f(w)_{d}$ , making  $\mathrm{DF}(w)_d$  approach 1. A word that frequently appears in all corpora (e.g., "I", "you") is punished, resulting in a lower  $\mathrm{DF}(w)_d$ . A word that seldom appears in corpus  $d$  but frequently appears in other corpora (e.g., "music" seldom appears in Ubuntu corpus, but is common in other corpora) has the lowest  $\mathrm{DF}(w)_d$ . Words that appear minimal times (e.g., once) in a corpus are ignored with Equation 9. Words that appear few times (e.g., twice or three times) are not dealt with, yet they are not of great influence in our experiments. We apply a normalization in the final step (Equation 11) to make  $\mathrm{DF}(w)_d$  of each corpus  $d$  range from 0 to 1.
We show  $\mathrm{DF}(w)_{\mathrm{Ubuntu}}$  and  $\mathrm{DF}(w)_{\mathrm{PersonaChat}}$  of some words in Table 3. We also show the results of TF-IDF (log normalization variant), a commonly used word importance weight, as a comparison. As expected, for the corpus Ubuntu and PersonaChat, most unique words  $w$  have very different  $\mathrm{DF}(w)_{\mathrm{Ubuntu}}$  and  $\mathrm{DF}(w)_{\mathrm{PersonaChat}}$ . Unique words of each corpus get the highest values for the corresponding corpus, like "upgrade" for the Ubuntu corpus and "music" for the PersonaChat corpus; these words receive the lowest values for incorrect corpora, like "upgrade" for PersonaChat and "music" for Ubuntu. The stress on unique words makes DF more suitable for our task.
Weighted Learning with DF Weighted learning weights the loss of the predication  $y^\prime$  for each target word  $w$  using  $\mathrm{DF}(w)_d$ . In the training period, each context is labeled with the corpus  $d$  it belongs
Word	TF-IDF(%)	DF(%)	αDF(α=100)
Ubuntu	PersonaChat	Ubuntu	PersonaChat	Ubuntu	PersonaChat
i	100.0	62.6	20.8	42.1	2.6	7.3
to	64.6	32.8	26.9	24.9	3.8	3.1
it	83.2	21.7	38.5	14.5	5.1	2.1
laptop	5.4	0.2	89.8	4.5	76.0	1.0
upgrade	6.8	0.1	95.6	0.4	91.2	1.0
file	15.7	0.1	96.0	0.3	86.4	0
windows	12.2	0.1	97.1	0.1	86.3	1.0
ubuntu	27.5	0	99.9	0	99.5	0
teacher	0.1	2.2	0.7	77.8	1.0	53.5
music	1.5	7.6	4.8	82.9	1.2	49.1
travel	0.1	3.1	0.3	88.9	1.0	57.1
hobby	0.1	1.6	0.6	94.3	1.1	81.7
hiking	0	1.5	0	97.6	0	91.8
Table 3: Normalized TF-IDF  $(\%)$ , DF  $(\%)$  and  $\alpha$ DF of some words for Ubuntu and PersonaChat (more examples on other corpora can be found in Section A)
to, so that the model can use the  $\mathrm{DF}(w)_d$  of the corresponding corpus. Here DF is calculated only on the training sets. In the generation step, corpus labels are not provided, so DF is not used. The loss is weighted as follows:
$$
\mathcal {L} _ {\text {w e i g h t e d}} = \mathrm {D F} (w) _ {d} \cdot \left(- \log \left(\operatorname {s o f t m a x} \left(y _ {w} ^ {\prime}\right)\right)\right), \tag {12}
$$
where  $y_w'$  represents the model's predicted score for the target word  $w$ . With the weighted loss, the model concentrates on words that are important to the corpus of the current context, and focuses less on frequent words or words that are not important to the current corpus. The structure is illustrated in Figure 1c.
Evaluation with DF For the generated responses to be relevant to a specific corpus, they have to
be similar to that corpus, which includes using important words of that corpus (e.g., responses generated for the Ubuntu corpus should have more technical words than other corpora). Thus, we propose DF as an evaluation metric that shows to what extent the generated responses use important words of the corresponding corpus. We want to decrease the influence of common words like “i”, “to”, etc., and thus address the important words. So we adopt exponential DF with  $\alpha$  as the base ( $\alpha$ DF):
$$
\alpha \mathrm {D F} (w) _ {d} = \left\{ \begin{array}{l l} 0 & \mathrm {D F} (w) _ {d} = 0 \\ \alpha^ {\mathrm {D F} (w) _ {d}} & \mathrm {D F} (w) _ {d} \neq 0, \end{array} \right. \tag {13}
$$
where  $\alpha$  is a constant.  $\alpha \mathrm{DF}(w)_d$  rescales  $\mathrm{DF}(w)_d$  by exponent with  $\alpha$  as a base. In our experiments, we set  $\alpha$  to be 100, which transforms the range of the metric from (0,1) to (0,100). This makes the difference between high and low  $\alpha \mathrm{DF}$  more significant than DF and gives a 100-scale score. For each corpus  $d\in D$ , we average  $\alpha \mathrm{DF}(w)_d$  on word  $w$  from the generated responses of each test set, which gives us  $\alpha \mathrm{DF}_d$  scores  $(d\in D)$  for each test set. Ideally, the generated responses of a specific corpus  $d$  should have a higher  $\alpha \mathrm{DF}_d$  score and lower  $\alpha \mathrm{DF}_{\overline{d}}$  score  $(\overline{d}\in \{d^{\prime}\in D\mid d^{\prime}\neq d\})$ . For example, generated responses of the Ubuntu test set should have a higher  $\alpha \mathrm{DF}_{\mathrm{Ubuntu}}$  score, while a lower  $\alpha \mathrm{DF}_{\overline{\mathrm{Ubuntu}}}$  score  $(\overline{\mathrm{Ubuntu}}\in \{d^{\prime}\in D\mid d^{\prime}\neq \mathrm{Ubuntu}\})$ .  $\alpha \mathrm{DF}_d$  scores for responses from the original test sets are the standard scores.
We show  $\alpha \mathrm{DF}(w)_{\mathrm{Ubuntu}}$  and  $\alpha \mathrm{DF}(w)_{\mathrm{PersonaChat}}$  (calculated purely on test set) in Table 3. As expected,  $\alpha \mathrm{DF}$  has a more significant difference between important words and common words.