SlowGuess/NAACL_2022_OCR huggingface dataset:
Bishal Santra\* IIT Kharagpur
Sumegh Roychowdhury  
IIT Kharagpur
Aishik Mandal IIT Kharagpur
Vasu Gurram  
IIT Kharagpur
Atharva Naik  
IIT Kharagpur
Manish Gupta\* Microsoft, India
Pawan Goyal IIT Kharagpur
$\spadesuit$  bsantraigi  $\text{一}$  gmail.com,  $\clubsuit$  gmanish  $\text{一}$  microsoft.com,  $\spadesuit$  pawang  $\text{一}$  cse.iitkgp.ac.in
Although many pretrained models exist for text or images, there have been relatively fewer attempts to train representations specifically for dialog understanding. Prior works usually relied on finetuned representations based on generic text representation models like BERT or GPT-2. But such language modeling pretraining objectives do not take the structural information of conversational text into consideration. Although generative dialog models can learn structural features too, we argue that the structure-unaware word-by-word generation is not suitable for effective conversation modeling. We empirically demonstrate that such representations do not perform consistently across various dialog understanding tasks. Hence, we propose a structure-aware Mutual Information based loss-function DMI (Discourse Mutual Information) for training dialog-representation models, that additionally captures the inherent uncertainty in response prediction. Extensive evaluation on nine diverse dialog modeling tasks shows that our proposed DMI-based models outperform strong baselines by significant margins.
Representation learning has transformed how we can apply machine learning to solve real-world problems. However, despite a vast body of research on pretrained language representations, there have been relatively fewer attempts to train representations specifically for dialog understanding. Prior works mostly relied on finetuned representations based on generic models like BERT (Devlin et al., 2019) or GPT-2 (Radford et al., 2019). In our experiments, we demonstrate that such representations do not perform uniformly across various dialog understanding tasks such as dialog-act classification, intent detection or dialog evaluation.
On the other hand, prior works on pretraining large-scale dialog models focused mainly on open-domain generation. These works evaluated their
models only on dialog generation (Zhang et al., 2020; Roller et al., 2021; Adiwardana et al., 2020) or tasks related directly to the pretraining objective (Henderson et al., 2020; Gao et al., 2020). Their effectiveness on other dialog understanding tasks like act classification or intent detection remains unexplored. So we ask the following research question: Can we learn enriched representations directly at the pretraining phase that are specifically helpful for dialog understanding?
Existing language modeling (causal or masked) pretraining objectives unfortunately are not the best to model dialogs for these reasons: (1) The model is not directly trained to learn the content discourse structure (e.g., context-response in dialogs). (2) Such models are trained to generate the response word-by-word rather than predicting a larger unit. (3) The inherent one-to-many nature of dialog generation implies that the encoding model should be able to capture uncertainty in the response prediction task, that such models ignore.
Hence, in this paper, we propose pretraining objectives for improved dialog modeling that turn the discourse-level organizational structure of texts from natural sources (e.g., documents, dialogs, or monologues) into a learnable objective. We call this objective the Discourse Mutual Information (DMI). The key insight towards the design of our pretraining objective is to capture representations that can account for a meaningful conversation out of a specific ordered sequences of utterances. We hope that a discourse-level pretraining objective with conversational data would guide the model to learn complex context-level features. For example, in Fig. 1, we illustrate the differences between standard language modeling (causal or masked) based pretraining objectives and a discourse-level reasoning task.
The second research question that we ask is whether discourse-level features learned using self-supervised pretraining outperform word-level pre
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1718 - 1734 July 10-15, 2022 ©2022 Association for Computational Linguistics
Figure 1: Possible reasoning involved in two types of pretraining: Word-level (left), Discourse-level (right). In a discourse-level reasoning task, the immediately preceding utterance may not be enough for understanding the full context. To predict the correct response, the model will need to capture both the larger context, in this case the topic of discussion, and the intent (e.g., asking for details) of the preceding utterance. In comparison, word-level reasoning is often easier and can be solved using local reasoning. Each of the three masked-words, in the left image, could have been predicted with reasonable confidence without any more information than the utterance itself.
training objectives for downstream dialog understanding tasks. Experimentally, we show that representations learned using the proposed objective function are highly effective compared to both existing discriminative as well as generative dialog models. In terms of various dialog understanding tasks, our models achieve state-of-the-art performances in several tasks (absolute improvements up to  $8.5\%$  and  $3.5\%$  in task accuracies in probing and finetuning setups, resp.) and perform consistently well across a variety of dialog understanding tasks, whereas baseline models usually have a rather imbalanced performance across tasks.
Overall, our main contributions are as follows.
- We propose DMI, a novel information-theoretic objective function for pretraining dialog representation.
- We release pretrained dialog-representation models in three different sizes (small, medium and base) based on our proposed self-supervised learning objectives<sup>1</sup>.
- We extensively evaluate our DMI based representations on multiple open-domain downstream tasks like intent detection, dialog-act classification, response retrieval, dialog reasoning, and response-generation evaluation, and beat state-of-the-art across nine tasks in both probe as well as finetune setups.
2.1 Dialog System Pretraining
There have been quite a few efforts towards utilizing existing representations or developing new pretrained models for dialog systems. While BERT (Devlin et al., 2019), ELMo (Peters et al., 2018), GPT-2 (Radford et al., 2019) and other general purpose large-scale pretrained networks are not specific to dialogs, transfer learning from such models could be reasonable. Basic language understanding capability available through these representations helps to get decent performance on many dialog-understanding tasks (Hosseini-Asl et al., 2020).
On the other hand, there have been various works on pretraining dialog specific representations or large-scale generation models. We summarize the properties of various previously proposed dialog-representation learning models in Table 1. DialoGPT (Zhang et al., 2020), Meena (Adiwardana et al., 2020) and BlenderBot (Roller et al., 2021) are large-scale Transformer-based language models, which are trained to generate the gold-response (as per the dataset) given a dialog context. ContextPretrain (Mehri et al., 2019), ConveRT (Henderson et al., 2020) and ConvFiT (Vulic et al., 2021) are trained on the response retrieval task using MultiWoz or Reddit conversations. DEB or Dialog Evaluation using BERT (Sai et al., 2020) is a model based on extended pretraining of the BERT architecture using Reddit data. DialogRPT (Gao et al., 2020), on the other hand, is pretrained to predict human-feedback (e.g., upvotes and downvotes) on comments to Reddit threads. This model is initial-
To access the pretrained dialog representation models and the source codes, please visit https://bsantraigi.github.io/DMI
Model	Training Data Size	Pretraining Obj.	Architecture	Param	Downstream Task
DialogGPT_Small	147M Dialogs	CE	GPT-2	125M	Generation /w MMI
DialogRPT	133M CR pairs	Response Ranking	DiaGPT	345M	Human Feedback Prediction
Blenderbot_Small	1.5B comments	CE	Tr. S2S	90M	Generation
Meena ‡	40B words, 341 GB text	CE	Evolved Tr. S2S	2.4B	Generation
ContextPretrain ‡	10k Dialogs, MultiWoz	NUR, NUG, MUR, I2	HRED	-	Multiwoz (DST, Act, NUG, NUR)
DEB	727M Dialogs	MLM, NSP	BERT	110M	Adv/Random Dialog Evaluation
ConveRT †	727M Dialogs	Response Selection	Tr. Encoder	29M	Response Selection
ConvFiT ‡	8% of 727M Dialogs + Intent data	Response Selection	BERT	110M	Intent Detection
DMI_Base	7.5-10% of 727M Dialogs	InfoNCE-S	Tr. Encoder	124M	9 Dialog-NLU tasks
Table 1: Survey of Pretrained Dialog Models. NUR: next utterance retrieval, NUG: next utterance generation, MUR: masked utterance retrieval, I2: inconsistency identification, CR: Context-response, S2S: Seq2Seq, Tr.: Transformer, CE: Cross-entropy, HRED: Hierarchical RNN Encoder-Decoder. † Pretrained checkpoints available but only for inference. ‡ Both source-code and checkpoints are not available.
ized using the weights of DialogGPT model. Wu et al. (2020) thoroughly investigate these existing pretrained representations, both generic and dialog specific, for understanding their effectiveness on various goal-oriented dialog-understanding tasks.
2.2 Self-supervised Representation Learning with InfoMax
Mutual Information maximization (InfoMax) is one of the popular approaches for self-supervised learning, first used by Oord et al. (2018) and Belghazi et al. (2018). Oord et al. (2018) proposed InfoNCE loss which is an estimator for lower bound to mutual information (MI) between two continuous-valued random variables. InfoNCE has also been used for other NLP applications like training sentence embeddings (SIMCSE (Gao et al., 2021)), question answering (QA-InfoMax (Yeh and Chen, 2019)), etc. Other estimators for mutual information have also been proposed like MINE (Mutual Information Neural Estimator) (Belghazi et al., 2018) and SMILE (Song and Ermon, 2020). In general, these estimators are also broadly studied in contrastive Learning (CL) literature for training both self-supervised (Mikolov et al., 2013; Devlin et al., 2019; Liu et al., 2019; Gao et al., 2021; Henderson et al., 2020; Vulic et al., 2021) and supervised models (Schroff et al., 2015; Gunel et al., 2020). Some prior works in the dialog generation domain have used the concept of mutual information to design loss functions or scoring mechanisms to improve specificity of the generated responses (Li et al., 2016; Yoo et al., 2020). These works predominantly used MI either as a regularizer, along with cross entropy loss, or as a scoring function for ranking generated responses in a post-processing step. In the next section, we derive our pretraining loss function DMI for conversational texts from an information-theoretic perspective.
3 Discourse Mutual Information
We define Discourse Mutual Information (DMI) as the mutual information between two random variables representing two different segments within the same discourse. This is a general concept that can be applied to any form of discourse, no matter the domain or type of signal. In this paper, we focus on dialog type discourses and representation learning for conversational texts. We define two random variables for the contexts  $(C)$  and responses  $(R)$  that jointly construct a valid conversation. Conversations between humans represent samples from the joint distribution  $P_{CR}$  of  $C$  and  $R$ . We pose the following learning problem, "learn continuous representations for the textual random variables  $C$  and  $R$  such that the true mutual information between  $C$  and  $R$  can be closely estimated."
In the remainder of this section we show that, if the lower bound on MI estimated by some representations of context and response is close to the true value, the representation of the context would be as predictive of the response as the natural language form itself. Existing generative training objectives as used in DialogGPT or Blenderbot are extremely focused on predicting target response only. Per-word cross-entropy loss, used for training these models, fails to take into account the inherent uncertainty in the context-to-response generation function. Adapting context representations so as to predict the target responses optimally, helps our proposed DMI-based models learn better dialog representations applicable to a versatile set of dialog understanding tasks.
Mutual Information between two random variables is defined as the reduction in uncertainty/entropy of one of the random variables by having knowledge about the value of the other random variable. Mathematically, this is written as  $I(X;Y) = H(X) - H(X|Y) = H(Y) - H(Y|X)$ .
Figure 2: Base Pretraining Architecture for DMI. In our implementations of the model,  $f_{\phi}$  denotes the transformer (Vaswani et al., 2017) based encoders. Context and response encoders share all parameters for efficient learning.  $d$  denotes sample dialogs from the training dataset.
Objective Function Formulation Let  $E_{c}$  and  $E_{r}$  be the representations for  $C$  and  $R$  based on some encoder. Using the data processing inequality from Information theory (Cover, 1999), we have
$$
I (C; R) \geq I \left(E _ {c}; E _ {r}\right) \tag {1}
$$
This tells us that MI between any encoded version of  $C$  and  $R$  will always be less or equal than the true mutual information. The equality will hold if  $E_{c}$  and  $E_{r}$  are both fully-invertible encoding processes (as opposed to representations which are lossy or compressive, and inversion is thus not possible). However, neural networks generally embed the data points in a low dimensional manifold by learning robust features that can represent the data points efficiently. Because of this, neural representations are usually not invertible. Now, the exact computation of MI is not possible for continuous-valued random variables. In recent years, various variational lower bounds have been proposed for estimating MI between continuous-valued random variables. Including the MI estimator  $(\hat{I}_{\theta})$ , the overall relation becomes
$$
I (C; R) \geq I \left(E _ {c}; E _ {r}\right) \geq \hat {I} _ {\theta} \left(E _ {c}; E _ {r}\right) \tag {2}
$$
This leads us to the proposed learning objective DMI:
$$
\max  _ {\theta , \phi} \hat {I} _ {\theta} \left(E _ {c} ^ {(\phi)}; E _ {r} ^ {(\phi)}\right) \tag {3}
$$
where  $\hat{I}_{\theta}(E_c;E_r)$  is a variational lower bound estimate of  $I(E_{c};E_{r})$  (Equation 1) parametrized by  $\theta$  and  $\phi$  denotes the parameters of the encoder used for encoding  $C$  (or  $R$ ) to  $E_{c}$  (or  $E_{r}$ ).
Loss function For training our models, we minimize a loss function depending on the estimator being used.
$$
\min _ {\theta , \phi} \left[ L _ {\theta , \phi} (C, R) = - \hat {I} _ {\theta , e s t i m a t o r} (E _ {c} ^ {(\phi)}, E _ {r} ^ {(\phi)}) \right]
$$
We experimented with various MI estimators from literature, namely, MINE (Belghazi et al., 2018), InfoNCE (Oord et al., 2018), JSD (Hjelm et al., 2019) and SMILE (Song and Ermon, 2020). These MI estimators generally compute samples of  $E_{c}$  and  $E_{r}$  using  $C$  and  $R$  drawn from the joint distribution  $P_{CR}$ . Based on our preliminary experiments, we found that InfoNCE estimator produces better representations. The InfoNCE MI-estimate is computed as,
$$
I (C; R) \geq \log N - L _ {N} \tag {4}
$$
$$
L _ {N} = - \frac {1}{2} \mathbb {E} _ {P _ {C R}} \left[ \log \frac {e ^ {f (c , r)}}{\sum_ {r ^ {\prime} \in R} e ^ {f (c , r ^ {\prime})}} \right]
$$
where  $N$  denotes the batch size, and  $f(c,r)$  is a scoring function for the  $\langle c,r\rangle$  pair.
InfoNCE-S: The original InfoNCE formulation only considers negative samples for one of the random variables, but does not pose any constraint on which of the variables should be considered for negative sampling. As identifying the true response, from a pool of negative samples, would require different reasoning than identifying the true context out of a pool, we consider both these cases and create a symmetric version of the InfoNCE loss function. The final expression of this loss is given in Equation 5 and we refer to it as InfoNCE-S. This considerably improves the speed of training and convergence, and also gives a boost to downstream task performance.
${}^{3}C,R,{E}_{c},{E}_{r}$  in caps denote the random variables, whereas the lowercased versions  $c,r,{e}_{c},{e}_{r}$  denote samples.
One general exception to this is a neural model/representation overfitted on some training data. In such cases, the model may exactly memorize the input/output pairs.
$$
\begin{array}{l} L _ {N} = - \frac {1}{2} \mathbb {E} _ {P _ {C R}} \left[ \log \frac {e ^ {f (c , r)}}{\sum_ {r ^ {\prime} \in R} e ^ {f (c , r ^ {\prime})}} \right] \\ - \frac {1}{2} \mathbb {E} _ {P _ {C R}} \left[ \log \frac {e ^ {f (c , r)}}{\sum_ {c ^ {\prime} \in C} e ^ {f (c ^ {\prime} , r)}} \right] \tag {5} \\ \end{array}
$$
For other loss functions, more detailed discussion can be found in the Appendix.
Comparison with ConveRT (Henderson et al., 2020): There are a couple of differences between ConveRT's contrastive loss and our DMI objective. ConveRT models the problem as a response selection task and focuses on modeling cosine similarity between the context and the response. On the other hand, we propose a generic similarity computation function  $f(c, r)$  in Eqn. 4 and 5. Another difference is in encoding the input. ConveRT splits the context into previous turns and current query, and encodes them independently. Our model encodes the entire context jointly and hence is capable of better learning the correlations between previous turns and current query.
4 DMI vs. Language Modeling Objectives
In this work, we focus on utilizing DMI for pretraining dialog representations incorporating strong discourse-level features. But why should the DMI objective learn better discourse-level features than models trained on conversational data using MLM or LM objectives? We can find the answer by looking at various LM-based objectives through the lens of InfoMax, as shown by Kong et al. (2020). They connected various pretraining objectives for natural language representations, including the ones used for training Skipgram, BERT and XLNet, to the InfoMax learning principle.
If we consider an input text  $T$  and a masking function  $M$  that returns a masked text  $\tilde{T}$  and the masked word  $w$ , the MLM objective is equivalent to  $\mathcal{L}_{MLM} = -\hat{I}_{\theta}(E^{(\phi)}(\tilde{T}),e_w)$  where,  $E^{(\phi)}$  is the language encoder (e.g., a Transformer encoder) and  $e_w$  is the embedding of the token  $w$ . Similarly, in the case of auto-regressive LMs like GPT-2, the InfoMax objective equivalent to the loss is  $\mathcal{L}_{autoLM} = -\hat{I}_{\theta}(E^{(\phi)}(T_{1:t - 1}),e_{T_t})$ , where  $T_{1:t - 1}$  is the input sequence till  $t - 1^{th}$  token and  $T_{t}$  is the  $t^{th}$  token.
Compared to these LM objectives, DMI focuses on optimizing  $I(E_{c},E_{r})$ , where  $c$  and  $r$  are two structural components from the discourse with des
ignated roles. This enables DMI to discover more important features at the discourse level.
5.1 Architecture
The exact encoder architecture and the pretraining pipeline has been shown in Figure 2. We use a dual encoder architecture for encoding the contexts and responses separately. We observe that sharing parameters between the two encoders leads to a more efficient learning process and faster convergence. We use vanilla transformer-based encoders (Vaswani et al., 2017) for encoding the natural language inputs. The first tokens for both context and response sequences are the special [CLS] tokens whose contextual embeddings from the encoder are used as the context or response representations. The utterances in the context are delimited by another special token [EOU] (for end-of-utterance). We construct the context using as many utterances from the dialog history as possible up to a maximum of 300 subword tokens. We use the Word-Piece tokenizer from BERT for tokenizing the input texts, with a vocabulary size of 30,522.
The scoring function  $f(c, r)$  in Eqs. 4 and 5 is implemented using a Bilinear dot product between the context and response representations:  $f(c, r) = e_c^T W e_r$  where,  $W$  is a square weight matrix trained along with other parameters in the model. This function can take any real value, positive or negative, thus allowing the  $\hat{I}_{\theta}(E_C^{(\phi)}; E_R^{(\phi)})$  function to take any positive real value. While any complicated function with that range could be chosen, we chose this as a simple formulation satisfying the range constraint and left most of the learning to the transformer and the projection matrix  $W$ .
5.2 Model Variants
We train three different scales of the DMI model: DMI_Small with 6 layers, DMI_Medium with 8 layers, and DMI_Base with 12 layers. All configurations use 12 attention heads and 768-dimensional embeddings. DMI_Small is initialized with "google/bert_uncased_L-6_H-768_A-12", DMI_Medium is initialized with "google/bert_uncased_L-8_H-768_A-12"
<sup>5</sup>We implemented all models and experiments using the PyTorch and Huggingface libraries.
6These pretrained model tags are from the Huggingface model repository.
and DMI_Base is initialized by weights from "RoBERTa-base" pretrained checkpoint, and further pretrained on the pretraining dataset (see §5.4). All of these models are trained using the InfoNCE-S estimator, unless specified otherwise.
5.3 Hyper-parameter Settings
We use Adam optimizer with a linear learning rate schedule for training both the models. Learning rate is first linearly increased to a max value of  $5 \times 10^{-5}$  during the warm-up phase (first 1000 steps). Following this, in the remaining training period, learning rate is linearly decayed down back to zero. Before training DMI_Base, we reset the parameters of the 12th self-attention layer, and it is trained again from scratch along with the weight matrix  $W$  using our DMI objective. The embedding layer and initial 11 self-attention layers of the RoBERTabase encoder are finetuned at a slower learning rate  $(5 \times 10^{-6})$  during our pretraining phase.
As the mutual information value obtained by the InfoNCE loss is upper bounded by  $\log (N)$ ,  $N$  being the batch size, we try to keep the value of  $N$  as large as possible. Both 8 and 12-layer models are trained on 4-GPU (4x32 GB V100s) systems with overall batch size of 480 and 384, respectively<sup>7</sup>. All the trained models will be publicly shared upon publication.
5.4 Pretraining Dataset
We pretrained all our models using the Reddit corpus (Reddit-727M conversational-data) released by Henderson et al., 2019. We ran the scripts released by the authors to recreate the dataset of 727M English conversations. Out of these 727M conversations, we utilize around  $7.5\%$  to  $10\%$  of the dataset to train our models, after which the validation loss generally saturates. In the rest of this paper, we will refer to this dataset as rMax, in short.
Dialog Unrolling for Pretraining For training our models, we need samples of context-response (CR) pairs. Each dialog is unrolled to create context-response pairs with each utterance in the dialog as a response, except the first one. Hence, for each dialog \( D = \{U_1, U_2, \ldots, U_T\} \), we generate the following set of samples \( S = \{(\mathcal{C}_t :
$U_{1},\ldots ,U_{t - 1};\mathcal{R}_{t}:U_{t}):t\in [2,T]\}$  . If we process the full rMax dataset, this leads to, approximately, 2.7B CR pairs.
5.5 MI Estimation