Venue: NAACL 2022

# Representation Learning for Conversational Data using Discourse Mutual Information Maximization

Bishal Santra\({}^{\clubsuit}\)

IIT Kharagpur &Sumegh Roychowdhury

IIT Kharagpur &Aishik Mandali

IIT Kharagpur IIT Kharagpur &Vasu Gurram

IIT Kharagpur &Atharva Naik

IIT Kharagpur &Manish Gupta\({}^{\clubsuit}\)

Microsoft, India &Pawan Goyal\({}^{\spadesuit}\)

IIT Kharagpur

###### Abstract

Although many pretrained models exist for text or images, there have been relatively fewer attempts to train representations specifically for dialog understanding. Prior works usually relied on finetuned representations based on generic text representation models like BERT or GPT-2. But such language modeling pretraining objectives do not take the structural information of conversational text into consideration. Although generative dialog models can learn structural features too, we argue that the structure-unaware word-by-word generation is not suitable for effective conversation modeling. We empirically demonstrate that such representations do not perform consistently across various dialog understanding tasks. Hence, we propose a structure-aware Mutual Information based loss-function DMI (Discourse Mutual Information) for training dialog-representation models, that additionally captures the inherent uncertainty in response prediction. Extensive evaluation on nine diverse dialog modeling tasks shows that our proposed DMI-based models outperform strong baselines by significant margins.

+
Footnote †: July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Representation learning has transformed how we can apply machine learning to solve real-world problems. However, despite a vast body of research on pretrained language representations, there have been relatively fewer attempts to train representations specifically for dialog understanding. Prior works mostly relied on finetuned representations based on generic models like BERT Devlin et al. (2019) or GPT-2 Radford et al. (2019). In our experiments, we demonstrate that such representations do not perform uniformly across various dialog understanding tasks such as dialog-act classification, intent detection or dialog evaluation.

On the other hand, prior works on pretraining large-scale dialog models focused mainly on open-domain generation. These works evaluated their models only on dialog generation Zhang et al. (2020); Roller et al. (2021); Adiwardana et al. (2020) or tasks related directly to the pretraining objective Henderson et al. (2020); Gao et al. (2020). Their effectiveness on other dialog understanding tasks like act classification or intent detection remains unexplored. So we ask the following research question: _Can we learn enriched representations directly at the pretraining phase that are specifically helpful for dialog understanding?_

Existing language modeling (causal or masked) pretraining objectives unfortunately are not the best to model dialogs for these reasons: (1) The model is not directly trained to learn the content discourse structure (e.g., context-response in dialogs). (2) Such models are trained to generate the response word-by-word rather than predicting a larger unit. (3) The inherent one-to-many nature of dialog generation implies that the encoding model should be able to capture uncertainty in the response prediction task, that such models ignore.

Hence, in this paper, we propose pretraining objectives for improved dialog modeling that turn the discourse-level organizational structure of texts from natural sources (e.g., documents, dialogs, or monologues) into a learnable objective. We call this objective the Discourse Mutual Information (DMI). The key insight towards the design of our pretraining objective is to capture representations that can account for a meaningful conversation out of a specific ordered sequences of utterances. We hope that a discourse-level pretraining objective with conversational data would guide the model to learn complex context-level features. For example, in Fig. 1, we illustrate the differences between standard language modeling (causal or masked) based pretraining objectives and a discourse-level reasoning task.

The second research question that we ask is _whether discourse-level features learned using self-supervised pretraining outperform word-level pre

training objectives for downstream dialog understanding tasks._ Experimentally, we show that representations learned using the proposed objective function are highly effective compared to both existing discriminative as well as generative dialog models. In terms of various dialog understanding tasks, our models achieve state-of-the-art performances in several tasks (absolute improvements up to 8.5% and 3.5% in task accuracies in probing and finetuning setups, resp.) and perform consistently well across a variety of dialog understanding tasks, whereas baseline models usually have a rather imbalanced performance across tasks.

Overall, our main contributions are as follows.

* We propose DMI, a novel information-theoretic objective function for pretraining dialog representation.
* We release pretrained dialog-representation models in three different sizes (small, medium and base) based on our proposed self-supervised learning objectives1. Footnote 1: To access the pretrained dialog representation models and the source codes, please visit [https://bsantraigi.github.io/DMI](https://bsantraigi.github.io/DMI)
* We extensively evaluate our DMI based representations on multiple open-domain downstream tasks like intent detection, dialog-act classification, response retrieval, dialog reasoning, and response-generation evaluation, and beat state-of-the-art across nine tasks in both probe as well as finetune setups.

## 2 Literature Review

### Dialog System Pretraining

There have been quite a few efforts towards utilizing existing representations or developing new pretrained models for dialog systems. While BERT Devlin et al. (2019), ELMo Peters et al. (2018), GPT-2 Radford et al. (2019) and other general purpose large-scale pretrained networks are not specific to dialogs, transfer learning from such models could be reasonable. Basic language understanding capability available through these representations helps to get decent performance on many dialogue-understanding tasks Hosseini-Asl et al. (2020).

On the other hand, there have been various works on pretraining dialog specific representations or large-scale generation models. We summarize the properties of various previously proposed dialog-representation learning models in Table 1. DialoGPT Zhang et al. (2020), Meena Adiwardana et al. (2020) and Blenderbot Roller et al. (2021) are large-scale Transformer-based language models, which are trained to generate the gold-response (as per the dataset) given a dialog context. ContextPretrain Mehri et al. (2019), ConveRT Henderson et al. (2020) and ConvFiT Vulic et al. (2021) are trained on the response retrieval task using MultiWoz or Reddit conversations. DEB or Dialog Evaluation using BERT Sai et al. (2020) is a model based on extended pretraining of the BERT architecture using Reddit data. DialogRPT Gao et al. (2020), on the other hand, is pretrained to predict human-feedback (e.g., upvotes and downvotes) on comments to Reddit threads. This model is initial

Figure 1: Possible reasoning involved in two types of pretraining: Word-level (left), Discourse-level (right). In a discourse-level reasoning task, the immediately preceding utterance may not be enough for understanding the full context. To predict the correct response, the model will need to capture both the larger context, in this case the topic of discussion, and the intent (e.g., asking for details) of the preceding utterance. In comparison, word-level reasoning is often easier and can be solved using local reasoning. Each of the three masked-words, in the left image, could have been predicted with reasonable confidence without any more information than the utterance itself.

 

ized using the weights of DialoGPT model. Wu et al. (2020) thoroughly investigate these existing pretrained representations, both generic and dialog specific, for understanding their effectiveness on various goal-oriented dialog-understanding tasks.

### Self-supervised Representation Learning with InfoMax

Mutual Information maximization (InfoMax) is one of the popular approaches for self-supervised learning, first used by Oord et al. (2018) and Belghazi et al. (2018). Oord et al. (2018) proposed InfoNCE loss which is an estimator for lower bound to mutual information (MI) between two continuous-valued random variables. InfoNCE has also been used for other NLP applications like training sentence embeddings (SIMCSE Gao et al. (2021)), question answering (QA-InfoMax Yeh and Chen (2019)), etc. Other estimators for mutual information have also been proposed like MINE (Mutal Information Neural Estimator) Belghazi et al. (2018) and SMILE Song and Ermon (2020). In general, these estimators are also broadly studied in contrastive Learning (CL) literature for training both self-supervised Mikolov et al. (2013); Devlin et al. (2019); Liu et al. (2019); Gao et al. (2021); Henderson et al. (2020); Vulic et al. (2021) and supervised models Schroff et al. (2015); Gunel et al. (2020). Some prior works in the dialog generation domain have used the concept of mutual information to design loss functions or scoring mechanisms to improve specificity of the generated responses Li et al. (2016); Yoo et al. (2020). These works predominantly used MI either as a regularizer, along with cross entropy loss, or as a scoring function for ranking generated responses in a post-processing step. In the next section, we derive our pretraining loss function DMI for conversational texts from an information-theoretic perspective.

## 3 Discourse Mutual Information

We define Discourse Mutual Information (DMI) as the mutual information2 between two random variables representing two different segments within the same discourse. This is a general concept that can be applied to any form of discourse, no matter the domain or type of signal. In this paper, we focus on dialog type discourses and representation learning for conversational texts. We define two random variables for the contexts (\(C\)) and responses (\(R\)) that jointly construct a valid conversation. Conversations between humans represent samples from the joint distribution \(P_{CR}\) of \(C\) and \(R\). We pose the following learning problem, _"learn continuous representations for the textual random variables \(C\) and \(R\) such that the true mutual information between \(C\) and \(R\) can be closely estimated."_

Footnote 2: Mutual Information between two random variables is defined as the reduction in uncertainty/entropy of one of the random variables by having knowledge about the value of the other random variable. Mathematically, this is written as \(I(X;Y)=H(X)-H(X|Y)=H(Y)-H(Y|X)\).

In the remainder of this section we show that, if the lower bound on MI estimated by some representations of context and response is close to the true value, the representation of the context would be as predictive of the response as the natural language form itself. Existing generative training objectives as used in DialoGPT or Blenderbot are extremely focused on predicting target response only. Per-word cross-entropy loss, used for training these models, fails to take into account the inherent uncertainty in the context-to-response generation function. Adapting context representations so as to predict the target responses optimally, helps our proposed DMI-based models learn better dialog representations applicable to a versatile set of dialog understanding tasks.

\begin{table}
\begin{tabular}{l c r r r r} \hline
* & Multiwoor (DIST, Act, NIC, INU) \\ _DEB_ & 727M Dialogs & MLM, MSP & BERT & 110M & Adv/Random Dialog Evaluation \\ _ConvRT_\(\dagger\) & 727M Dialogs & Response Selection & Tr. Encoder & 29M & Response Selection \\ _ConrIT_\(\ddagger\) & 8\% of 727M Dialogs + Intent data & Response Selection & BEET & 110M & Latent Detection \\ _DMtBase_ & 7.5-10\% of 727M Dialogs & InfoNCE-S & Tr. Encoder & 124M & 9 Dialog-NLU tasks \\ \hline \end{tabular}
\end{table}
Table 1: Survey of Pretrained Dialog Models. NUR: next utterance retrieval, NUG: next utterance generation, MUR: masked utterance retrieval, I2: inconsistency identification, CR: Context-response, S2S: Seq2Seq, Tr.. Transformer, CE: Cross-entropy, HRED: Hierarchical RNN Encoder-Decoder. \(\dagger\) Pretrained checkpoints available but only for inference. \(\ddagger\) Both source-code and checkpoints are not available.
 

Objective Function FormulationLet \(E_{c}\) and \(E_{r}\) be the representations3 for \(C\) and \(R\) based on some encoder. Using the data processing inequality from Information theory Cover, 1999), we have

Footnote 3: \(C,R,E_{c},E_{r}\) in caps denote the random variables, whereas the lowercased versions \(c,r,e_{c},e_{r}\) denote samples.

\[I(C;R)\geq I(E_{c};E_{r}) \tag{1}\]

This tells us that MI between any encoded version of \(C\) and \(R\) will always be less or equal than the true mutual information. The equality will hold if \(E_{c}\) and \(E_{r}\) are both fully-invertible encoding processes (as opposed to representations which are lossy or compressive, and inversion is thus not possible). However, neural networks generally embed the data points in a low dimensional manifold by learning robust features that can represent the data points efficiently. Because of this, neural representations are usually not invertible.4 Now, the exact computation of MI is not possible for continuous-valued random variables. In recent years, various variational lower bounds have been proposed for estimating MI between continuous-valued random variables. Including the MI estimator (\(\hat{I}_{\theta}\)), the overall relation becomes

Footnote 4: One general exception to this is a neural model/representation overfitted on some training data. In such cases, the model may exactly memorize the input/output pairs.

\[I(C;R)\geq I(E_{c};E_{r})\geq\hat{I}_{\theta}(E_{c};E_{r}) \tag{2}\]

This leads us to the proposed learning objective DMI:

\[\max_{\theta,\phi}\hat{I}_{\theta}(E^{(\phi)}_{c};E^{(\phi)}_{r}) \tag{3}\]

where \(\hat{I}_{\theta}(E_{c};E_{r})\) is a variational lower bound estimate of \(I(E_{c};E_{r})\) (Equation 1) parametrized by \(\theta\) and \(\phi\) denotes the parameters of the encoder used for encoding \(C\) (or \(R\)) to \(E_{c}\) (or \(E_{r}\)).

Loss functionFor training our models, we minimize a loss function depending on the estimator being used.

\[\min_{\theta,\phi}\left[L_{\theta,\phi}(C,R)=-\hat{I}_{\theta,estimator}(E^{( \phi)}_{c},E^{(\phi)}_{r})\right]\]

We experimented with various MI estimators from literature, namely, MINE Belghazi et al. (2018), InfoNCE Oord et al. (2018), JSD Hjelm et al. (2019) and SMILE Song and Ermon (2020). These MI estimators generally compute samples of \(E_{c}\) and \(E_{r}\) using \(C\) and \(R\) drawn from the joint distribution \(P_{CR}\). Based on our preliminary experiments, we found that InfoNCE estimator produces better representations. The InfoNCE MI-estimate is computed as,

\[I(C;R) \geq\log N-L_{N} \tag{4}\] \[L_{N} =-\frac{1}{2}\mathbb{E}_{P_{CR}}\left[\log\frac{e^{f(c,r)}}{\sum_ {r^{\prime}\in R}e^{f(c,r^{\prime})}}\right]\]

where \(N\) denotes the batch size, and \(f(c,r)\) is a scoring function for the \(\langle c,r\rangle\) pair.

**InfoNCE-S:** The original InfoNCE formulation only considers negative samples for one of the random variables, but does not pose any constraint on which of the variables should be considered for negative sampling. As identifying the true response, from a pool of negative samples, would require different reasoning than identifying the true context out of a pool, we consider both these cases and create a symmetric version of the InfoNCE loss function. The final expression of this loss is given in Equation 5 and we refer to it as InfoNCE-S. This considerably improves the speed of training and convergence, and also gives a boost to downstream task performance.

Figure 2: Base Pretraining Architecture for DMI. In our implementations of the model, \(f_{\phi}\) denotes the transformer Vaswani et al. (2017) based encoders. Context and response encoders share all parameters for efficient learning. \(d\) denotes sample dialogs from the training dataset.

 

\[L_{N}=-\frac{1}{2} \mathbb{E}_{P_{CR}}\left[\log\frac{e^{f(c,r)}}{\sum_{r^{\prime}\in R} e^{f(c,r^{\prime})}}\right]\] \[-\frac{1}{2} \mathbb{E}_{P_{CR}}\left[\log\frac{e^{f(c,r)}}{\sum_{c^{\prime} \in C}e^{f(c^{\prime},r)}}\right] \tag{5}\]

For other loss functions, more detailed discussion can be found in the Appendix.

**Comparison with ConveRT Henderson et al. (2020)**: There are a couple of differences between ConveRT's contrastive loss and our DMI objective. ConveRT models the problem as a response selection task and focuses on modeling cosine similarity between the context and the response. On the other hand, we propose a generic similarity computation function \(f(c,r)\) in Eqn. 4 and 5. Another difference is in encoding the input. ConveRT splits the context into previous turns and current query, and encodes them independently. Our model encodes the entire context jointly and hence is capable of better learning the correlations between previous turns and current query.

## 4 DMI vs. Language Modeling Objectives

In this work, we focus on utilizing DMI for pretraining dialog representations incorporating strong discourse-level features. _But why should the DMI objective learn better discourse-level features than models trained on conversational data using MLM or LM objectives?_ We can find the answer by looking at various LM-based objectives through the lens of InfoMax, as shown by Kong et al. (2020). They connected various pretraining objectives for natural language representations, including the ones used for training Skipgram, BERT and XLNet, to the InfoMax learning principle.

If we consider an input text \(T\) and a masking function \(M\) that returns a masked text \(\tilde{T}\) and the masked word \(w\), the MLM objective is equivalent to \(\mathcal{L}_{MLM}=-\hat{I}_{\theta}(E^{(\phi)}(\tilde{T}),e_{w})\) where, \(E^{(\phi)}\) is the language encoder (e.g., a Transformer encoder) and \(e_{w}\) is the embedding of the token \(w\). Similarly, in the case of auto-regressive LMs like GPT-2, the InfoMax objective equivalent to the loss is \(\mathcal{L}_{autoLM}=-\hat{I}_{\theta}(E^{(\phi)}(T_{1:t-1}),e_{T_{k}})\), where \(T_{1:t-1}\) is the input sequence till \(t-1^{th}\) token and \(T_{t}\) is the \(t^{th}\) token.

Compared to these LM objectives, DMI focuses on optimizing \(I(E_{c},E_{r})\), where \(c\) and \(r\) are two structural components from the discourse with designated roles. This enables DMI to discover more important features at the discourse level.

## 5 Experiments

### Architecture

The exact encoder architecture and the pretraining pipeline has been shown in Figure 2. We use a dual encoder architecture for encoding the contexts and responses separately. We observe that sharing parameters between the two encoders leads to a more efficient learning process and faster convergence. We use vanilla transformer-based encoders5(Vaswani et al., 2017) for encoding the natural language inputs. The first tokens for both context and response sequences are the special [CLS] tokens whose contextual embeddings from the encoder are used as the context or response representations. The utterances in the context are delimited by another special token [EOU] (for end-of-utterance). We construct the context using as many utterances from the dialog history as possible up to a maximum of 300 subword tokens. We use the WordPiece tokenizer from BERT for tokenizing the input texts, with a vocabulary size of 30,522.

Footnote 5: We implemented all models and experiments using the PyTorch and Huggingface libraries.

The scoring function \(f(c,r)\) in Eqs. 4 and 5 is implemented using a Bilinear dot product between the context and response representations: \(f(c,r)=e_{c}^{T}We_{r}\) where, \(W\) is a square weight matrix trained along with other parameters in the model. This function can take any real value, positive or negative, thus allowing the \(\hat{I}_{\theta}(E_{C}^{(\phi)};E_{R}^{(\phi)})\) function to take any positive real value. While any complicated function with that range could be chosen, we chose this as a simple formulation satisfying the range constraint and left most of the learning to the transformer and the projection matrix W.

### Model Variants

We train three different scales of the DMI model: **DMI_Small** with 6 layers, **DMI_Medium** with 8 layers, and **DMI_Base** with 12 layers. All configurations use 12 attention heads and 768-dimensional embeddings. DMI_Small is initialized with "google/bert_uncased_L-6_H-768_A-12*6, DMI_Medium is initialized with "google/bert_uncased_L-8_H-768_A-12 

and DMI_Base is initialized by weights from "_RoBERTa-base_" pretrained checkpoint, and further pretrained on the pretraining dataset (see SS5.4). All of these models are trained using the InfoNCE-S estimator, unless specified otherwise.

### Hyper-parameter Settings

We use Adam optimizer with a linear learning rate schedule for training both the models. Learning rate is first linearly increased to a max value of \(5\times 10^{-5}\) during the warm-up phase (first 1000 steps). Following this, in the remaining training period, learning rate is linearly decayed down back to zero. Before training DMI_Base, we reset the parameters of the 12th self-attention layer, and it is trained again from scratch along with the weight matrix \(W\) using our DMI objective. The embedding layer and initial 11 self-attention layers of the RoBERTa-base encoder are finetuned at a slower learning rate (\(5\times 10^{-6}\)) during our pretraining phase.

As the mutual information value obtained by the InfoNCE loss is upper bounded by \(\log(N)\), \(N\) being the batch size, we try to keep the value of \(N\) as large as possible. Both 8 and 12-layer models are trained on 4-GPU (4x32 GB V100s) systems with overall batch size of 480 and 384, respectively7. All the trained models will be publicly shared upon publication.

Footnote 7: Training time: A maximum of 2 weeks of training time was allowed for 8-layer and 12-layer models. Though, the training process saturates long before the maximum allowed time, and we evaluate our models based on checkpoints when the best validation scores are first obtained.

### Pretraining Dataset

We pretrained all our models using the Reddit corpus **Reddit-727M conversational-data** released by Henderson et al., 2019. We ran the scripts released by the authors to recreate the dataset of 727M English conversations. Out of these 727M conversations, we utilize around 7.5% to 10% of the dataset to train our models, after which the validation loss generally saturates. In the rest of this paper, we will refer to this dataset as **rMax**, in short.

Dialog Unrolling for PretrainingFor training our models, we need samples of context-response (CR) pairs. Each dialog is unrolled to create context-response pairs with each utterance in the dialog as a response, except the first one. Hence, for each dialog \(D=\{U_{1},U_{2},\ldots,U_{T}\}\), we generate the following set of samples \(S=\{(\mathcal{C}_{t}:U_{1},\dots,U_{t-1};\mathcal{R}_{t}:U_{t}):t\in[2,T]\}\). If we process the full rMax dataset, this leads to, approximately, 2.7B CR pairs.

### MI Estimation

During pretraining, we compare the checkpoints from different epochs and across hyperparameter settings in terms of the bits of mutual information extracted by the trained representation on an unseen set of dialogs. This is calculated as \(MI_{valid}=\log(N)-L_{N}\) (see SS3 for more details). As per the Information Bottleneck theory Tishby et al. (2000), the mutual information learned between the two observed random variables can be factorized into two components, namely, predictive and redundant information. Predictive information generally identifies whether the features learned by the representation are useful for a downstream task. The redundant information is caused by features that do not help in any downstream tasks. Such features can exist due to noise or spurious correlation in the dataset, or even overfitting. Hence, we train our final models on a fraction of the rMax dataset but only for one epoch (i.e., we never repeat the samples) which removes any possibility of overfitting.

Predictive features identified based on a fixed set of downstream tasks Tishby et al. (2000); Alemi et al. (2017) may not be a sufficient to assess other features learned in the training process. Since, ideally, we want to maximize the amount of predictive information in the representation, we compare the bits of MI on the training set against the bits of MI on an unseen validation set, as captured by the learned representation. To make sure that we do not assume anything about the domain or the conversation topics, we use the validation set of dialogs from the open-domain Daily Dialog dataset Li et al. (2017).

### Downstream Tasks

Instead of focusing on a single downstream task like many previous works on dialog representation learning, we consider a more versatile range of tasks to evaluate the learned representations from DMI or the baseline models. To find out whether a certain representation is effective for some downstream task, we evaluate in two setups: probe and finetune. In both cases, the pretrained model is used along with an MLP classifier of fixed complexity Pimentel et al. (2020). In probing setup, we only train the parameters of the MLP classifier.

 

In finetuning setup, we also train the pretrained model parameters along with the MLP classifier parameters. We use the context and response representations from our models as the input to the MLP classifier.

For downstream tasks, we have two reasoning tasks based on the MuTual dataset Cui et al. (2020), three classification tasks based on conversational intent detection Casanueva et al. (2020), emotion detection Welivita and Pu (2020) and act classification Stolcke et al. (1998), and four dialog evaluation tasks based on the DailyDialog++ dataset DD++, Sai et al. (2020)8. Table 2 shows dataset details and metrics for these nine tasks. Both MuTual and DailyDialog++ datasets have an adversarial configuration for the respective tasks, which allows us to assess each of the evaluated models in adversarial settings also.

Footnote 8: Note that DailyDialog++ is different from DailyDialog.

## 6 Results and Discussions

### Pretraining DMI based Representations

During pretraining, we used "Validation MI" to evaluate model checkpoints. As the goal of our models is to learn a representation that captures maximum MI between the context and the response texts, this metric tracks how well the learned representation captures the mutual information between contexts and responses of unseen dialogs.

We use the validation split from Daily Dialog dataset as our validation set for evaluation the model during pretraining. It is not specific to a domain and, hence, covers a versatile range of topics. This set comprises 1,000 full conversations between two persons which on unrolling leads to 7,069 context-response (CR) pairs. We illustrate the variation in validation-MI metric against training steps in Fig. 3 in the Appendix.

### Comparison of Representations on Downstream Task Performance

In this set of experiments, we probe/finetune the DMI models with various downstream tasks that require knowledge of many different types of dialogue-understanding features. The results of our probing and finetuning experiments are shown in Table 3.

We have used two types of models as our baselines: generic pretrained models and dialog-specific pretrained models. RoBERTa, BERT, T5 Raffel et al. (2019), GPT-2 are all trained on large corpora of generic web-crawled English text. But, since these models were not specifically pretrained on any dialog corpus, they may suffer from poor performance on certain dialog understanding tasks. Hence, we consider DialoGPT9, DialogRPT, DEB and ConveRT models, which were trained on conversational data. For DialogRPT, we used "human-vs-rand" checkpoint released by authors. All models are 12-layer except Blender_Small (8 layers), ConveRT (6 layers), DialogRPT (24 layers) and DMI_Medium (8 layers). We used the publicly available model checkpoints for all baselines, wherever possible. The ConveRT model's checkpoint has been removed from Github10 by its authors. Hence, it was only possible for us to MLP-probe the representations, without finetuning of the model, based on a cached version released by another user under a valid license11. Pretrained checkpoints for Meena, ContextPretrain and ConvFiT are not available, and hence we do not compare with them.

Footnote 9: DialoGPT and DEB are based on GPT-2 and BERT models and were further pretrained on conversations from Reddit. They use the original loss functions of GPT-2/BERT.

Footnote 10: [https://github.com/PolyAI-LDN/polyai-models](https://github.com/PolyAI-LDN/polyai-models)

Footnote 11: [https://github.com/davidalami/ConveRT](https://github.com/davidalami/ConveRT)

#### 6.2.1 Results in Probing Setup

We observe that, on average, DEB and ConveRT have good performance among the baselines. How

\begin{table}
\begin{tabular}{l l r r r r} \hline \hline
**Task** & **Description** & **Train** & **Valid** & **Test** & **Metric** \\ \hline Banking77 & Intent 77-class Classification & 8,002 & 2,001 & 3,080 & Accuracy \\ SWDA & Dialog Act 41-class Classification & 213,543 & 56,729 & 4,514 & Accuracy \\ MuTual & Reasoning as Response Selection & 25,516 & 2,336 & 3,544 & R@1, R@2, MRR \\ MuTual Plus & MuTual + Safe response candidate & 25,516 & 2,836 & 3,544 & R@1, R@2, MRR \\ DD++ & Dialog Evaluation & 92,590 & 10,280 & 11,420 & Accuracy \\ DD++Adv & Train: Adv. neg., Test: Adv neg. samples & 92,590 & 10,280 & 11,420 & Accuracy \\ DD++Cross & Train: Random neg., Test: Adv neg. samples & 92,590 & 10,280 & 11,420 & Accuracy \\ DD++Full & Train: All samples, Test: Adv. neg. samples & 138,886 & 10,280 & 11,420 & Accuracy \\ Empathetic Intent & Emotion and Intent 44-class Classification & 25,023 & 3,544 & 3,225 & Accuracy \\ \hline \hline \end{tabular}
\end{table}
Table 2: Downstream task details. Adv.: Adversarial, Neg.: Negative 

ever, the RoBERTa model outperforms all other baselines on the MuTual task by a significant margin. In the MuTual Plus task, the DEB model outperforms other models in the R@1 and MRR metrics. ConveRT performs the best among all baselines on the other tasks. ConveRT's loss function is also contrastive in nature and is similar to ours. This explains the model's generally high strength across the tasks among all the baselines.

Our DMI_Base beats ConveRT on all the tasks, and DMI_Medium beats the baseline on 7 out of 9 tasks. We believe DD++ tasks to be the most demanding ones with respect to context-level understanding. Here, all non-dialog baselines have a weaker performance, with DEB and ConveRT being the best of the bunch. These are also the tasks where our models excel the most, with both DMI_Medium and DMI_Base beating all baselines with strong margins. DD++/cross is the most difficult among all four DD++ tasks. Here, the model is trained on random negative samples and tested on a dataset with human-curated adversarial negatives. Our DMI_Base beats the best baseline on DD++/cross by 8.54 points. This shows the superior quality of context representations from our models.

#### 6.2.2 Results in Finetuning Setup

In the finetuning setup, on average, RoBERTa and DialogRPT have good performance among the baselines. DialogRPT performs well for DD++ tasks while Blender works well for the MuTual task. For all other tasks, RoBERTa is the best baseline, even outperforming models especially trained for dialog tasks (like DialoGPT).

Similar to the probe setup, DMI_Base beats base

\begin{table}
\begin{tabular}{l c r r r r r r r r r r r r} \hline \hline  & B77 & SWDA & E-Intent & \multicolumn{3}{c}{MuTual} & \multicolumn{3}{c}{MuTual Plus} & \multicolumn{3}{c}{DD++} & \multicolumn{3}{c}{DD++/adv} & \multicolumn{3}{c}{DD++/cross} & \multicolumn{1}{c}{DD++/full} \\ \hline Model & Acc. & Acc. & Acc. & R@1 & R@2 & MRR & R@1 & R@2 & MRR & Acc. & Acc. & Acc. & Acc. \\ \hline DMI\_Base & **93.93** & **74.50** & 64.62 & 56.43 & **79.91** & 74.27 & **52.14** & **75.06** & **71.09** & **91.03** & 96.39 & 76.01 & 92.61 \\ DMI\_Base - Sym & 93.28 & 72.69 & **65.18** & **57.34** & 77.88 & **74.32** & 48.08 & 72.69 & 68.60 & 90.94 & **96.65** & **76.45** & **93.13** \\ DMI\_Base - RGB & 92.34 & 74.10 & 69.96 & 53.84 & 77.31 & 72.47 & 50.34 & 72.80 & 69.81 & 87.23 & 92.95 & 73.53 & 87.85 \\ DMI\_Base+Sym - Rob & 91.59 & 73.55 & 60.71 & 54.06 & 75.40 & 72.24 & 47.97 & 71.45 & 68.24 & 86.79 & 92.96 & 70.29 & 87.13 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Ablation study results for the finetune setup for our base model on 9 downstream tasks. “RoB” \(\rightarrow\) No RoBERTa initialization. ”-Sym” \(\rightarrow\) Training with non-symmetric version of InfoCNE. (DD++: DailyDialog++, B77: Banking77 task, R@k: Recall at k, MRR: Mean reciprocal rank).

\begin{table}
\begin{tabular}{l l r r r r r r r r r r r r r} \hline \hline  & & B77 & SWDA & E-Intent & \multicolumn{3}{c}{Mutual} & \multicolumn{3}{c}{MuTual Plus} & \multicolumn{3}{c}{DD++} & \multicolumn{3}{c}{DD++/adv} & \multicolumn{3}{c}{DD++/cross} & \multicolumn{1}{c}{DD++/full} \\ \hline Model & Acc. & Acc. & Acc. & R@1 & R@2 & MRR & R@1 & R@2 & MRR & Acc. & Acc. & Acc. & Acc. \\ \hline \multirow{3}{*}{\begin{tabular}{} \end{tabular} } & RoBERTa\_Base & 72.84 & 67.18 & 50.45 & **49.70** & **75.20** & **70.00** & 43.60 & 66.60 & 65.10 & 55.75 & 84.20 & 65.11 & 68.76 \\  & BERT Base & 72.74 & 67.99 & 46.84 & 45.40 & 72.80 & 67.30 & 42.60 & 67.70 & 64.90 & 60.39 & 86.56 & 65.25 & 72.50 \\  & T5\_Base & 60.82 & 68.79 & 44.50 & 43.20 & 69.40 & 65.60 & 38.30 & 65.70 & 62.20 & 57.46 & 84.14 & 61.23 & 63.35 \\  & GPT-2\_Small & 76.64 & 69.17 & 49.94 & 49.42 & 75.04 & 66.05 & 40.75 & 66.70 & 63.46 & 66.77 & 82.06 & 67.53 & 73.93 \\  & DialoGPT-Small & 53.00 & 65.10 & 43.42 & 92.80 & 53.50 & 55.15 & 25.51 & 57.56 & 54.05 & 63.63 & 78.02 & **70.61** & 70.77 \\  & Blender\_Small & 70.39 & 70.11 & 48.52 & 41.42 & 68.06 & 64.29 & 42.89 & 68.85 & 65.18 & 60.07 & 65.14 & 57.76 & 68.20 \\  & ConvRetr & **89.88** & **71.36** & **55.47** & 45.30 & 72.00 & 67.00 & 49.00 & **69.00** & 64.30 & **79.14** & **88.67** & 69.59 & **80.86** \\  & DialogRPT & 81.54 & 67.92 & 50.74 & 39.50 & 68.60 & 68.30 & 34.20 & 61.50 & 59.20 & 74.11 & 81.29 & 68.49 & 67.20 \\  & DEB & 79.18 & 68.50 & 45.31 & 45.40 & 74.00 & 67.50 & **45.00** & 67.70 & **66.00** & 86.67 & 67.25 & 67.77 \\ \cline{2-16}  & DMI\_Small & 89.81 & 72.33 & 55.72 & 51.24 & 74.94 & 70.76 & 46.39 & 70.09 & 67.14 & 85.01 & 91.51 & 75.75 & 86.34 \\  & DMI\_Medium & 90.42 & 72.49 & 57.33 & 52.48 & 75.62 & 71.47 & 46.61 & 72.46 & 67.79 & 85.80 & 91.38 & 76.73 & 86.94 \\  & DMI\_Base & 91.43 & 72.73 & 60.00 & 52.48 & 76.41 & 71.65 & 48.98 & 71.33 & 68.73 & 86.91 & 91.98 & 79.15 & 88.32 \\ \cline{2-16}  & \(\Delta\) & 1.55 & 1.37 & 4.53 & 2.78 & 1.21 & 1.65 & 3.98 & 2.33 & 7.77 & 3.31 & 8.54 & 7.46 \\ \hline \hline \multirow{3}{*}{
\begin{tabular}{} \end{tabular} } & RoBERTa\_Base & **92.75** & 73.61 & **62.81** & **48.42** & **72.20** & 69.70 & **49.55** & **73.70** & **69.50** & 90.00 & 95.70 & **73.76** & 91.09 \\  & BERT\_Base & 92.27 & 72.99 & 60.12 & 47.86 & 73.93 & 68.00 & 49.10 & 72.55 & 69.00 & 87.05 & 94.33 & 67.70 & 88.82 \\  & TS & 89.11 & **73.77** & 60.66 & 49.77 & 73.93 & 69.80 & 43.00 & 66.93 & 64.90 & 82.03 & 90.89 & 65.85 & 85.63 \\  & GPT-2\_Small & 92.49 & 72.62 & 58.44 & 48.42 & 72.69 & 68.90 & 45.71 & 70.99 & 67.10 & 85.69 & 93.60 & 68.43 & 87.83 \\  & DialoGPT-Small & 92.59 & 73.48 & 

line methods by significant margins. In general, finetune results are better than probe results across all models, as expected.

Our large-scale RoBERTa-initialized DMI_Base model outperforms the best baseline for all tasks, by a considerable margin. Additionally, our DMI-based models are able to perform well uniformly across all tasks, unlike even baselines like DialoGPT, DialogRPT and Blenderbot models which are explicitly trained on dialog data. This makes DMI the best overall model for dialog related tasks. Across multiple tasks, we show qualitative examples where our proposed DMI-based models provide accurate results, in the Appendix.

### Ablations

We evaluate the importance of using RoBERTa based pretraining as well as the symmetric version of the InfoNCE loss in Table 4. We observe that RoBERTa based pretraining helps significantly across all tasks. The symmetric InfoNCE improves performance for SWDA and MuTual Plus tasks.

## 7 Conclusions and Future work

In this paper, we proposed the concept of Discourse Mutual Information (DMI) which is better suited for learning dialog-specific features in a self-supervised manner. Using the InfoMax principle we formulated a pretraining method for dialog-specific representation learning. Across 9 downstream dialog understanding tasks, our 12-layer model outperforms state-of-the-art methods. Further, we showed that on most of these tasks, even our 8-layer model outperforms standard 12-layer pretrained models. These experiments show the potential of the proposed DMI objective towards building dialog understanding models. We will make the code and pretrained model checkpoints available on request, instructions can be found here [https://bsantraigi.github.io/DMI](https://bsantraigi.github.io/DMI). Although we experimented only with dialog modeling in this paper, we believe that the proposed DMI objective is generic enough to be applied to any type of discourse in any domain. In the future, we would like to explore how to harness DMI representations for generative conversation modeling.

## 8 Acknowledgments

This work was partially supported by Microsoft Academic Partnership Grant (MAPG) 2021. The first author was also supported by Prime Minister's Research Fellowship (PMRF), India.

## 9 Ethical considerations

Like many other pretrained language representation models, the proposed model may also have learned patterns associated with exposure bias. Interpretability associated with the output is rather limited, hence users should use the outputs carefully. The proposed model ranks possible response candidates, and does not filter out any "problematic" candidates. Thus, for applications, where candidate responses could be problematic, (e.g., offensive, hateful, abusive, etc.), users should carefully filter them out before providing them as input to our model.

All the datasets used in this work are publicly available. We did not collect any new dataset as part of this work.

Banking77 Casanueva et al., 2020 has been obtained from [https://github.com/PolyAI-LDN/task-specific-datasets](https://github.com/PolyAI-LDN/task-specific-datasets). It is available under a Creative Commons Attribution 4.0 International license with details here12.

Footnote 12: [https://github.com/PolyAI-LDN/task-specific-datasets/blob/master/LICENSE](https://github.com/PolyAI-LDN/task-specific-datasets/blob/master/LICENSE)

SWDA Stolcke et al., 1998: The dataset has been obtained from [http://compprag.christopherpotts.net/swda.html](http://compprag.christopherpotts.net/swda.html). This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.

E-Intent Welivita and Pu, 2020: The dataset was downloaded from [https://github.com/auradha1992/EmpatheticIntents](https://github.com/auradha1992/EmpatheticIntents). The original dataset is available at [https://github.com/facebookresearch/EmpatheticDialogues](https://github.com/facebookresearch/EmpatheticDialogues) which is under the Creative Commons Attribution 4.0 International license.

MuTual and MuTual-plus Cui et al., 2020: The datasets have been downloaded from [https://github.com/Nealcly/MuTual](https://github.com/Nealcly/MuTual). Licensing is unclear; the authors do not mention any license information or terms of use.

DailyDialog++ Sai et al., 2020: The dataset was downloaded from [https://github.com/iitmnlp/DailyDialog-plusplus](https://github.com/iitmnlp/DailyDialog-plusplus). The data is available under the MIT License.

 

rMax or Reddit-727M conversational-data Henderson et al. (2019): the dataset has been obtained from [https://github.com/PolyAI-LDN/conversational-datasets/tree/master/reddit](https://github.com/PolyAI-LDN/conversational-datasets/tree/master/reddit). The dataset is available under the Apache License Version 2.0.

## References

* Adiwardana et al. (2020) Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a human-like open-domain chatbot. _arXiv preprint arXiv:2001.09977_.
* Alemi et al. (2017) Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. 2017. Deep variational information bottleneck. In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
* Belghazi et al. (2018) Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, R. Devon Hjelm, and Aaron C. Courville. 2018. Mutual information neural estimation. In _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmassan, Stockholm, Sweden, July 10-15, 2018_, volume 80 of _Proceedings of Machine Learning Research_, pages 530-539. PMLR.
* Casanueva et al. (2020) Inigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic. 2020. Efficient intent detection with dual sentence encoders. In _Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI_, pages 38-45, Online. Association for Computational Linguistics.
* Cover (1999) Thomas M Cover. 1999. _Elements of information theory_. John Wiley & Sons.
* Cui et al. (2020) Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou. 2020. MuTual: A dataset for multi-turn dialogue reasoning. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 1406-1416, Online. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In _EMNLP_, pages 6894-6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Gao et al. (2020) Xiang Gao, Yizhe Zhang, Michel Galley, Chris Brockett, and Bill Dolan. 2020. Dialogue response ranking training with large-scale human feedback data. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 386-395, Online. Association for Computational Linguistics.
* Gunel et al. (2020) Beliz Gunel, Jingfei Du, Alexis Conneau, and Veselin Stoyanov. 2020. Supervised contrastive learning for pre-trained language model fine-tuning. In _International Conference on Learning Representations_.
* Henderson et al. (2019) Matthew Henderson, Pawel Budzianowski, Inigo Casanueva, Sam Coope, Daniela Gerz, Girish Kumar, Nikola Mrksic, Georgios Spithourakis, Pei-Hao Su, Ivan Vulic, and Tsung-Hsien Wen. 2019. A repository of conversational datasets. In _Proceedings of the First Workshop on NLP for Conversational AI_, pages 1-10, Florence, Italy. Association for Computational Linguistics.
* Henderson et al. (2020) Matthew Henderson, Inigo Casanueva, Nikola Mrksic, Pei-Hao Su, Tsung-Hsien Wen, and Ivan Vulic. 2020. ConvexRT: Efficient and accurate conversational representations from transformers. In _EMNLP_, pages 2161-2174, Online. Association for Computational Linguistics.
* Hjelm et al. (2019) R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. 2019. Learning deep representations by mutual information estimation and maximization. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
* Hosseini-Asl et al. (2020) Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-oriented dialogue. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
* Kong et al. (2020) Lingpeng Kong, Cyprien de Masson d'Autume, Lei Yu, Wang Ling, Zihang Dai, and Dani Yogatama. 2020. A mutual information maximization perspective of language representation learning. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
* Li et al. (2016) Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and William B Dolan. 2016. A diversity-promoting objective function for neural conversation models. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 110-119.

 

Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A manually labelled multi-turn dialogue dataset. In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 986-995, Taipei, Taiwan. Asian Federation of Natural Language Processing.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Mehri et al. (2019) Shikib Mehri, Evgenia Razumovskaia, Tiancheng Zhao, and Maxine Eskenazi. 2019. Pretraining methods for dialog context representation learning. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3836-3845, Florence, Italy. Association for Computational Linguistics.
* Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. _arXiv preprint arXiv:1301.3781_.
* van den Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. _arXiv preprint arXiv:1807.03748_.
* Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
* Pimentel et al. (2020) Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. 2020. Information-theoretic probing for linguistic structure. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4609-4622, Online. Association for Computational Linguistics.
* Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. _OpenAI blog_, 1(8):9.
* Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. _arXiv preprint arXiv:1910.10683_.
* Roller et al. (2021) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021. Recipes for building an open-domain chatbot. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 300-325, Online. Association for Computational Linguistics.
* Bai et al. (2020) Ananya B. Bai, Akash Kumar Mohankumar, Siddhartha Arora, and Mitesh M. Khapra. 2020. Improving dialog evaluation with a multi-reference adversarial dataset and large scale pretraining. _Transactions of the Association for Computational Linguistics_, 8:810-827.
* Schroff et al. (2015) Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 815-823.
* Song and Ermon (2020) Jiaming Song and Stefano Ermon. 2020. Understanding the limitations of variational mutual information estimators. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
* Stolcke et al. (1998) Andreas Stolcke, Elizabeth Shriberg, Rebecca Bates, Noah Coccaro, Daniel Jurafsky, Rachel Martin, Marie Meteer, Klaus Ries, Paul Taylor, Carol Van Ess-Dyekema, et al. 1998. Dialog act modeling for conversational speech. In _AAAI Spring Symposium on Applying Machine Learning to Discourse Processing_, pages 98-105.
* Tishby et al. (2000) Naftali Tishby, Fernando C Pereira, and William Bialek. 2000. The information bottleneck method. _arXiv preprint physics/0004057_.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_, pages 5998-6008.
* Vulic et al. (2021) Ivan Vulic, Pei-Hao Su, Samuel Coope, Daniela Gerz, Pawel Budzianowski, Iliogo Casanueva, Nikola Mrksic, and Tsung-Hsien Wen. 2021. ConvFiT: Conversational fine-tuning of pretrained language models. In _EMNLP_, pages 1151-1168, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Welivita and Pu (2020) Anuradha Welivita and Pearl Pu. 2020. A taxonomy of empathetic response intents in human social conversations. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 4886-4899, Barcelona, Spain (Online). International Committee on Computational Linguistics.
* Wu et al. (2020) Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, and Caiming Xiong. 2020. TOD-BERT: Pre-trained natural language understanding for task-oriented dialogue. In _EMNLP_, pages 917-929, Online. Association for Computational Linguistics 

Yi-Ting Yeh and Yun-Nung Chen. 2019. QAnfomax: Learning robust question answering system by mutual information maximization. In _EMNLP-IJCNLP_, pages 3370-3375, Hong Kong, China. Association for Computational Linguistics.
* Yoo et al. (2020) Kang Min Yoo, Hanbit Lee, Franck Dernoncourt, Trung Bui, Walter Chang, and Sang-goo Lee. 2020. Variational hierarchical dialog autoencoder for dialog state tracking data augmentation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 3406-3425.
* Zhang et al. (2020) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. DIALOGPT : Large-scale generative pre-training for conversational response generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_, pages 270-278, Online. Association for Computational Linguistics.

## Appendix A Mutual Information Estimators

In this paper, we experiment with various different MI estimators, and found InfoNCE-S to be the best (both in terms of accuracy as well as training speed). The mathematical formulation of these estimators is provided below.

1. InfoNCE was proposed by Oord et al. (2018). It connects to the mutual information value \(I(X;Y)\) as, \[I(X;Y) \geq\log(N)-L_{N}\] \[L_{N} =-\mathbb{E}_{P_{XY}}\left[\log\frac{e^{f(x,y)}}{\sum_{y^{\prime }\in Y}e^{f(x,y^{\prime})}}\right]\]
2. MINE Belghazi et al. (2018) \[I(X;Y)\geq sup_{\theta\in\Theta}\big{[}I_{\theta}^{(MINE)}(X;Y)=\] \[\mathbb{E}_{P_{XY}}[T(x,y)]-\log\mathbb{E}_{P_{X}\times P_{Y}}[ e^{T(x,y)}]\big{]}\]
3. JSD Hjelm et al. (2019) \[I_{\theta}^{(JSD)}(X;Y) =\mathbb{E}_{P_{XY}}[-sp(-T(x,y))]\] \[-\mathbb{E}_{P_{X}\times P_{Y}}[sp(T(x,y))]\]
4. SMILE Song and Ermon (2020) \[I_{\theta}^{(smile)}(X;Y) =\mathbb{E}_{P_{XY}}[T(x,y)]\] \[~{}~{}~{}~{}~{}~{}-\log\mathbb{E}_{P_{X}\times P_{Y}}[clip(e^{T( x,y)},e^{-\tau},e^{\tau})]\]

Use of the InfoMax objective for self-supervised learning has been more prevalent in the computer vision domain than in NLP. Although as Kong et al. (2020) have previously shown, many existing loss functions used for training NLP models can be derived directly from the InfoMax framework. Kong et al. (2020) had only focused on various language model objectives that focus on words given the surrounding context. The authors showed that this objective translates to maximizing mutual information between the context and the missing word within the context.

In dialog domain also, InfoMax-equivalent loss functions have been used. First, Henderson et al. (2020) used contrastive formulation of the response selection task as a pretraining objective for dialog representation. Other prior works on response selection models often used a binary-cross entropy loss for training. Both these loss functions are actually equivalent to various lower bound estimators for mutual information. In the QAInfoMax model Yeh and Chen (2019), the authors used the DeepInfoMax loss function Hjelm et al. (2019) as a regularizer and showed that representations learned with or in-presence of an InfoMax regularizer are more resilient to adversarial attacks while maintaining the same level of task performance. We also observe the same effect in our DD++/cross experiments. This is because of the self-supervised yet task-specific nature of the loss function.

## Appendix B Response Retrieval Experiment

We wanted to investigate if the proposed model can rank good responses higher compared to more generic/bland ones. Hence to test against an extreme setting we simulate a response selection task for a very large pool using the test set of Daily Dialog Li et al. (2017) dataset. We took all the \(\sim\)7000 responses from test set of the daily dialog dataset as the response pool. Next, for a few randomly selected context examples, we illustrate the top two ranked as well as ground truth responses for two full conversations in Tables 5 and 6. Of course, the

Figure 3: Validation-MI profile during pretraining



ground truth response was removed from the pool for each context. The ranking of responses were done using the \(f(c,r)\) function from the trained DMI_Base model. From the examples of response selection, we can observe that the model is able to both avoid blend responses and select responses that are relevant to the current context even from such a large pool. This shows the usefulness of dialog specific pretrained representation trained using the DMI objective.

## Appendix C Prediction Samples and Error Analysis

In Table 7, we show sample predictions from the DailyDialog++/cross task (Sai et al., 2020). As DialoGPT has the best performance in the probe setup, on this task among the baselines, we choose it for error analysis. We randomly sampled 11 instances where the DialoGPT model made a mistake and observed the behavior of our DMI_Base model on these samples. We see that our model correctly predicts for all 6 out of 6 negative samples and out of the 5 positive samples DMI_Base predicts the label of 2 samples correctly (overall 8/11 correct predictions by our model). This shows that our model has a better understanding of the context and response inputs, which makes it robust against the adversarial negative samples. As can be seen in samples 2, 3, 5 and 6, the incorrect predictions by the DialoGPT model might have been caused by presence of common or similar meaning tokens (_cook_, _food; million; long; employee_) between context and response. This means that DialoGPT often relies on weak token-based cues for prediction.

For error analysis on the Empathetic-Intent (E-Intent) task (Welivita and Pu, 2020), we chose the ConveRT model as the baseline to compare against predictions from our DMI_Base model. First, we randomly select 10 samples from the test set of the E-Intent task where the baseline ConveRT model makes a mistake. Then the predictions from the DMI_Base model are observed on these 10 samples. The input utterances, true labels and the predictions made by the model are shown in Table 8. Out of these 10 samples, DMI_Base is able to predict the labels for 6 instances correctly. We notice that though sample inputs often contain more than one emotion, the one denoted by the gold label is generally the primary one. Our model is able to capture this emotion correctly more often than the baseline, with such mixed-emotion samples.

Fig. 4 shows the confusion matrix for our DMI_Base model for the Empathetic-Intent task. The accuracy is highest for afraid, acknowledging and questioning classes (each above 95%). Some of the most confusing pairs of classes are (annoyed, wishing), (anxious, apprehensive), (caring, confident), (content, grateful), (content, lonely).

 



\begin{table}
\begin{tabular}{|p{142.3pt}|l|l|} \hline Dialog & Rank-1 Response & Rank-2 Response \\ \hline \hline
**S1:** Could I have my bill , please ? & Sure . Here is your receipt . & Sure . Your cash back is $ 13 . And we’ll bring out your files in two minutes . \\ \hline
**S2:** Certainly , sfr . & Thanks . Now can I make the full deposit ? & Thank you ! Would you like an aisle seat or a window seat ? \\ \hline
**SI:** I’ m afraid there's been a mistake . & I am really sorry too , maybe I can give you a call sometime . & Oh , I’m sorry.However , if you could help me out , I’ll double the pay for the hours worked \\ \hline
**S2:** I’m sorry , sfr . What seems to be the trouble ? so I needed the money back & Not much . I had to pay an unexpected bill , So I need the money back \\ \hline
**SI:** I believe you have charged me twice for the same thing . Look , the figure of 6.5 dollar appears here , then again here . & One moment , please , sir ... Here’s your hill . Would you like to check and see if the amount is correct ? \\ \hline \end{tabular}
\end{table}
Table 6: Retrieval Example 2: Top two ranked responses from a large pool, as well as ground truth response for a conversation. Note that every line corresponds to one utterance in the conversation either from speaker S1 or speaker S2. **How to read the table:** For any context (all entries in first column upto any row), the ground truth response can be automatically obtained from the immediate next entry in first column. The response selected by the model in any cell \((t,1~{}\text{or}~{}2)\) is for the true context from row \(1\) to row \(t\) in the first column.

\begin{table}
\begin{tabular}{|p{190.7pt}|p{180.7pt}|p{180.7pt}|} \hline Dialog & Rank-1 Response & Rank-2 Response \\ \hline \hline
* Hey man , you wanna buy some weed ? & Yes , smoking or non-smoking ? & Sure , what
* sp ? \\ \hline
**S2:** Some what ? & My cigarettes . & Come on man ! I even got dope and acid ! Try some chronic ! \\ \hline
**Si:** Weed ! You know ? Pot , Ganja , Mary Jane some chronic ! & T ms story , Mrs . Lin , We don’t allow smoting in the office. & Ha ! That’s right . You used to smoke . I forgot ! \\ \hline
**S2:** Oh , numn , no thanks . & Come on man ! I even got dog and acid ! Try some ! & Come on , you can at least try a little , besides your cigarette. \\ \hline
**Si:** I also have blow if you prefer to do a few lines . & No , thanks . On , yes , could I have some juice for the main bar ? & No , thanks . I ’ve had enough . I’ll have my bill , please. \\ \hline
**S2:** No , I am ok , really . & Great ! Would you prefer smoking or non-smoking ? & Come on , you can at least try a little , besides your cigarette. \\ \hline
**Si:** Come on man ! I even get dope and acid ! Try some ! & I will , but let’s go to the disco after dimer . & No, thanks . I’ve had more than enough.In fact , I must be running along . \\ \hline
**S2:** Do you really have all of these drugs ? Where do you get them from ? & I found them at JC Pennies . & They are my personal effects . \\ \hline
**Si:** I got my connections ! Just tell me what you want and ! It even give you one once free . & Himm ... If I tell you what . It’ll lend you what & No , thanks . Oh , yes , could I have some juice for the mini bar ? \\ \hline
* Sounds good ! Let
* s see , I want . & Here you go,On any other day , it would cost me a fortune , but it ‘s on special offer today work , but otherwise , you “Il have to re-do it it. \\ \hline
* Yeah ? & But of course ! Well , it
* s been great talking to you , but I have to get going . & Good . I was thinking that I’d like to invite you to watch a movie . I can meet you at the cinema gate . \\ \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 5: Retrieval Example 1: Top two ranked responses from a large pool, as well as ground truth response for a conversation. Note that every line corresponds to one utterance in the conversation either from speaker S1 or speaker S2. **How to read the table:*
* For any context (all entries in first column upto any row), the ground truth response can be automatically obtained from the immediate next entry in first column. The response selected by the model in any cell \((t,1~{}\text{or}~{}2)\) is for the true context from row \(1\) to row \(t\) in the first column.




\begin{table}
\begin{tabular}{l l c c c c c} \hline \hline
**ID** & **Context** & **Candidate Response** & **Gold** & **DialoGPT** & **DMI_Base** \\ \hline  & \multicolumn{5}{c}{_DML Base predicts correctly_} \\ \cline{3-6}
1 & All right. I’ll take it. __eou... Do you like to use chopsticks & When you get closer, you see that each horizontal section is made up of two pieces that converge in a right angle. & 0 & 1 & 0 \\ \hline
2 & And you’ll have to sell your motorcycle. And your cameras. Right? & I go to the temple twice a week so I & 0 & 1 & 0 \\ \(\underline{~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}





\begin{table}
\begin{tabular}{c l c c c} \hline \hline
**ID** & **Input Utterance** & **Gold Label** & **ConveRT** & **DMI_Base** \\ \hline  & \multicolumn{4}{c}{_DMI\_Base predicts correctly_} \\ \multirow{2}{*}{1} & i feel very thankful for everything that i have, i live a really good life in my liking & grateful & content & grateful \\ \multirow{2}{*}{2} & I’m training a new girl at work. She is doing so good for her first week! & \multirow{2}{*}{proud} & confident & proud \\ \cline{3-5}  & It broke my heart today when I went to the grocery store and found out that they were out of Dean’s French Onion Dip. & disappointed & devastated & disappointed \\ \multirow{2}{*}{4} & My wife’s birthday is coming up. I got her a gift and the party planned out way ahead of time this year. & prepared & surprised & prepared \\ \multirow{2}{*}{5} & My friend helped me to pack & grateful & trusting & grateful \\ \multirow{6}{*}{6} & For two years now I’ve been walking with help of a walker, following a botched hip operation. Recently, at a physical therapy session, I was able to walk with a cane the length of the treatment room. I felt quite good about myself! & \multirow{2}{*}{proud} & caring & proud \\ \cline{3-5}  & \multicolumn{4}{c}{_DMI\_Base predicts wrongly_} \\ \multirow{7}{*}{7} & I was trying to plan my wedding by getting a caterer, and they kept blowing us off over and over again. & furious & disappointed & disappointed \\ \cline{3-5}  & Being a successful single mothr. & \multirow{2}{*}{proud} & content & content \\ \cline{3-5}  & We were over at our friend’s house for a dinner and I was in the kitchen helping her cook. I had melted butter in a baking dish to make dessert, and I poured cold milk into it like the recipe said to do. It ended up cracking the dish. I felt bad. I offered to buy her a new one. & guilty & caring & ashamed \\ \multirow{2}{*}{9} & One time I had done really well in a class. I fully expected to get an A in it & \multirow{2}{*}{anticipating} & confident & disappointed \\ \cline{3-5}  & & & & \\ \end{tabular}
\end{table}
Table 8: Example Predictions on the Empathetic-Intent (E-Intent) task by ConveRT and our DMI_Base model.



* [105] M. Cacciari, G. P. Salam, and R. S. Thorneval, "The Monash-LoT: A New Event Catalog for the LHC Run II", _JHEP_**07** (2014) 063, doi:10.1088/1126-6708/2014/063, arXiv:1409.02361v1 [physics.ins-detectors].
* [106] J. Alwall et al., "A new method to combine boosted top quark pair production with a single top quark in the event at NLO QCD", _Phys. Rev. D_**88** no. 6, 054012, doi:10.1103/PhysRevD.88.054012, arXiv:hep-ph/9909344v1 [physics.ins-detectors].
* [107] T. Sjostrand, S. Mrenna, and P. Z. Skands, "PYTHIA 6.4 physics and manual", _Comput. Phys. Commun._**191** (2015) 159, doi:10.1016/j.cpc.2015.01.024, arXiv:1406.0261v1 [physics.ins-detectors-in-the-art].
* [108] CMS Collaboration, "Event generator tunes obtained from underlying event and multiparton scattering measurements", _Eur. Phys. J. C_**76** (2016) 155, doi:10.1140/epjc/s10052-016-3988-x, arXiv:1512.0017v1 [physics.ins-detectors-in-the-art].
* [109] ATLAS Collaboration, "Measurement of the inclusive W boson cross section using the ATLAS detector at the LHC", _Phys. Lett. B_**716** (2016) 1, doi:10.1016/j.physletb.2016.08.021, arXiv:1502.0408v1 [physics.ins-detectors-in-the-art].
* [110] ATLAS Collaboration, "Measurements of the inclusive W boson cross sections in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Rev. D_**88** no. 6, 054012, doi:10.1103/PhysRevD.88.054012, arXiv:1502.0408v1 [physics.ins-detectors-in-the-art].
* [111] ATLAS Collaboration, "Measurement of the inclusive W boson cross section in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Rev. D_**88** no. 6, 054012, doi:10.1103/PhysRevD.88.054012, arXiv:1502.0408v1 [physics.ins-detectors-in-the-art].
* [112] ATLAS Collaboration, "Measurement of the inclusive W boson cross section in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Rev. D_**88** no. 6, 054012, doi:10.1103/PhysRevD.88.054012, arXiv:1502.0408v1 [physics.ins-detectors-in-the-art].
* [113] ATLAS Collaboration, "Measurement of the inclusive W boson cross section in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Rev. D_**88** no. 6, 054012, doi:10.1103/PhysRevD.88.054012, arXiv:1502.0408v1 [physics.ins-detectors-in-the-art].
* [114] ATLAS Collaboration, "Measurement of the inclusive W boson cross section in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Rev. D_**88** no. 6, 054012, doi:10.1103/PhysRevD.88.054012, arXiv:1502.0408v1 [physics.ins-detectors-in-the-art].
* [115] ATLAS Collaboration, "Measurement of the inclusive W boson cross section in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Rev. D_**88** no. 6, 054012, doi:10.1103/PhysRevD.88.054012, arXiv:1502.0408v1 [physics.ins-detectors-in-the-art].
* [116] ATLAS Collaboration, "Measurement of the inclusive W boson cross section in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Rev. D_**88** no. 6, 054012, doi:10.1103/PhysRevD.88.054012, arXiv:1502.0408v1 [physics.ins-detectors-in-the-art].
* [117] ATLAS Collaboration, "Measurement of the inclusive W boson cross section in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Rev. D_**88** no. 6, 054012, doi:10.1103/PhysRevD.88.054012, arXiv:1502.0408v1 [physics.ins-detectors-in-the-art].
* [118] ATLAS Collaboration, "Measurement of the inclusive W boson cross section in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Rev. D_**88** no. 6, 054012, doi:10.1103/PhysRevD.88.054012, arXiv:1502.0408v1 [physics.ins-detectors-in-the-art].
* [119] ATLAS Collaboration, "Measurement of the inclusive W boson cross section in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Rev. D_**88** no. 6, 054012, doi:10.1103/PhysRevD.88.054012, arXiv:1502.0408v1 [physics.ins-detectors-in-the-art].
* [120] ATLAS Collaboration, "Measurement of the inclusive W boson cross section in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Rev. D_**88** no. 6, 054012, doi:10.1103/PhysRevD.88.054012, arXiv:1502.0408v1 [physics.ins-detectors-in-the-art].
* [121] ATLAS Collaboration, "Measurement of the inclusive W boson cross section in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Rev. D_**88** no. 6, 054012, doi:10.1103/PhysRevD.88.054012, arXiv:1502.0408v1 [physics.ins-detectors-in-the-art].
* [122] ATLAS Collaboration, "Measurement of the inclusive W boson cross section in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Rev. D_**88** no. 6, 054012, doi:10.1103/PhysRevD.88.054012, arXiv:1502.0408v1 [physics.ins-detectors-in-the-art].
* [123] ATLAS Collaboration, "Measurement of the inclusive W boson cross section in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Rev. D_**88** no. 6, 054012, doi:10.1103/PhysRevD.88.054012, arXiv:1502.0408v1 [physics.ins-detectors-in-the-art].
* [124] ATLAS Collaboration, "Measurement of the inclusive W boson cross section in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Rev. D_**88** no. 6, 054012, doi:10.1103/PhysRevD.88.054012, arXiv:1502.0408v1 [physics.ins-detectors-in-the-art].
* [125] ATLAS Collaboration, "Measurement of the inclusive W boson cross section in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Rev. D_**88** no. 6, 054012, doi:10.1103/PhysRevD.88.054012, arXiv:1502.0408v1 [physics.ins-detectors-in-the-art].
* [126] ATLAS Collaboration, "Measurement of the inclusive W boson cross section in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Rev. D_**88** no. 6, 054012, doi:10.1103/PhysRevD.88.054012, arXiv:1502.0408v1 [physics.ins-detectors-in-the-art].
* [127] ATLAS Collaboration, "Measurement of the inclusive W boson cross section in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Rev. D_**88** no. 6, 054012, doi:10.1103/PhysRevD.88.054012, arXiv:1502.0408v1 [physics.ins-detectors-in-the-art].
* [128] ATLAS Collaboration, "Measurement of the inclusive W boson cross section in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Rev. D_**88** no. 6, 054012, doi:10.1103/PhysRevD.88.054012, arXiv:1502.0408v1 [physics.ins-detectors-in-the-art].
* [129] ATLAS Collaboration, "Measurement of the inclusive W boson cross section in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Rev. D_**88** no. 6, 054012, doi:10.1103/PhysRevD.88.054012, arXiv:1502.0408v1 [physics.ins-detectors-in-the-art].
* [200] ATLAS Collaboration, "Measurement of the inclusive W boson cross section in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Rev. D_**88** no. 6, 054012, doi:10.1103/PhysRevD.88.054012, arXiv:1502.0408v1 [physics.ins-detectors-in-the-art].
* [21] ATLAS Collaboration, "Measurement of the inclusive W boson cross section in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Rev. D_**88** no. 6, 054012, doi:10.1103/PhysRevD.88.054012, arXiv:1502.0408v1 [physics.ins-detectors-in-the-art].
* [22] ATLAS Collaboration, "Measurement of the inclusive W boson cross section in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Rev. D_**88** no. 6, 054012, doi:10.1103/PhysRevD.88.054012, arXiv:1502.0408v1 [physics.ins-detectors-in-the-art].
* [23] ATLAS Collaboration, "Measurement of the inclusive W boson cross section in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Rev. D_**88** no. 6, 054012, doi:10.1103/PhysRevD.88.054012, arXiv:1502.0408v1 [physics.ins-detectors-in-the-art].
* [24] ATLAS Collaboration, "Measurement of the inclusive W boson cross section in pp collisions at \(\sqrt{