SlowGuess/NAACL_2022_OCR huggingface dataset:
During pretraining, we compare the checkpoints from different epochs and across hyperparameter settings in terms of the bits of mutual information extracted by the trained representation on an unseen set of dialogs. This is calculated as  $MI_{valid} = \log (N) - L_N$  (see §3 for more details). As per the Information Bottleneck theory (Tishby et al., 2000), the mutual information learned between the two observed random variables can be factorized into two components, namely, predictive and redundant information. Predictive information generally identifies whether the features learned by the representation are useful for a downstream task. The redundant information is caused by features that do not help in any downstream tasks. Such features can exist due to noise or spurious correlation in the dataset, or even overfitting. Hence, we train our final models on a fraction of the rMax dataset but only for one epoch (i.e., we never repeat the samples) which removes any possibility of overfitting.
Predictive features identified based on a fixed set of downstream tasks (Tishby et al., 2000; Alemi et al., 2017) may not be a sufficient to assess other features learned in the training process. Since, ideally, we want to maximize the amount of predictive information in the representation, we compare the bits of MI on the training set against the bits of MI on an unseen validation set, as captured by the learned representation. To make sure that we do not assume anything about the domain or the conversation topics, we use the validation set of dialogs from the open-domain Daily Dialog dataset (Li et al., 2017).
5.6 Downstream Tasks
Instead of focusing on a single downstream task like many previous works on dialog representation learning, we consider a more versatile range of tasks to evaluate the learned representations from DMI or the baseline models. To find out whether a certain representation is effective for some downstream task, we evaluate in two setups: probe and finetune. In both cases, the pretrained model is used along with an MLP classifier of fixed complexity (Pimentel et al., 2020). In probing setup, we only train the parameters of the MLP classifier.
7Training time: A maximum of 2 weeks of training time was allowed for 8-layer and 12-layer models. Though, the training process saturates long before the maximum allowed time, and we evaluate our models based on checkpoints when the best validation scores are first obtained.
Task	Description	Train	Valid	Test	Metric
Banking77	Intent 77-class Classification	8,002	2,001	3,080	Accuracy
SWDA	Dialog Act 41-class Classification	213,543	56,729	4,514	Accuracy
MuTual	Reasoning as Response Selection	25,516	2,836	3,544	R@1, R@2, MRR
MuTual Plus	MuTual + Safe response candidate	25,516	2,836	3,544	R@1, R@2, MRR
DD++	Dialog Evaluation	92,590	10,280	11,420	Accuracy
DD++/Adv	Train: Adv. neg., Test: Adv neg. samples	92,590	10,280	11,420	Accuracy
DD++/Cross	Train: Random neg., Test: Adv neg. samples	92,590	10,280	11,420	Accuracy
DD++/Full	Train: All samples, Test: Adv. neg. samples	138,885	10,280	11,420	Accuracy
Empathetic Intent	Emotion and Intent 44-class Classification	25,023	3,544	3,225	Accuracy
Table 2: Downstream task details. Adv.: Adversarial, Neg.: Negative
In finetuning setup, we also train the pretrained model parameters along with the MLP classifier parameters. We use the context and response representations from our models as the input to the MLP classifier.
For downstream tasks, we have two reasoning tasks based on the MuTual dataset (Cui et al., 2020), three classification tasks based on conversational intent detection (Casanueva et al., 2020), emotion detection (Welivita and Pu, 2020) and act classification (Stolcke et al., 1998), and four dialog evaluation tasks based on the DailyDialog++ dataset  $(\mathrm{DD} + + ,$  Sai et al.,  $2020)^{8}$  . Table 2 shows dataset details and metrics for these nine tasks. Both MuTual and DailyDialog++ datasets have an adversarial configuration for the respective tasks, which allows us to assess each of the evaluated models in adversarial settings also.
6.1 Pretraining DMI based Representations
During pretraining, we used "Validation MI" to evaluate model checkpoints. As the goal of our models is to learn a representation that captures maximum MI between the context and the response texts, this metric tracks how well the learned representation captures the mutual information between contexts and responses of unseen dialogs.
We use the validation split from Daily Dialog dataset as our validation set for evaluation the model during pretraining. It is not specific to a domain and, hence, covers a versatile range of topics. This set comprises 1,000 full conversations between two persons which on unrolling leads to 7,069 context-response (CR) pairs. We illustrate the variation in validation-MI metric against training steps in Fig. 3 in the Appendix.
6.2 Comparison of Representations on Downstream Task Performance
In this set of experiments, we probe/finetune the DMI models with various downstream tasks that require knowledge of many different types of dialog-understanding features. The results of our probing and finetuning experiments are shown in Table 3.
We have used two types of models as our baselines: generic pretrained models and dialog-specific pretrained models. RoBERTa, BERT, T5 (Raffel et al., 2019), GPT-2 are all trained on large corpora of generic web-crawled English text. But, since these models were not specifically pretrained on any dialog corpus, they may suffer from poor performance on certain dialog understanding tasks. Hence, we consider DialogGPT $^{9}$ , DialogRPT, DEB and ConveRT models, which were trained on conversational data. For DialogRPT, we used "human-vs-rand" checkpoint released by authors. All models are 12-layer except Blender_Small (8 layers), ConveRT (6 layers), DialogRPT (24 layers) and DMI_Medium (8 layers). We used the publicly available model checkpoints for all baselines, wherever possible. The ConveRT model's checkpoint has been removed from Github $^{10}$  by its authors. Hence, it was only possible for us to MLP-probe the representations, without finetuning of the model, based on a cached version released by another user under a valid license $^{11}$ . Pretrained checkpoints for Meena, ContextPretrain and ConvFiT are not available, and hence we do not compare with them.
6.2.1 Results in Probing Setup
We observe that, on average, DEB and ConveRT have good performance among the baselines. How-
Note that DailyDialog++ is different from DailyDialog.
${}^{9}$  DialogGPT and DEB are based on GPT-2 and BERT models and were further pretrained on conversations from Reddit. They use the original loss functions of GPT-2/BERT.
10https://github.com/PolyAI-LDN/ polyai-models
$^{11}$ https://github.com/davidalami/Convert
B77	SWDA	E-Intent	MuTual	MuTual Plus	DD++	DD+/adv	DD+/cross	DD+/full
Model	Acc.	Acc.	Acc.	R@1	R@2	MRR	R@1	R@2	MRR	Acc.	Acc.	Acc.	Acc.
Probing	RoBERTa_Base	72.84	67.18	50.45	49.70	75.20	70.00	43.60	66.60	65.10	55.75	84.20	65.11	68.76
BERT_Base	72.74	67.99	46.84	45.40	72.80	67.30	42.60	67.70	64.90	60.39	86.56	65.25	72.50
T5_Base	60.82	68.79	44.50	43.20	69.40	65.60	38.30	65.70	62.20	57.46	84.14	61.23	63.35
GPT-2_Small	76.64	69.17	49.94	44.92	70.54	66.60	40.75	66.70	63.46	67.37	82.06	67.53	73.93
DialoGPT_Small	53.00	65.10	43.42	29.80	53.50	55.15	25.51	57.56	54.05	63.63	78.02	70.61	70.77
Blender_Small	70.39	70.11	48.52	41.42	68.06	64.29	42.89	68.85	65.18	60.07	65.14	57.76	68.20
ConveRT	89.88	71.36	55.47	45.30	72.00	67.00	40.90	69.00	64.30	79.14	88.67	69.59	80.86
DialogRPT	81.54	67.92	50.74	39.50	66.80	63.00	34.20	61.50	59.20	74.11	81.29	68.49	67.20
DEB	79.18	68.50	45.31	45.10	74.00	67.50	45.00	67.70	66.00	70.66	86.07	67.25	67.77
DMI_Small	89.81	72.33	55.72	51.24	74.94	70.76	46.39	70.09	67.14	85.01	91.51	75.75	86.34
DMI_Medium	90.42	72.49	57.33	52.48	75.62	71.47	46.61	72.46	67.79	85.80	91.38	76.73	86.94
DMI_Base	91.43	72.73	60.00	52.48	76.41	71.65	48.98	71.33	68.73	86.91	91.98	79.15	88.32
Δ	1.55	1.37	4.53	2.78	1.21	1.65	3.98	2.33	2.73	7.77	3.31	8.54	7.46
Finetuning	RoBERTa_Base	92.75	73.61	62.81	48.42	77.20	69.70	49.55	73.70	69.50	90.00	95.70	73.76	91.09
BERT_Base	92.27	72.29	60.12	47.86	73.93	68.80	49.10	72.35	69.00	87.05	94.33	67.70	88.82
T5	89.11	73.77	60.66	49.77	73.93	69.80	43.00	66.93	64.90	82.03	90.89	65.85	85.63
GPT-2_Small	92.49	72.62	58.44	48.42	72.69	68.90	45.71	70.99	67.10	85.69	93.60	68.43	87.83
DialoGPT_Small	92.59	73.48	59.33	49.32	75.17	69.80	47.86	73.02	68.44	83.68	91.99	64.06	85.54
Blender_Small	91.59	71.10	58.31	52.93	75.85	71.80	47.97	70.99	68.30	86.83	92.29	66.39	87.82
DialogRPT	92.70	72.02	62.13	52.14	76.19	71.40	46.95	70.54	67.66	90.26	95.81	73.34	91.25
DEB	92.53	72.14	59.69	48.19	74.49	69.00	46.95	70.65	67.80	85.74	94.05	64.42	89.02
DMI_Small	92.44	71.29	61.05	55.42	75.28	72.92	47.63	72.01	68.19	87.57	94.99	77.33	88.96
DMI_Medium	92.76	71.53	62.88	55.76	77.88	73.56	50.68	73.25	70.04	89.12	95.63	78.26	90.80
DMI_Base	93.93	74.50	64.62	56.43	79.91	74.27	52.14	75.06	71.09	91.03	96.39	81.69	92.61
Δ	1.18	0.73	1.81	3.50	2.71	2.47	2.59	1.36	1.59	0.77	0.59	7.93	1.35
Table 3: Results from probing (top) and finetuning (bottom) setups on 9 downstream tasks for assessing dialog understanding. (DD++): DailyDialog++, B77: Banking77 task, R@k: Recall at k, MRR: Mean reciprocal rank). Our model consistently performs better than SOTA on all the tasks in both probing as well as finetuning setups.
B77	SWDA	E-Intent	MuTual	MuTual Plus	DD++	DD++/adv	DD++/cross	DD++/full
Model	Acc.	Acc.	Acc.	R@1	R@2	MRR	R@1	R@2	MRR	Acc.	Acc.	Acc.	Acc.
DMI_Base	93.93	74.50	64.62	56.43	79.91	74.27	52.14	75.06	71.09	91.03	96.39	76.01	92.61
DMI_Base - Sym	93.28	72.69	65.18	57.34	77.88	74.32	48.08	72.69	68.60	90.94	96.65	76.45	93.13
DMI_Base - RoB	92.34	74.10	60.96	53.84	77.31	72.47	50.34	72.80	69.81	87.23	92.95	73.53	87.85
DMI_Base - Sym - RoB	91.59	73.55	60.71	54.06	75.40	72.24	47.97	71.45	68.24	86.79	92.96	70.29	87.13
Table 4: Ablation study results for the finetune setup for our base model on 9 downstream tasks. “-RoB” → No RoBERTa initialization. “-Sym” → Training with non-symmetric version of InfoNCE. (DD++: DailyDialog++, B77: Banking77 task, R@k: Recall at k, MRR: Mean reciprocal rank).
ever, the RoBERTa model outperforms all other baselines on the MuTual task by a significant margin. In the MuTual Plus task, the DEB model outperforms other models in the R@1 and MRR metrics. ConveRT performs the best among all baselines on the other tasks. ConveRT's loss function is also contrastive in nature and is similar to ours. This explains the model's generally high strength across the tasks among all the baselines.
Our DMI_Base beats ConveRT on all the tasks, and DMI_Medium beats the baseline on 7 out of 9 tasks. We believe  $\mathrm{DD} + +$  tasks to be the most demanding ones with respect to context-level understanding. Here, all non-dialog baselines have a weaker performance, with DEB and ConveRT being the best of the bunch. These are also the tasks where our models excel the most, with both DMI_Base and DMI_Base beating all baselines
with strong margins.  $\mathrm{DD} + + /$  cross is the most difficult among all four  $\mathrm{DD} + +$  tasks. Here, the model is trained on random negative samples and tested on a dataset with human-curated adversarial negatives. Our DMI_Base beats the best baseline on  $\mathrm{DD} + + /$  cross by 8.54 points. This shows the superior quality of context representations from our models.
6.2.2 Results in Finetuning Setup
In the finetuning setup, on average, RoBERTa and DialogRPT have good performance among the baselines. DialogRPT performs well for  $\mathrm{DD}++$  tasks while Blender works well for the MuTual task. For all other tasks, RoBERTa is the best baseline, even outperforming models especially trained for dialog tasks (like DialogGPT).
Similar to the probe setup, DMI_Base beats base
line methods by significant margins. In general, finetune results are better than probe results across all models, as expected.
Our large-scale RoBERTa-initialized DMI_Base model outperforms the best baseline for all tasks, by a considerable margin. Additionally, our DMI-based models are able to perform well uniformly across all tasks, unlike even baselines like DialogPPT, DialogRPT and Blenderbot models which are explicitly trained on dialog data. This makes DMI the best overall model for dialog related tasks. Across multiple tasks, we show qualitative examples where our proposed DMI-based models provide accurate results, in the Appendix.
We evaluate the importance of using RoBERTa based pretraining as well as the symmetric version of the InfoNCE loss in Table 4. We observe that RoBERTa based pretraining helps significantly across all tasks. The symmetric InfoNCE improves performance for SWDA and MuTual Plus tasks.
7 Conclusions and Future work
In this paper, we proposed the concept of Discourse Mutual Information (DMI) which is better suited for learning dialog-specific features in a self-supervised manner. Using the InfoMax principle we formulated a pretraining method for dialog-specific representation learning. Across 9 downstream dialog understanding tasks, our 12-layer model outperforms state-of-the-art methods. Further, we showed that on most of these tasks, even our 8-layer model outperforms standard 12-layer pretrained models. These experiments show the potential of the proposed DMI objective towards building dialog understanding models. We will make the code and pretrained model checkpoints available on request, instructions can be found here https://bsantraigi.github.io/DMI. Although we experimented only with dialog modeling in this paper, we believe that the proposed DMI objective is generic enough to be applied to any type of discourse in any domain. In the future, we would like to explore how to harness DMI representations for generative conversation modeling.
This work was partially supported by Microsoft Academic Partnership Grant (MAPG) 2021. The
first author was also supported by Prime Minister's Research Fellowship (PMRF), India.
9 Ethical considerations
Like many other pretrained language representation models, the proposed model may also have learned patterns associated with exposure bias. Interpretability associated with the output is rather limited, hence users should use the outputs carefully. The proposed model ranks possible response candidates, and does not filter out any "problematic" candidates. Thus, for applications, where candidate responses could be problematic, (e.g., offensive, hateful, abusive, etc.), users should carefully filter them out before providing them as input to our model.
All the datasets used in this work are publicly available. We did not collect any new dataset as part of this work.
Banking77 Casanueva et al., 2020 has been obtained from https://github.com/PolyAI-LDN/ task-specific-datasets. It is available under a Creative Commons Attribution 4.0 International license with details here<sup>12</sup>.
SWDA Stolcke et al., 1998: The dataset has been obtained from http://compprag.christopherpotts.net/swda.html. This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.
E-Intent Welivita and Pu, 2020: The dataset was downloaded from https://github.com/ anuradha1992/EmpatheticIntents. The original dataset is available at https: //github.com/facebookresearch/ EmpatheticDialogues which is under the Creative Commons Attribution 4.0 International license.
MuTual and MuTual-plus Cui et al., 2020: The datasets have been downloaded from https://github.com/Nealcly/MuTual. Licensing is unclear; the authors do not mention any license information or terms of use.
DailyDialog++ Sai et al., 2020: The dataset was downloaded from https://github.com/ iitmnp/DailyDialog-plusplus. The data is available under the MIT License.
$^{12}$ https://github.com/PolyAI-LDN/ task-specific-datasets/blob/master/LICENSE
rMax or Reddit-727M conversational-data Henderson et al., 2019: the dataset has been obtained from https://github.com/PolyAI-LDN/conversational-datasets/tree/master/redit. The dataset is available under the Apache License Version 2.0.
Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977.
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. 2017. Deep variational information bottleneck. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, R. Devon Hjelm, and Aaron C. Courville. 2018. Mutual information neural estimation. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 530-539. PMLR.
Inigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 38-45, Online. Association for Computational Linguistics.
Thomas M Cover. 1999. Elements of information theory. John Wiley & Sons.
Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou. 2020. MuTual: A dataset for multi-turn dialogue reasoning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1406-1416, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In EMNLP, pages 6894-6910, Online
and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Xiang Gao, Yizhe Zhang, Michel Galley, Chris Brockett, and Bill Dolan. 2020. Dialogue response ranking training with large-scale human feedback data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 386-395, Online. Association for Computational Linguistics.
Belize Gunel, Jingfei Du, Alexis Conneau, and Veselin Stoyanov. 2020. Supervised contrastive learning for pre-trained language model fine-tuning. In International Conference on Learning Representations.
Matthew Henderson, Paweł Budzianowski, Inigo Casanueva, Sam Coope, Daniela Gerz, Girish Kumar, Nikola Mrksic, Georgios Spithourakis, Pei-Hao Su, Ivan Vulić, and Tsung-Hsien Wen. 2019. A repository of conversational datasets. In Proceedings of the First Workshop on NLP for Conversational AI, pages 1-10, Florence, Italy. Association for Computational Linguistics.
Matthew Henderson, Inigo Casanueva, Nikola Mrksić, Pei-Hao Su, Tsung-Hsien Wen, and Ivan Vulić. 2020. ConveRT: Efficient and accurate conversational representations from transformers. In EMNLP, pages 2161-2174, Online. Association for Computational Linguistics.
R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. 2019. Learning deep representations by mutual information estimation and maximization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-oriented dialogue. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Lingpeng Kong, Cyprien de Masson d'Autume, Lei Yu, Wang Ling, Zihang Dai, and Dani Yogatama. 2020. A mutual information maximization perspective of language representation learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and William B Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 986-995, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Shikib Mehri, Evgeniia Razumovskaia, Tiancheng Zhao, and Maxine Eskenazi. 2019. Pretraining methods for dialog context representation learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3836-3845, Florence, Italy. Association for Computational Linguistics.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. 2020. Information-theoretic probing for linguistic structure. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4609-4622, Online. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of