SlowGuess/NAACL_2022_OCR huggingface dataset:
the European Chapter of the Association for Computational Linguistics: Main Volume, pages 300-325, Online. Association for Computational Linguistics.
Ananya B. Sai, Akash Kumar Mohankumar, Siddhartha Arora, and Mitesh M. Khapra. 2020. Improving dialog evaluation with a multi-reference adversarial dataset and large scale pretraining. Transactions of the Association for Computational Linguistics, 8:810-827.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815-823.
Jiaming Song and Stefano Ermon. 2020. Understanding the limitations of variational mutual information estimators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Andreas Stolcke, Elizabeth Shriberg, Rebecca Bates, Noah Coccaro, Daniel Jurafsky, Rachel Martin, Marie Meteor, Klaus Ries, Paul Taylor, Carol Van Ess-Dykema, et al. 1998. Dialog act modeling for conversational speech. In AAAI Spring Symposium on Applying Machine Learning to Discourse Processing, pages 98-105.
Naftali Tishby, Fernando C Pereira, and William Bialek. 2000. The information bottleneck method. arXiv preprint physics/0004057.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Ivan Vulic, Pei-Hao Su, Samuel Coope, Daniela Gerz, Paweł Budzianowski, Inigo Casanueva, Nikola Mrkšić, and Tsung-Hsien Wen. 2021. ConvFiT: Conversational fine-tuning of pretrained language models. In EMNLP, pages 1151-1168, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Anuradha Welivita and Pearl Pu. 2020. A taxonomy of empathetic response intents in human social conversations. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4886-4899, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, and Caiming Xiong. 2020. TOD-BERT: Pre-trained natural language understanding for task-oriented dialogue. In EMNLP, pages 917-929, Online. Association for Computational Linguistics.
Yi-Ting Yeh and Yun-Nung Chen. 2019. QAIinfomax: Learning robust question answering system by mutual information maximization. In EMNLP-IJCNLP, pages 3370-3375, Hong Kong, China. Association for Computational Linguistics.
Kang Min Yoo, Hanbit Lee, Franck Dernoncourt, Trung Bui, Walter Chang, and Sang-goo Lee. 2020. Variational hierarchical dialog autoencoder for dialog state tracking data augmentation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3406-3425.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. DIALOGPT: Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 270-278, Online. Association for Computational Linguistics.
A Mutual Information Estimators
In this paper, we experiment with various different MI estimators, and found InfoNCE-S to be the best (both in terms of accuracy as well as training speed). The mathematical formulation of these estimators is provided below.
1. InfoNCE was proposed by Oord et al. (2018). It connects to the mutual information value  $I(X;Y)$  as,
$$
I (X; Y) \geq \log (N) - L _ {N}
$$
$$
L _ {N} = - \mathbb {E} _ {P _ {X Y}} \left[ \log \frac {e ^ {f (x , y)}}{\sum_ {y ^ {\prime} \in Y} e ^ {f (x , y ^ {\prime})}} \right]
$$
2. MINE (Belghazi et al., 2018)
$$
I (X; Y) \geq s u p _ {\theta \in \Theta} \left[ I _ {\theta} ^ {(M I N E)} (X; Y) = \right.
$$
$$
\mathbb {E} _ {P _ {X Y}} [ T (x, y) ] - \log \mathbb {E} _ {P _ {X} \times P _ {Y}} [ e ^ {T (x, y)} ] \bigg ]
$$
3. JSD (Hjelm et al., 2019)
$$
\begin{array}{l} I _ {\theta} ^ {(J S D)} (X; Y) = \mathbb {E} _ {P _ {X Y}} [ - s p (- T (x, y)) ] \\ - \mathbb {E} _ {P _ {X} \times P _ {Y}} [ s p (T (x, y)) ] \\ \end{array}
$$
4. SMILE (Song and Ermon, 2020)
$$
\begin{array}{l} I _ {\theta} ^ {(s m i l e)} (X; Y) = \mathbb {E} _ {P _ {X Y}} [ T (x, y) ] \\ - \log \mathbb {E} _ {P _ {X} \times P _ {Y}} [ c l i p (e ^ {T (x, y)}, e ^ {- \tau}, e ^ {\tau}) ] \\ \end{array}
$$
Use of the InfoMax objective for self-supervised learning has been more prevalent in the computer vision domain than in NLP. Although as Kong et al.
(2020) have previously shown, many existing loss functions used for training NLP models can be derived directly from the InfoMax framework. Kong et al. (2020) had only focused on various language model objectives that focus on words given the surrounding context. The authors showed that this objective translates to maximizing mutual information between the context and the missing word within the context.
In dialog domain also, InfoMax-equivalent loss functions have been used. First, Henderson et al. (2020) used contrastive formulation of the response selection task as a pretraining objective for dialog representation. Other prior works on response selection models often used a binary-cross entropy loss for training. Both these loss functions are actually equivalent to various lower bound estimators for mutual information. In the QAInfoMax model (Yeh and Chen, 2019), the authors used the Deep-InfoMax loss function (Hjelm et al., 2019) as a regularizer and showed that representations learned with or in-presence of an InfoMax regularizer are more resilient to adversarial attacks while maintaining the same level of task performance. We also observe the same effect in our  $\mathrm{DD} + + / \mathrm{cross}$  experiments. This is because of the self-supervised yet task-specific nature of the loss function.
Figure 3: Validation-MI profile during pretraining
B Response Retrieval Experiment
We wanted to investigate if the proposed model can rank good responses higher compared to more generic/bland ones. Hence to test against an extreme setting we simulate a response selection task for a very large pool using the test set of Daily Dialog (Li et al., 2017) dataset. We took all the  $\sim 7000$  responses from test set of the daily dialog dataset as the response pool. Next, for a few randomly selected context examples, we illustrate the top two ranked as well as ground truth responses for two full conversations in Tables 5 and 6. Of course, the
ground truth response was removed from the pool for each context. The ranking of responses were done using the  $f(c, r)$  function from the trained DMI_Base model. From the examples of response selection, we can observe that the model is able to both avoid blend responses and select responses that are relevant to the current context even from such a large pool. This shows the usefulness of dialog specific pretrained representation trained using the DMI objective.
C Prediction Samples and Error Analysis
In Table 7, we show sample predictions from the DailyDialog++/cross task (Sai et al., 2020). As DialogGPT has the best performance in the probe setup, on this task among the baselines, we choose it for error analysis. We randomly sampled 11 instances where the DialogGPT model made a mistake and observed the behavior of our DMI_Base model on these samples. We see that our model correctly predicts for all 6 out of 6 negative samples and out of the 5 positive samples DMI_Base predicts the label of 2 samples correctly (overall 8/11 correct predictions by our model). This shows that our model has a better understanding of the context and response inputs, which makes it robust against the adversarial negative samples. As can be seen in samples 2, 3, 5 and 6, the incorrect predictions by the DialogGPT model might have been caused by presence of common or similar meaning tokens (cook, food; million; long; employee) between context and response. This means that DialogGPT often relies on weak token-based cues for prediction.
For error analysis on the Empathetic-Intent (E-Intent) task (Welivita and Pu, 2020), we chose the ConveRT model as the baseline to compare against predictions from our DMI_Base model. First, we randomly select 10 samples from the test set of the E-Intent task where the baseline ConveRT model makes a mistake. Then the predictions from the DMI_Base model are observed on these 10 samples. The input utterances, true labels and the predictions made by the model are shown in Table 8. Out of these 10 samples, DMI_Base is able to predict the labels for 6 instances correctly. We notice that though sample inputs often contain more than one emotion, the one denoted by the gold label is generally the primary one. Our model is able to capture this emotion correctly more often than the baseline, with such mixed-emotion samples.
Fig. 4 shows the confusion matrix for our
DMI_Base model for the Empathetic-Intent task. The accuracy is highest for afraid, acknowledging and questioning classes (each above  $95\%$ ). Some of the most confusing pairs of classes are (annoyed, wishing), (anxious, apprehensive), (caring, confident), (content, grateful), (content, lonely).
Dialog	Rank-1 Response	Rank-2 Response
S1: Hey man , you wanna buy some weed ?	Yes , smoking or non-smoking ?	Sure , what ’ s up ?
S2: Some what ?	My cigarettes .	Come on man ! I even got dope and acid ! Try some !
S1: Weed ! You know ? Pot , Ganja , Mary Jane some chronic !	I'm sorry , Mrs . Lin . We don’t allow smok-
ing in the office .	Ha ! That’s right . You used to smoke . I
forgot !
S2: Oh , umm , no thanks .	Come on man ! I even got dope and acid !
Try some !	Come on , you can at least try a little , besides
your cigarette .
S1: I also have blow if you prefer to do a few lines .	No , thanks . Oh , yes , could I have some
juice for the mini bar ?	No , thanks . I ’ Ve had enough . I’ll have
my bill , please .
S2: No , I am ok , really .	Great ! Would you prefer smoking or non-
smoking ?	Come on , you can at least try a little , besides
your cigarette .
S1: Come on man ! I even got dope and acid ! Try
some !	I will , but let’s go to the disco after dinner .	No , thanks . I’ve had more than enough.In
fact , I must be running along .
S2: Do you really have all of these drugs ? Where
do you get them from ?	I found them at JC Pennies .	They are my personal effects .
S1: I got my connections ! Just tell me what you
want and I ’ ll even give you one ounce for free .	Hmm … I’ll tell you what . I’ll lend you four
thousand dollars , but you have to pay me
back next week .	No , thanks . Oh , yes , could I have some
juice for the mini bar ?
S2: Sounds good ! Let ’ s see , I want .	Here you go. On any other day , it would cost
me a fortune , but it ’ s on special offer today .	All right , Maria . I ’ ll give you until tomor-
row at 4:00 to produce a satisfactory piece of
work , but otherwise , you ’ ll have to re-do
it .
S1: Yeah ?	But of course ! Well , it ’ s been great talking
to you , but I have to get going .	Good . I was thinking that I ’ d like to invite
you to watch a movie . I can meet you at the
cinema gate .
Table 5: Retrieval Example 1: Top two ranked responses from a large pool, as well as ground truth response for a conversation. Note that every line corresponds to one utterance in the conversation either from speaker S1 or speaker S2. How to read the table: For any context (all entries in first column upto any row), the ground truth response can be automatically obtained from the immediate next entry in first column. The response selected by the model in any cell  $(t,1$  or 2) is for the true context from row 1 to row  $t$  in the first column.
Dialog	Rank-1 Response	Rank-2 Response
S1: Could I have my bill, please?	Sure. Here is your receipt.	Sure. Your cash back is $ 13. And we'll bring out your fries in two minutes.
S2: Certainly, sir.	Thanks. Now can I make the full deposit?	Thank you! Would you like an aisle seat or a window seat?
S1: I'm afraid there's been a mistake.	I am really sorry too, maybe I can give you a call sometime.	Oh, I'm sorry. However, if you could help me out, I'll double the pay for the hours worked.
S2: I'm sorry, sir. What seems to be the trouble?	Not much. I had to pay an unexpected bill, so I needed the money back.	Oh, nothing special. I'm just a bit tired.
S1: I believe you have charged me twice for the same thing. Look, the figure of 6.5 dollar appears here, then again here.	One moment, please, sir. ... Here's your bill. Would you like to check and see if the amount is correct?	Sir, I deleted the $ 10, but I had to add a $ 2 service charge to your bill.
Table 6: Retrieval Example 2: Top two ranked responses from a large pool, as well as ground truth response for a conversation. Note that every line corresponds to one utterance in the conversation either from speaker S1 or speaker S2. How to read the table: For any context (all entries in first column upto any row), the ground truth response can be automatically obtained from the immediate next entry in first column. The response selected by the model in any cell  $(t,1$  or 2) is for the true context from row 1 to row  $t$  in the first column.
ID	Context	Candidate Response	Gold	DiaGPT	DMI_Base
DMI_Base predicts correctly
1	All right. I'll take it. _eou_ Do you like to use chopsticks _eou_ Yes, I like using chopsticks.	When you get closer, you see that each horizontal section is made up of two pieces that converge in a right angle.	0	1	0
2	And you'll have to sell your motorcycle. And your cameras. Right? _eou_ Maybe I'll cook once or twice a week. How is that?	I go to the temple twice a week so I prefer vegetarian food.	0	1	0
3	But I heard the box office rose up to 15 million in the first week. _eou_ Box office can't explain everything. I do not think it is cheerful or well-made. The plot is old and the female character is not pretty. _eou_ My sister has given me two tickets for tonight. It is called' The life of Rose', a French movie.	I got 1 million views on my youtube channel in one week.	0	1	0
4	Glad you like it. By the way, is this your first time to China, Mr. White? _eou_ Yes, as a representative of IBM. I hope to conclude some business with you. _eou_ We also hope to expand our business with you.	May I know what and all process you have?	1	0	1
5	Good. I have to go right now. I really hope this meeting doesn't last too long. _eou_ They usually go on for ages. _eou_ I'll stop by if I have time later. Make sure everyone knows that we must stick to the deadlines.	I don't cut my hair because I really like to keep it long.	0	1	0
6	Of course. The main thing is that all our work must be completed on schedule. We even allow our employee to go home early if they finish their work early. _eou_ How often do you have meetings? _eou_ You should attend a department meeting every Monday morning. There are other meetings for people working together on certain projects. Department heads also attend an interdepartmental meeting each week.	In the newsletter, I gave employees column references this week.	0	1	0
7	Sounds interesting! That must be very convenient. _eou_ Yes, you're right. I can blog wherever and whenever I'm on the move. It's especially good when I'm on a business trip and my laptop happens to be away from me. _eou_ How can you do that?	I sank parents money into my business it is not convenient.	0	1	0
8	There is a wait right now to use the computers. _eou_ That's fine. _eou_ Would you please write your name on this list?	Sure, please give me a pen.	1	0	1
DMI_Base predicts wrongly
9	How much cash would you like? _eou_ I want $150. _eou_ Here 's your $150.	Well! I never forget your help.	1	0	0
10	I see, sir. This one is very good. _eou_ Is it? _eou_ You may rest assured. It sells well. _eou_ May I have a look at the introduction?	It has been recommended by top nutritionists.	1	0	0
11	Sir, tell us about your experience with Super Bulk-up. _eou_ Well, it's completely changed my life. _eou_ Tell us how.	The change is right in front of you, isn't it?	1	0	0
Table 7: Sample Predictions from the DD++/Cross task. In DD++/Cross, the models are trained using randomly sampled negatives and tested on curated adversarial negative samples. In each sample, the input context comprises the utterances, previous to the response, spoken by the two participants. Such utterances within a context are delimited by a special token “_eou__”.
ID	Input Utterance	Gold Label	ConveRT	DMI_Base
DMI_Base predicts correctly
1	i feel very thankful for everything that i have, i live a really good life in my liking	grateful	content	grateful
2	I'm training a new girl at work. She is doing so good for her first week!	proud	confident	proud
3	It broke my heart today when I went to the grocery store and found out that they were out of Dean's French Onion Dip.	disappointed	devastated	disappointed
4	My wife's birthday is coming up. I got her a gift and the party planned out way ahead of time this year.	prepared	surprised	prepared
5	My friend helped me to pack	grateful	trusting	grateful
6	For two years now I've been walking with help of a walker, following a botched hip operation. Recently, at a physical therapy session, I was able to walk with a cane the length of the treatment room. I felt quite good about myself!	proud	caring	proud
DMI_Base predicts wrongly
7	I was trying to plan my wedding by getting a caterer, and they kept blowing us off over and over again.	furious	disappointed	disappointed
8	Being a successful single mothr.	proud	content	content
9	We were over at our friend's house for a dinner and I was in the kitchen helping her cook. I had melted butter in a baking dish to make dessert, and I poured cold milk into it like the recipe said to do. It ended up cracking the dish. I felt bad. I offered to buy her a new one.	guilty	caring	ashamed
10	One time I had done really well in a class. I fully expected to get an A in it	anticipating	confident	disappointed
Table 8: Example Predictions on the Empathetic-Intent (E-Intent) task by ConveRT and our DMI_Base model.
Figure 4: Confusion Matrix for our DMI_Base model for the Empathetic-Intent task.