SlowGuess/NAACL_2021_OCR huggingface dataset:
Algorithm 2 shows an outline of the proposed method. At each iteration within the training loop, we back-propagate the  $S$  and  $T$  losses and store the gradients of layer parameters. For parameters related to the  $S$  head, we follow a simple gradient descent update. For body parameters, we only update the parameters if the inner product of the  $S$  and  $T$  tasks is positive or at a small random outcome with the probability of  $\alpha$ . To guarantee the convergence of the source task, we allow all parameters to be selected at each step at least with a very small probability of  $\alpha$ . In our experiments, we consider  $\alpha$  as a hyperparameter taking values in  $\{0.001, 0.005, 0.01, 0.05, 0.1\}$ . Additional care is required when updating the  $T$  head layer parameters as the  $\mathbb{D}_T$  is usually much smaller than  $\mathbb{D}_S$  and the  $T$  head is prone to overfitting. We use a validation set from task  $T$  to detect overfitting for the  $T$  head and early stop the updates. Note that a hyperparameter  $\lambda$  is used to set the frequency of the  $T$  head updates after the early stopping. Having less frequent head updates allows the  $T$  head to gradually improve and adapt to the changes in the body without getting overfitted. In our experiments, we search for proper  $\lambda$  values in  $\{0.001, 0.002, 0.005, 0.01\}$ .
In contrast to other works in the literature which mostly leverage the alignment of concatenated gradients (Lopez-Paz and Ranzato, 2017; Luo et al., 2020), we propose layer-wise similarity measurements providing more granularity and more adaptability. Also, the suggested approach does not require any inner loop optimization process or gradient projection and hence is scalable to large-scale problems. The only computational and memory overhead is to store the model gradients with respect to each task and to compute inner products between the layer parameters.
The method explained in this section is general to few-shot transfer learning and joint training settings where a large source dataset is being used to achieve representations that are most useful for a final target task. For our use-case, we use the suggested approach considering the source task,  $S$ ,
Note that the block selection operation is discrete, either a certain parameter belongs to the block or not, but the distribution  $\mathcal{B}$  can be a continuous or discrete probability distribution.
as the self-supervised contrastive objective and the target task,  $T$ , as the user satisfaction prediction task. In our experiments, after the joint training process, we reinitialize the  $T$  head and finetune the network for the  $T$  task. We found this approach to be helpful to achieve the best results as the jointly trained  $T$  head is often slightly overfitted.
We used PyTorch (Paszke et al., 2017) to train our models. For each case, we continue the training for the maximum number of epochs (10 in our experiments) and select the best model based on the validation performance. We conducted our experiments on a cluster of 48 NVIDIA V100 GPUs (16 GB memory, 8 GPUs per machine). It took between about 6 hours to 27 hours to run individual experiments, depending on the case.
For each experiment, we report Area Under the ROC Curve (AUC-ROC) and Area Under the Precision-Recall Curve (AUC-PR) as the performance measures. The results for the in-domain and out-of-domain held-out test sets are reported
separately. Note that there is an imbalance in the frequency of SAT and DSAT labels, and also there is a difference in the label distribution for the indomain and out-of-domain test sets. To ensure the statistical significance of the results, each experiment is repeated four times using random initializations reporting the mean and standard deviations.
Figure 4 shows a comparison of the in-domain test results for the supervised training and the self-supervised contrastive pretraining methods. For each case, we report the in-domain test performance using models trained with a different number of annotated training samples. The x-axis is plotted in the log scale. It can be seen that the contrastive self-supervised approach is much more data-efficient compared to the supervised approach as it leverages the pool of unlabeled data.
Figure 5 shows a comparison between the supervised training and the self-supervised pretraining methods on the out-of-domain test set. Similar to the in-domain case, there is a significant gap between the labeled data efficiency of these approaches. However, compared to the in-domain case, using even all training samples, the gap does not appear to close. In other words, for the out-of-domain test set the self-supervised approach is not only more data-efficient but also tends to generalize better. In a real-world conversation system, the out-of-domain generalization can be crucial as many different new skills are being developed and included in the system every day, making the traditional in-domain human annotation less practical due to the required annotation turnaround time.
In Figure 6 and Figure 7, we compare the in-domain and out-of-domain performance of the self-supervised pretraining method with the proposed few-shot learning method. As it can be seen from Figure 6, the in-domain AUC-PR and AUC-ROC for the few-shot learning are consistently better than the self-supervised pretraining approach. Note that the performance gap closes at about 5000 samples; perhaps because it is enough training data for fine-tuning and successfully transferring the pretrained model. The out-of-domain performances as reported in Figure 7 show better results for the few-shot approach but the margin of improvement is relatively smaller than the in-domain case.
Note that in the presented results, we focused our comparisons to methods that are scalable and lever
Figure 4: Comparison of the supervised training baseline and the proposed self-supervised pretraining methods for the in-domain test set using different number of training samples (left: AUC-PR, right: AUC-ROC).
Figure 5: Comparison of the supervised training baseline and the proposed self-supervised pretraining methods for the out-of-domain test set using different number of training samples (left: AUC-PR, right: AUC-ROC).
age human annotation data for turn-level satisfaction prediction, excluding approaches using human-engineered and skill-specific metrics as well as methods that only consider the quality of conversation from the language perspective.
Table 3 in Appendix B presents a qualitative comparison of the baseline supervised training and the self-supervised approach suggested in this paper. Here, to highlight the generalization and data-efficiency of each method, we limit the number of annotated samples to 1024 random samples from the training set of the  $\mathbb{D}_{sup}$  dataset. For this table, we provide sample sessions that are chosen with an emphasis on more difficult requests, unclear requests, or requests involving 3p skills.  $U$  and  $R$  indicate the targeted utterance and response, while  $U + x$  and  $R + x$  indicate the context utterance and responses appearing  $x$  turns after the targeted turn.
From the provided examples, it can be inferred that the self-supervised approach provides a deeper understanding of the user-agent interaction and is
able to generalize better even for infrequent 3p skills. It is consistent with the quantitative results presented in the paper.
This paper suggested a self-supervised objective to learn user-agent interactions leveraging large amounts of unlabeled data available. In addition to the standard fine-tuning approach, this paper presented a novel few-shot transfer learning method based on adjusting the RBCD block selection distribution to favor layer parameters with source and target gradients pointing in similar directions. According to the experiments using real-world data, the proposed approach not only requires significantly less number of annotations, but also generalizes better for unseen out-of-domain skills.
Figure 6: Comparison of the self-supervised pretraining and the proposed few-shot learning methods for the indomain test set using different number of training samples (left: AUC-PR, right: AUC-ROC).
Figure 7: Comparison of the self-supervised pretraining and the proposed few-shot learning methods for the out-of-domain test set using different number of training samples (left: AUC-PR, right: AUC-ROC).
Praveen Kumar Bodigutla, Lazaros Polymenakos, and Spyros Matsoukas. 2019. Multi-domain conversation quality evaluation via user satisfaction estimation. arXiv preprint arXiv:1911.08567.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
R Devon et al. 2020. Representation learning with video deep infomax. arXiv preprint arXiv:2007.13278.
Hongchao Fang and Pengtao Xie. 2020. Cert: Contrastive self-supervised learning for language understanding. arXiv preprint arXiv:2005.12766.
Steve Fox, Kuldeep Karnawat, Mark Mydland, Susan Dumais, and Thomas White. 2005. Evaluating implicit measures to improve web search. ACM Transactions on Information Systems (TOIS), 23(2):147-168.
Michael Gutmann and Aapo Hyvarinen. 2010. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 297-304.
Ahmed Hassan. 2012. A semi-supervised approach to modeling web search satisfaction. In Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval, pages 275-284.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. 2018. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations.
Kalervo Järvelin, Susan L Price, Lois ML Delcambre, and Marianne Lykke Nielsen. 2008. Discounted cumulated gain based evaluation of multiple-query in sessions. In European Conference on Information Retrieval, pages 4-15. Springer.
Jiepu Jiang, Ahmed Hassan Awadallah, Rosie Jones, Umut Ozertem, Imed Zitouni, Ranjitha Gurunath Kulkarni, and Omar Zia Khan. 2015. Automatic online evaluation of intelligent assistants. In
Proceedings of the 24th International Conference on World Wide Web, pages 506-516.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Junnan Li, Ziwei Xu, Wong Yongkang, Qi Zhao, and Mohan Kankanhalli. 2020. Gradmix: Multi-source transfer across domains and tasks. In The IEEE Winter Conference on Applications of Computer Vision, pages 3019-3027.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. arXiv preprint arXiv:1603.08023.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Zhaoyu Wang, Li Mian, Jing Zhang, and Jie Tang. 2020. Self-supervised learning: Generative or contrastive. arXiv, pages arXiv-2006.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
David Lopez-Paz and Marc'Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. In Advances in neural information processing systems, pages 6467-6476.
Yan Luo, Yongkang Wong, Mohan S Kankanhalli, and Qi Zhao. 2020.  $n$ -reference transfer learning for saliency prediction. arXiv preprint arXiv:2007.05104.
Yu Nesterov. 2012. Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM Journal on Optimization, 22(2):341-362.
Jekaterina Novikova, Ondrej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for nlg. arXiv preprint arXiv:1707.06875.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318.
Dookun Park, Hao Yuan, Dongmin Kim, Yinglei Zhang, Matsoukas Spyros, Young-Bum Kim, Ruhi Sarikaya, Edward Guo, Yuan Ling, Kevin Quinn, et al. 2020. Large-scale hybrid approach for predicting user satisfaction with conversational agents. arXiv preprint arXiv:2006.07113.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch.
Louisa Pragst, Stefan Ultes, and Wolfgang Minker. 2017. Recurrent neural network interaction quality estimation. In Dialogues with Social Robots, pages 381-393. Springer.
Niklas Rach, Wolfgang Minker, and Stefan Ultes. 2017. Interaction quality estimation using long short-term memories. In Proceedings of the 18th Annual SIG-dial Meeting on Discourse and Dialogue, pages 164-169.
Ashwin Ram, Rohit Prasad, Chandra Khatri, Anu Venkatesh, Raefer Gabriel, Qing Liu, Jeff Nunn, Behnam Hedayatnia, Ming Cheng, Ashish Nagar, et al. 2018. Conversational ai: The science behind the alexa prize. arXiv preprint arXiv:1801.03604.
Trieu H Trinh, Minh-Thang Luong, and Quoc V Le. 2019. Self-supervised pretraining for image embedding. arXiv preprint arXiv:1906.02940.
Stephen J Wright. 2015. Coordinate descent algorithms. Mathematical Programming, 151(1):3-34.
Tiansheng Yao, Xinyang Yi, Derek Zhiyuan Cheng, Felix Yu, Aditya Menon, Lichan Hong, Ed H Chi, Steve Tjoa, Evan Ettinger, et al. 2020. Self-supervised learning for deep models in recommendations. arXiv preprint arXiv:2007.12865.
A Annotation Protocol
In the following, we provide a summary of main points considered to produce the annotations used in this paper<sup>3</sup>:
- Human annotators were trained to annotate samples i.e., we do not use domain specific metrics or other automated success measures as annotation.
- It was made clear to the annotators that the task is turn-level user satisfaction, and not the overall satisfaction over session. Also, instructions were provided on how to handle ASR errors, repeated requests, multiple users in one utterance, and many other special cases.
- The annotators were provided the targeted turn as well as a few context turns. This helped them to better understand the actual user intention and judge accordingly.
- They were asked to rate the system's response quality in terms of the user satisfaction on the scale of 1 to 5, from terrible to excellent. See Table 2 for score categories and an example of each category.
- To ensure the quality of annotations, each sample was annotated multiple times by different annotators.
Table 2: Examples of the annotation scores.
Score	Example
1(Terrible)	U: Is bacon really bad for your health?R: Sorry, I'm not sure.
2(Bad)	U: How's the humidity?R: Sorry, I don't have daily forecast information.
3(Okay)	U: Play a song for the baby.R: Skill [SKILL NAME] can help you with that.Did you want to enable it?
4(Good)	U: What's the temperature going to be tomorrow?R: Tomorrow in London, you can look for cloudy weather with a high of 65 and low of 52 degrees.
5(Excellent)	U: Cancel alarm.R: 6:35 PM alarm cancelled.
In our analysis, we consider all samples having a score of 3 or better as SAT, and DSAT otherwise. Also, in our data pipeline, we considered different annotations of the same utterance as different samples. However, care was taken in the data split
process to ensure there is no train data contamination in our validation and test sets.
B Qualitative Results
Table 3 presents a qualitative comparison of the baseline supervised training and the self-supervised approach suggested in this paper. Here, to highlight the generalization and data-efficiency of each method, we limit the number of annotated samples to 1024 random samples from the training set of the  $\mathbb{D}_{sup}$  dataset. For this table, we provide sample sessions that are chosen with an emphasis on more difficult requests, unclear requests, or requests involving 3p skills.  $U$  and  $R$  indicate the targeted utterance and response, while  $U + x$  and  $R + x$  indicate the context utterance and responses appearing  $x$  turns after the targeted turn.
From these examples, it can be inferred that the self-supervised approach provides a deeper understanding of the user-agent interaction and is able to generalize better even for infrequent 3p skills. It is consistent with the quantitative results presented in the paper.
3Certain details were omitted to comply with confidentiality requirements.
Table 3: Qualitative comparison of predictions for different methods on a set of selected test samples.
Session	Label	Prediction	Comments
\( Sup^a \)	\( Self^b \)
U-1: what what high school bas-ketball team do you like	R-1: I don't have a favorite yet.
U: what is your favorite m. l. b. team	R: My home team is [NAME], and you've gotta root root root for the home team	SAT	DSAT	SAT	Asking for favorite,agent using advanced language and slang
U+1: what's your favorite movie	R+1: The Empire Strikes Back" is my all-time favorite. It's impressive.
U-1: is it going to rain tomorrow	R-1: No rain is expected in New York tomorrow.
U: is it raining now in new york	R: No rain is expected in New York tomorrow	DSAT	SAT	DSAT	Asking for weather, re- wording and repeating the request
U+1: is it raining today	R+1: It's raining right now. You can expect about 0.05 inches.
U+2: set an alarm for five p.m.	R+1: Alarm set for 5:00 PM.
U: close the garage	R: Ok, Here's Garadget - Smart Garage The door 'garage' was already closed	SAT	DSAT	SAT	Device interaction, us- ing a less common 3p skill
U: teach me	R: Me's definition is first-person singular pronoun, indicating the speaker, used as the object of a verb or preposition	DSAT	SAT	DSAT	Knowledge question,vague request agent fails to understand
U+1: pause	R+1: [EMPTY]
U-2: what is in the middle of the milky way galaxy	R-2: A storm of exploding stars circling a black hole.
U-1: name all the countries in the world	R-1: There are 193 UN- recognized countries, including China, India, the United States, and Indonesia.
U: read us a really long list	R: Resuming your most recently purchased book, Everything I Know About Lean I Learned in First Grade	DSAT	SAT	DSAT	Knowledge question,agent fails to understand user's intention
U+1: cancel	R+1: [EMPTY]
U+2: name all the countries in europe	R+2: I know about fifty countries in Europe. They are the United Kingdom, Turkey, [48 MORE NAMES]
U-2: call	R-2: Which Phone Number, Contact or Device do you want to call?
U-1: p.m	R-1: I heard you ask for p.m., but I can't find that name in your list of contacts or devices.
U: call liam	R: I can't call Liam's phone number. I can call contacts who have signed up for [AGENT] calling, and I can call phone numbers in the United States, Canada, and Mexico	SAT	DSAT	SAT	Call request, failed due to system limitation but user understands
U+1: thank you [AGENT]	R+1: You bet. And good evening.
Supervised training baseline
${}^{b}$  Self-supervised contrastive learning method