SlowGuess/NAACL_2021_OCR huggingface dataset:
Mohammad Kachuee*  
UCLA Computer Science  
Los Angeles, CA  
mkachuee@cs.ucla.edu
Hao Yuan  
Amazon Alexa AI  
Seattle, WA
Young-Bum Kim  
Amazon Alexa AI  
Seattle, WA
Sungjin Lee  
Amazon Alexa AI  
Seattle, WA
{yuanha,youngbum,sungjinl}  $@$  amazon.com
Turn-level user satisfaction is one of the most important performance metrics for conversational agents. It can be used to monitor the agent's performance and provide insights about defective user experiences. While end-to-end deep learning has shown promising results, having access to a large number of reliable annotated samples required by these methods remains challenging. In a large-scale conversational system, there is a growing number of newly developed skills, making the traditional data collection, annotation, and modeling process impractical due to the required annotation costs and the turnaround times. In this paper, we suggest a self-supervised contrastive learning approach that leverages the pool of unlabeled data to learn user-agent interactions. We show that the pre-trained models using the self-supervised objective are transferable to the user satisfaction prediction. In addition, we propose a novel few-shot transfer learning approach that ensures better transferability for very small sample sizes. The suggested few-shot method does not require any inner loop optimization process and is scalable to very large datasets and complex models. Based on our experiments using real data from a large-scale commercial system, the suggested approach is able to significantly reduce the required number of annotations, while improving the generalization on unseen skills.
Nowadays automated conversational agents such as Alexa, Siri, Google Assistant, Cortana, etc. are widespread and play an important role in many different aspects of our lives. Their applications vary from storytelling and education for children to assisting the elderly and disabled with their daily activities. Any successful conversational agent should be able to communicate in different languages and accents, understand the conversation
context, analyze the query paraphrases, and route the requests to various skills available for handling the user's request (Ram et al., 2018).
In such a large-scale system with many components, it is crucial to understand if the human user is satisfied with the automated agent's response and actions. In other words, it is desirable to know if the agent is communicating properly and providing the service that is expected by the user. In the literature, it is referred to as targeted turn-level satisfaction as we are only interested in the user's satisfaction for a certain conversation turn given the context of the conversation, and not the overall satisfaction for the whole conversation (Park et al., 2020). Perhaps the most basic use of a user satisfaction model would be to monitor the performance of an agent and to detect defects as a first step to fix issues and improve the system. Anticipating user dissatisfaction for a certain turn in a conversation, an agent would be able to ask the user for repeating the request or providing more information, improving the final experience. Also, a powerful user satisfaction model can be used as a ranking or scoring measure to select the most satisfying response among a set of candidates and hence guiding the conversation.
The problem of user satisfaction modeling has recently attracted significant research attention (Jiang et al., 2015; Bodigutla et al., 2019; Park et al., 2020; Pragst et al., 2017; Rach et al., 2017). These methods either rely on annotated datasets providing ground-truth labels to train and evaluate (Bodigutla et al., 2019) or rely on ad hoc or human-engineered metrics that do not necessarily model the true user satisfaction (Jiang et al., 2015). Access to reliable annotations to be used in building satisfaction models has been very challenging partly due to the fact that a large-scale conversation system supports many different devices as well as voice, language, and application components, providing access to a wide variety of skills. The traditional approach of collecting samples from the live system traffic and
*Work done as an intern at Amazon Alexa AI.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4053-4064. June 6-11, 2021. ©2021 Association for Computational Linguistics
tasking human annotators to label samples would not be scalable due to the cost of annotations as well as the turn-around time required to collect and annotate data for a new skill or feature. Note that onboarding new skills in a timely manner is a crucial to ensure active skill developer engagement.
To address this problem, we propose a novel training objective and transfer learning scheme that significantly improves not only the data efficiency but also the model generalization to unseen skills. In summary, we make the following contributions:
- We propose a contrastive self-supervised training objective that can leverage virtually any unlabeled conversation data to learn user-agent interactions.
- We show that the proposed method can be used to pre-train state-of-the-art deep language models and the acquired knowledge is transferable to the user satisfaction prediction.
- We suggest a novel and scalable few-shot transfer learning approach that is able to improve the label efficiency even further in the case of few-shot transfer learning.
- We conduct extensive experiments using data from a large-scale commercial conversational system, demonstrating significant improvements to label efficiency and generalization.
User Satisfaction in Conversational Systems
The traditional approach to evaluating a conversational system is to evaluate different functionalities or skills individually. For instance, for a knowledge question answering or web search skill, one can use response quality metrics commonly used to evaluate search system and ranking systems such as nDCG (Jarvelin et al., 2008; Hassan, 2012; Fox et al., 2005). While these methods provide justifiable measures for certain skills, they are not extendable to a large number of skills, especially for skills without a set of proper hand-engineered features and metrics, or newly developed third-party skills (Bodigutla et al., 2019).
Another, more general, line of research is to evaluate the performance of a conversation system from the language point of view. Here, the objective is to measure how natural, syntactically and semantically, an automated agent is able to
interact with a human user. For instance, using generic metrics such as BLEU (Papineni et al., 2002) or ROUGE (Lin, 2004) one can measure how the agent's responses are consistent with a set of provided ground-truth answers. However, these approaches not only suffer from shortcomings such as inconsistency with the human understanding (Liu et al., 2016; Novikova et al., 2017) but also are not practical for a real-world conversation system due to their dependence on ground-truth responses.
A more recent approach is to use human annotations specifically tailored for the user satisfaction task as a source of supervision to train end-to-end prediction models (Bodigutla et al., 2019). Jiang et al. (2015) suggested training individual models for 6 general skills and devised engineered features to link user actions to the user satisfaction for each studied skill. Park et al. (2020) proposed a hybrid method to learn from human annotation and user feedback data that is scalable and able to model user satisfaction across a large number of skills.
Gutmann and Hyvarinen (2010) was the first study to propose the idea of noise-contrastive learning in the context of a capturing a distribution using an objective function to distinguish samples of the target distribution from samples of an artificially generated noise distribution. Contrastive predictive coding (CPC) (Oord et al., 2018) suggested the idea of using an NCE objective to train an auto-regressive sequence representation model. Deep InfoMax (Hjelm et al., 2018) used self-supervised contrastive learning in an architecture where a discriminator is trained to distinguish between representations of the same image (positive samples) or representations of different images (negative samples). While many different variations of contrastive methods have been suggested, the main idea remains the same: defining a self-supervised objective to distinguish between the hidden representations of samples from the original distribution and samples from a noise distribution (Trinh et al., 2019; Devon et al., 2020; Yao et al., 2020).
Few-shot Transfer Learning
Few-shot transfer learning is a very active and broad subject of research. We limit the scope of our study to methods in which a form of gradient supervision is provided by a target task to ensure the efficient transferability of representations trained on a source task. Lopez-Paz and Ranzato (2017)
suggested the idea of joint multi-task training and using the cosine similarity of the concatenated network gradients from the source and target tasks. For gradients with negative cosine distance, they project the source gradients to a more aligned direction by solving a quadratic programming problem. Luo et al. (2020) continued that line and suggested a method in the context of few-shot transfer learning, showing that using even a few samples from the target task can significantly improve the transferability of the trained models. Li et al. (2020) presented a similar idea but suggested adjusting learning rates for each layer to improve the cosine similarity of different tasks. While these methods show promising results, they only measure the similarity between concatenated gradient vectors consisting of all network parameters which is a very rough measure of alignment. Also, they require solving for a quadratic or iterative optimization problem as an inner loop in the training procedure that can be computationally expensive and often prohibitive for large-scale problems.
3 User Satisfaction Modeling
In this paper, we consider the conversational interaction between a human user and an automated agent. Each interaction consists of a set of turns in which the user provides an utterance and the agent provides appropriate responses. A set of turns that are happening within a certain time window are grouped as a conversation session. Formally, we can represent a session as a set of turns:
$$
S _ {i} = \left\{\left(U _ {i} ^ {t = 0}, R _ {i} ^ {t = 0}\right), \dots , \left(U _ {i} ^ {t = T}, R _ {i} ^ {t = T}\right) \right\} \tag {1}
$$
Here,  $S_{i}$  represents session  $i$  consisting of a set of turns as tuples of utterance and responses,  $(U_{i}^{t}, R_{i}^{t})$ , for the first turn  $t = 0$  to the last turn  $t = T$  in that session.
In the context of turn-level user satisfaction modeling, we are interested in the classification of a certain targeted turn within a session as either satisfying (SAT) or dissatisfying (DSAT). Note that the satisfaction here is defined based on the agent's response given a certain utterance and the context (i.e., other session turns). We use the notation  $Y_{i}^{t^{*}} \in \{\mathrm{SAT},\mathrm{DSAT}\}$  to indicate the user satisfaction for the targeted turn  $t = t^{*}$  of session  $i$ . See Figure 2 for examples of SAT/DSAT interactions.
Session	Label
U0: play clocks
R0: Playing clocks by Coldplay.	SAT
[User listening for 1 minute]	用户的listening)
U0: play clock
R0: Sorry, I cannot find the song.	DSAT
U1: play clocks	(repeated request)
R1: Playing clocks by Coldplay.
[User listening for 1 minute]
U0: turn on the tv
R0: Did you mean Living Room TV?	SAT
U1: yes
R1: [EMPTY]	(no further request)
U0: call chris
R0: Calling chris.	DSAT
U1: stop	(ASR error, barge-in)
R1: [EMPTY]
U1: call christina
R1: Calling Christina.
[Phone call for 2 minutes]
Figure 2: A few examples of SAT and DSAT turns to illustrate the importance of the conversation context.
In this study, we use real-world data from Alexa, a large-scale commercial conversational agent. Specifically, we use a dataset of about 891,000 real-world conversation sessions in which a certain turn within each session is annotated by a human annotator as SAT or DSAT. Human annotators had access to the session context and followed a standard labeling protocol (further information is provided in Appendix A). As a preprocessing step, we limited turns within each session to a window of five turns: at most two turns before the targeted turn, the targeted turn, and at most two turns after the targeted turn. This labeled dataset is denoted as  $\mathbb{D}_{sup}$ .
In addition to  $\mathbb{D}_{sup}$ , we also use a large pool of real-world session data without any annotation or label. This dataset is about twice the size of  $\mathbb{D}_{sup}$  but as we are not limited to targeted turns, we keep all session turns and decide context windows based on a randomized data augmentation step. The resulting effective sample size is significantly larger than  $\mathbb{D}_{sup}$ . We denote this unlabeled dataset as  $\mathbb{D}_{unsup}$ . As both datasets were sampled from real traffic, we ensured that there is no overlap between  $\mathbb{D}_{unsup}$  and the evaluation splits of  $\mathbb{D}_{sup}$ .
The conversations cover a wide variety of internally developed (1p) and third-party (3p) developer skills. Due to the imbalanced traffic, in our
<sup>1</sup>Due to confidentiality concerns, we are not able to disclose the exact annotation protocols and data specifications.
Figure 1: Overview of the suggested network architecture. In our architecture, BERT encoder with average pool at the last layer is used as the LM. We consider a context window of at most  $2\mathrm{T} + 1$  turns. Heads are simple MLPs classifiers with one hidden layer.
Property	Size
Total number of samples	≈ 891,000
Total number of 1p skills	> 20
Total number of 3p skills	> 1500
Ratio of SAT to DSAT samples	> 20
Table 1: Dataset statistics for  ${\mathbb{D}}_{sup}$
datasets, there is a huge variation between the number of samples for different skills. For instance, 1p skills such as music or weather have hundreds of thousands of samples while many 3p skills only have less than 10 samples throughout our datasets. To properly evaluate the performance of our predictors on such imbalanced data, we proposed a novel approach to split the data and to evaluate. We build two test sets: a test set measuring in-domain performance and another test set to measure the out-of-domain generalization. The in-domain test set consists of samples from skills that the train set covers. The out-of-domain test set measures the performance on skills that are not covered by the train set. Ideally, we would like to observe good classification performance in both test splits, indicating the ability of our models to learn and model the current major traffic and to generalize to less frequent or future traffic. Based on this, we split  $\mathbb{D}_{sup}$  to  $70\%$  train,  $15\%$  validation, and the rest for the test (about  $1/5$  of test samples are out-of-domain and  $4/5$  are in-domain). The in-domain and out-of-domain test sets consist of 17 and 275 skills, respectively. The  $\mathbb{D}_{unsup}$  is randomly split to  $80\%$  train and the rest for validation, regardless of skills. Table 1 presents a summary of dataset statistics for  $\mathbb{D}_{sup}$ .
Figure 1 shows a high-level drawing of the network architecture used in our experiments. It consists of
a language model (LM) that encodes utterance and response pairs to vector representations. Here, we consider up to T turns before and after the targeted turn. To further summarize the list of the previous or next turns, we use GRU layers (Chung et al., 2014). Then, an average pool is used to produce a representation vector,  $z$ , for each session. Note that before the pooling, simple non-linear MLPs are used to transform each partial representation. Finally,  $z$  is used as an input to a set of different head networks, responsible for making predictions for different objectives.
Regarding the LM, we use the standard BERT encoder (Devlin et al., 2018) architecture pre-trained as suggested by Liu et al. (2019). To make a fixed-length representation of the utterance response pairs i.e. turn semantics, we use an average pool at the last encoder layer of the BERT token representations. We also tried other approaches such as using the classification token instead of pooling, but based on our initial results simple pooling performed consistently better.
We share our BERT-based LM parameters across the network to encode the session turns. However, we train separate GRU networks to summarize the previous and next turns. The output dimension of the LM is equal to 768, the size of the standard BERT hidden layer. The hidden layer and output size of our GRUs are 256, and we use 2-layer bidirectional GRUs. Each head is a simple MLP with a single hidden layer of size 256 followed by a ReLU nonlinearity. The final network consists of about 117.7 million parameters from which about 110 million is related to BERT and the rest is for GRUs, heads, etc.
3.4 Supervised Learning Baseline
As a baseline approach, we use the network defined in Section 3.3 with a binary classification head to distinguish SAT and DSAT samples. Here, we
use labels provided by  $\mathbb{D}_{sup}$  and a binary cross-entropy (BCE) loss function. An Adam optimizer (Kingma and Ba, 2014) with a batch size of 512 is used to train the network for 10 epochs. The base learning rate for all non-BERT layers is set to  $10^{-3}$ , while for BERT layers, we use a smaller learning rate of  $5 \times 10^{-5}$ . The learning rates are decayed with a factor 5 twice at  $60\%$  and  $80\%$  of total iterations. Unless indicated otherwise, we use a similar training setup for other experiments suggested in this paper.
4 Self-Supervised Contrastive Learning
4.1 Self-Supervised Objective
We define a self-supervised objective in which the model is tasked to distinguish real sessions from unreal (or noisy) sessions. Any unlabeled dataset, such as  $\mathbb{D}_{unsup}$  can be used to sample real sessions. To generate unreal textual information, different approaches have been suggested in the literature such as back-translation (Fang and Xie, 2020), generative modeling (Liu et al., 2020), or even random word substitutions.
In this work, we leverage the multi-turn and structured nature of sessions to generate noise samples by simply shuffling the targeted utterances/responses within each training batch (see Figure 3 for an example). Intuitively, the noise samples are sessions in which the targeted utterance or response does not belong to the rest of the session. Therefore, the model has to capture the joint distribution of the context and targeted turns. Algorithm 1 shows an overview of the sample generation and training process for the proposed contrastive objective.
Figure 3: A toy example demonstrating the generation of unreal samples from a batch of two real samples. Session context is omitted for brevity.
4.2 Contrastive Pretraining
The objective introduced in Section 4.1 is not directly applicable to be used as a user satisfaction model. One approach to leverage the pool of unsupervised data is to pre-train the model on unlabeled data using the self-supervised objective, and then attach a classifier head and finetune the network to distinguish SAT and DSAT samples. In our implementation, we pre-train using the self-supervised objective on  $\mathbb{D}_{unsup}$  for 10 epochs, then train a classifier head on  $\mathbb{D}_{sup}$  for another 10 epochs; adjusting the learning rates for the network body to  $\times 0.1$  of the base learning rates (see Section 3.4 for more information on the learning rate setup).
4.3 Few-Shot Learning
In the pretraining approach, we solely relied on the loose semantic relationship between the self-supervised and the user satisfaction modeling tasks. However, it is desirable to have a representation that is not only solving the self-supervised task but is also useful for the final objective. In other words, we have a source task  $(S)$  which we have a large number of training samples and a target task  $(T)$  with a limited number of samples that is our main interest. The idea is to use information from the target task during the source training such that the trained model is most compatible with the target.
Let us assume we have datasets  $\mathbb{D}_S$  and  $\mathbb{D}_T$  corresponding to the source  $(S)$  and target  $(T)$  tasks as well as inference functions for each task:  $f_{S}(.|\theta ,\omega_{S})$  and  $f_{T}(.|\theta ,\omega_{T})$ . In this notation,  $\theta$  rep
resents shared network parameters (i.e., the body in our architecture) and  $\omega$  represents task-specific parameters (i.e., a head in our architecture). Formally, when optimizing for task  $S$ , we are interested in:
$$
\underset {\theta , \omega_ {S}} {\arg \min } \mathbb {E} _ {\boldsymbol {x}, y \sim \mathbb {D} _ {S}} \left[ L _ {S} \left(f _ {S} (\boldsymbol {x} \mid \theta , \omega_ {S}), y\right) \right], \tag {2}
$$
where  $L_{S}$  is the loss function for the source task. A simple gradient descent step to solve this problem can be written as:
$$
\theta^ {t + 1} \longleftarrow \theta^ {t} - \eta \nabla_ {\theta} \mathbb {E} \left(L _ {S} \left(f _ {S} \left(\boldsymbol {x} \mid \theta^ {t}, \omega_ {S} ^ {t}\right), y\right)\right),
$$
$$
\omega_ {S} ^ {t + 1} \leftarrow \omega_ {S} ^ {t} - \eta \nabla_ {\omega_ {S}} \mathbb {E} \left(L _ {S} \left(f _ {S} (\boldsymbol {x} \mid \theta^ {t}, \omega_ {S} ^ {t}), y\right)\right). \tag {3}
$$
However, we are interested in optimization steps that do not increase the loss value for task  $T$ :
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x}, y \sim \mathbb {D} _ {T}} \left[ L _ {T} \left(f _ {T} \left(\boldsymbol {x} \mid \theta^ {t + 1}, \omega_ {T} ^ {t + 1}\right), y\right) \right] \leq \tag {4} \\ \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y} \sim \mathbb {D} _ {T}} \left[ L _ {T} \left(f _ {T} (\boldsymbol {x} \mid \theta^ {t}, \omega_ {T} ^ {t}), \boldsymbol {y}\right) \right]. \\ \end{array}
$$
Considering (4) as an optimization constraint can potentially halt the optimization because improvements to the source objective do not directly translate to improvements to the target task. In other words, the constraint above may not be always directly satisfiable using gradient steps in the source domain.
To overcome this issue, instead of using gradient descent, we define the problem as a Randomize Block Coordinate Descent (RBCD) (Nesterov, 2012; Wright, 2015) optimization. At each RBCD iteration, only a subset of model parameters, i.e. a block noted as  $\pmb{b}$ , is sampled from a distribution  $\mathcal{B}$  and used for the gradient descent update:
$$
\begin{array}{l} \boldsymbol {b} \sim \mathcal {B} \quad , \\ \theta_ {\boldsymbol {b}} ^ {t + 1} \leftarrow \theta_ {\boldsymbol {b}} ^ {t} - \eta \nabla_ {\boldsymbol {b}} \mathbb {E} \left(L _ {S} \left(f _ {S} \left(\boldsymbol {x} \mid \theta^ {t}, \omega_ {S} ^ {t}\right), y\right)\right). \tag {5} \\ \end{array}
$$
Note that we only use the RBCD optimization for the network body parameters  $(\theta)$ , while the head parameters  $(\omega_{S}$  and  $\omega_{T})$  are optimized using a regular gradient descent optimization.
In this work, we propose the idea of adjusting the block selection distribution,  $\mathcal{B}$ , such that parameters having more aligned source and target gradients have more chance of being selected:
$$
\mathcal {B}: P r (i \in \boldsymbol {b}) \propto \left\langle \nabla_ {i, S} L _ {S}, \nabla_ {i, T} L _ {T} \right\rangle , \tag {6}
$$
where the inputs to  $L_{S}$  and  $L_{T}$  are omitted for brevity. Intuitively, (6) is used to discourage parameter updates that are not aligned with the T task
which can be viewed as a soft method to enforce the constraint in (4). Here, there are multiple options to define the granularity of the block selection such as layer-wise, neuron-wise, or element-wise. Based on our initial experiments, we found that defining the block elements to be layer-wise results in the best performance.