SlowGuess/NAACL_2021_OCR huggingface dataset:
Shikib Mehri\* Mihail Eric
$\spadesuit$ Language Technologies Institute, Carnegie Mellon University, Amazon Alexa AI
amehri@cs.cmu.edu, mihaeric@amazon.com
A key challenge of dialog systems research is to effectively and efficiently adapt to new domains. A scalable paradigm for adaptation necessitates the development of generalizable models that perform well in few-shot settings. In this paper, we focus on the intent classification problem which aims to identify user intents given utterances addressed to the dialog system. We propose two approaches for improving the generalizability of utterance classification models: (1) observers and (2) example-driven training. Prior work has shown that BERT-like models tend to attribute a significant amount of attention to the [CLS] token, which we hypothesize results in diluted representations. Observers are tokens that are not attended to, and are an alternative to the [CLS] token as a semantic representation of utterances. Example-driven training learns to classify utterances by comparing to examples, thereby using the underlying encoder as a sentence similarity model. These methods are complementary; improving the representation through observers allows the example-driven model to better measure sentence similarities. When combined, the proposed methods attain state-of-the-art results on three intent prediction datasets (BANKING77, CLINC150, HWU64) in both the full data and few-shot (10 examples per intent) settings. Furthermore, we demonstrate that the proposed approach can transfer to new intents and across datasets without any additional training.
Task-oriented dialog systems aim to satisfy a user goal in the context of a specific task such as booking flights (Hemphill et al., 1990), providing transit information (Raux et al., 2005), or acting as a tour guide (Budzianowski et al., 2018). Task-oriented dialog systems must first understand the user's goal
by extracting meaning from a natural language utterance. This problem is known as intent prediction and is a vital component of task-oriented dialog systems (Hemphill et al., 1990; Coucke et al., 2018). Given the vast space of potential domains, a key challenge of dialog systems research is to effectively and efficiently adapt to new domains (Rastogi et al., 2019). Rather than adapting to new domains by relying on large amounts of domain-specific data, a scalable paradigm for adaptation necessitates the development of generalizable models that perform well in few-shot settings (Casanueva et al., 2020; Mehri et al., 2020).
The task of intent prediction can be characterized as a two step process: (1) representation (mapping a natural language utterance to a semantically meaningful representation) and (2) prediction (inferring an intent given a latent representation). These two steps are complementary and interdependent, thereby necessitating that they be jointly improved. Therefore, to enhance the domain adaptation abilities of intent classification systems we propose to (1) improve the representation step through observers and (2) improve the prediction step through example-driven training.
While BERT (Devlin et al., 2018) is a strong model for natural language understanding tasks (Wang et al., 2018), prior work has found a significant amount of BERT's attention is attributed to the [CLS] and [SEP] tokens, though these special tokens do not attribute much attention to the words of the input until the last layer (Clark et al., 2019; Kovaleva et al., 2019). Motivated by the concern that attending to these tokens is causing a dilution of representations, we introduce observers. Rather than using the latent representation of the [CLS] token, we instead propose to have tokens which attend to the words of the input but are not attended to. In this manner, we disentangle BERT's attention with the objective of improving the semantic content captured by the utterance representations.
Work done while Shikib was at Amazon
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2979-2992 June 6-11, 2021. ©2021 Association for Computational Linguistics
A universal goal of language encoders is that inputs with similar semantic meanings have similar latent representations (Devlin et al., 2018). To maintain consistency with this goal, we introduce example-driven training wherein an utterance is classified by measuring similarity to a set of examples corresponding to each intent class. While standard approaches implicitly capture the latent space to intent class mapping in the learned weights (i.e., through a classification layer), example-driven training makes the prediction step an explicit non-parametric process that reasons over a set of examples. By maintaining consistency with the universal goal of language encoders and explicitly reasoning over the examples, we demonstrate improved generalizability to unseen intents and domains.
By incorporating both observers and example-driven training on top of the CONVBERT model (Mehri et al., 2020), we attain state-of-the-art results on three intent prediction datasets: BANKING77 (Casanueva et al., 2020), CLINC150 (Larson et al., 2019), and HWU64 (Liu et al., 2019) in both full data and few-shot settings. To measure the generalizability of our proposed models, we carry out experiments evaluating their ability to transfer to new intents and across datasets. By simply modifying the set of examples during evaluation and without any additional training, our example-driven approach attains strong results on both transfer to unseen intents and across datasets. This speaks to the generalizability of the approach. Further, to demonstrate that observers mitigate the problem of diluted representations, we carry out probing experiments and show that the representations produced by observers capture more semantic information than the [CLS] token.
The contributions of this paper are as follows: (1) we introduce observers in order to avoid the potential dilution of BERT's representations, by disentangling the attention, (2) we introduce example-driven training which explicitly reasons over a set of examples to infer the intent, (3) by combining our proposed approaches, we attain state-of-the-art results across three datasets on both full data and few-shot settings, and (4) we carry out experiments demonstrating that our proposed approach is able to effectively transfer to unseen intents and across datasets without any additional training.
In this section, we describe several methods for the task of intent prediction. We begin by describing two baseline models: a standard BERT classifier (Devlin et al., 2018) and CONVBERT with task-adaptive masked language modelling (Mehri et al., 2020). The proposed model extends the CONVBERT model of Mehri et al. (2020) through observers and example-driven training. Given the aforementioned two step characterization of intent prediction, observers aim to improve the representation step while example-driven training improves the prediction step.
2.1 BERT Baseline
Across many tasks in NLP, large-scale pre-training has resulted in significant performance gains (Wang et al., 2018; Devlin et al., 2018; Radford et al., 2018). To leverage the generalized language understanding capabilities of BERT for the task of intent prediction, we follow the standard fine-tuning paradigm. Specifically, we take an off-the-shelf BERT-base model and perform end-to-end supervised fine-tuning on the task of intent prediction.
2.2 Conversational BERT with Task-Adaptive MLM
Despite the strong language understanding capabilities exhibited by pre-trained models, modelling dialog poses challenges due to its intrinsically goal-driven, linguistically diverse, and often informal/noisy nature. To this end, recent work has proposed pre-training on open-domain conversational data (Henderson et al., 2019; Zhang et al., 2019b). Furthermore, task-adaptive pre-training wherein a model is trained in a self-supervised manner on a dataset prior to fine-tuning on the same dataset, has been shown to help with domain adaptation (Mehri et al., 2019; Gururangan et al., 2020; Mehri et al., 2020). Our models extend the CONVBERT model of Mehri et al. (2020) which (1) pre-trained the BERT-base model on a large open-domain dialog corpus and (2) performed task-adaptive masked language modelling (MLM) as a mechanism for adapting to specific datasets.
2.3 Observers
The pooled representation of BERT-based models is computed using the [CLS] token. Analysis of BERT's attention patterns has demonstrated that a significant amount of attention is attributed to
https://github.com/alexa/DialoGLUE/
Figure 1: A visualization of the observers. The observer node attends to other tokens at each layer, however it is never attended to. While this figure only depicts one observer – we include multiple observers and average their final representation.
the [CLS] and [SEP] tokens (Clark et al., 2019; Kovaleva et al., 2019). It is often the case that over half of the total attention is to these tokens (Clark et al., 2019). Furthermore, the [CLS] token primarily attends to itself and [SEP] until the final layer (Kovaleva et al., 2019). It is possible that attending to these special BERT tokens, in combination with the residual connections of the BERT attention heads, is equivalent to a no-op operation. However, it is nonetheless a concern that this behavior of attending to tokens with no inherent meaning (since [CLS] does not really attend to other words until the final layer) results in the latent utterance level representations being diluted.
We posit that a contributing factor of this behavior is the entangled nature of BERT's attention: i.e., the fact that the [CLS] token attends to words of the input and is attended to by the words of the input. This entangled behavior may inadvertently cause the word representations to attend to [CLS] in order to better resemble its representation and therefore make it more likely that the [CLS] token attends to the word representations. In an effort to mitigate this problem and ensure the representation contains more of the semantic meaning of the utterance, we introduce an extension to traditional BERT fine-tuning called observers.
Observers, pictured in Figure 1, attend to the tokens of the input utterance at every layer of the BERT-based model however they are never attended to. The representation of the observers in
the last layer is then used as the final utterance level representation. In this manner, we aim to disentangle the relationship between the representation of each word in the input and the final utterance level representation. By removing this bi-directional relationship, we hope to avoid the risk of diluting the representations (by inadvertently forcing them to attend to a meaningless [CLS] token) and therefore capture more semantic information in the final utterance level representation. Throughout our experiments we use 20 observer tokens (which are differentiated only by their position embeddings) and average their final representations. The positions of the observer tokens is consistent across all utterances (last 20 tokens in the padded sequence). Specifically, the concept of observers modifies  $\mathcal{F}$  in Equations 1 and 2. While we maintain the BERT-based model architecture, we instead produce the utterance level representation by averaging the representations of the observer tokens and using that for classification rather than the [CLS] token.
2.4 Example-Driven Training
A universal goal of language encoders is for inputs with similar semantic meanings to have similar latent representations. BERT (Devlin et al., 2018) has been shown to effectively identify similar sentences (Reimers and Gurevych, 2019) even without additional fine-tuning (Zhang et al., 2019a). Through example-driven training, we aim to reformulate the task of intent prediction to be more consistent with
2 Compute similarity to the examples
3 Map the similarity scores to the intents
Figure 2: A visualization of the three step process of computing a probability distribution over the set of intents in our example-driven formulation.
this universal goal of language encoders.
Using a BERT-like encoder, we train an intent classification model to (1) measure the similarity of an utterance to a set of examples and (2) infer the intent of the utterance based on the similarity to the examples corresponding to each intent. Rather than implicitly capturing the latent space to intent class mapping in our learned weights (i.e., through a classification layer), we make this mapping an explicit non-parametric process that reasons over a set of examples. Our formulation, similar to metric-based meta learning (Koch et al., 2015), only performs gradient updates for the language encoder, which is trained for the task of sentence similarity. Through this example-formulation, we hypothesize that the model will better generalize in few-shot scenarios, as well as to rare intents.
We are given (1) a language encoder  $\mathcal{F}$  that encodes an utterance to produce a latent representation, (2) a natural language utterance  $utt$ , and (3) a set of  $n$  examples  $\{(x_1,y_1),\ldots ,(x_n,y_n)\}$  where  $x_{1,\dots,n}$  are utterances and  $y_{1,\dots,n}$  are their corresponding intent labels. With  $\mathcal{F}$  being a BERT-like model, the following equations describe example-driven intent classification:
$$
\boldsymbol {u} = \mathcal {F} (u t t) \tag {1}
$$
$$
\boldsymbol {X} _ {i} = \mathcal {F} (\mathrm {x} _ {i}) \tag {2}
$$
$$
\boldsymbol {\alpha} = \operatorname {s o f t m a x} \left(\boldsymbol {u} ^ {T} \cdot \boldsymbol {X}\right) \tag {3}
$$
$$
P (c) = \sum_ {i: y _ {i} = c} \alpha_ {i} \tag {4}
$$
The equations above describe a non-parametric
process for intent prediction. Instead, through the example-driven formulation (visualized in Figure 2), the underlying language encoder (e.g., BERT) is being trained for the task of sentence similarity. A universal goal of language encoders is that inputs with similar semantic meaning should have similar latent representations. By formulating intent prediction as a sentence similarity task, we are adapting BERT-based encoders in a way that is consistent with this universal goal. We hypothesize that in contrast to the baseline models, this formulation facilitates generalizability and has the potential to better transfer to new intents and domains.
At training time, we populate the set of examples in a two step process: (i) for each intent class that exists in the training batch, we sample one different utterance of the same intent class from the training set and (ii) we randomly sample utterances from the training set until we have a set of examples that is double the size of the training batch size (128 example utterances). During inference, our example set is comprised of all the utterances in the training data.
We evaluate our methods on three intent prediction datasets: BANKING77 (Casanueva et al., 2020), CLINC150 (Larson et al., 2019), and HWU64 (Liu et al., 2019). These datasets span several domains and consist of many different intents, making them more challenging and more reflective of commercial settings than commonly used intent predic
tion datasets like SNIPs (Coucke et al., 2018). BANKING77 contains 13,083 utterances related to banking with 77 different fine-grained intents. CLINC150 contains 23,700 utterances spanning 10 domains (e.g., travel, kitchen/dining, utility, small talk, etc.) and 150 different intent classes. HWU64 includes 25,716 utterances for 64 intents spanning 21 domains (e.g., alarm, music, IoT, news, etc.).
Casanueva et al. (2020) forego a validation set for these datasets and instead only use a training and testing set. We instead follow the setup of Mehri et al. (2020), wherein a portion of the training set is designated as the validation set.
We evaluate in two experimental settings following prior work (Casanueva et al., 2020; Mehri et al., 2020): (1) using the full training set and (2) using 10 examples per intent or approximately  $10\%$  of the training data. In both settings, we evaluate on the validation set at the end of each epoch and perform early stopping with a patience of 20 epochs for a maximum of 100 epochs. Since the few-shot experiments are more sensitive to initialization and hyperparameters, we repeat the few-shot experiments 5 times and take an average over the experimental runs. For the few-shot settings, our models use only the few-shot training data for both masked language modelling and as examples at inference time in the example-driven models (i.e., they do not see any additional data). Our experiments with observers all use 20 observers, however we include an ablation in the appendix (Table 6; see supplementary materials).
Our experimental results, as well as the results obtained by Casanueva et al. (2020) and Mehri et al. (2020) are shown in Table 1. Combining observers and example-driven training results in (1) SoTA results across the three datasets and (2) a significant improvement over the BERT-base model, especially in the few-shot setting  $(+5.02\%)$  on average).
Furthermore, the results show that the use of observers is particularly conducive to the example-driven training setup. Combining these two approaches gains strong improvements over the ConvBERT + MLM model (few-shot: +4.98%, full data: +0.41%). However, when we consider the two proposed approaches independently, there is no consistent improvement for both example-driven (few-shot: -0.46% full data: +0.24%) and ob
servers (few-shot:  $+0\%$ , full data:  $-0.42\%$ ). The fact that these two methods are particularly conducive to each other signifies the importance of using them jointly. The representation step of intent prediction is tackled by observers, which aim to better capture the semantics of an input by disentangling the attention and therefore avoiding the dilution of the representations. The prediction step, is improved through example-driven training which uses the underlying BERT-based model to predict intents by explicitly reasoning over a set of examples. This characterization highlights the importance of jointly addressing both steps of the process simultaneously. Using observers alone does not lead to significant improvements because the linear classification layer cannot effectively leverage the improved representations. Using example-driven training alone does not lead to significant improvements because the [CLS] representations do not capture enough of the underlying utterance semantics. The enhanced semantic representation of observers is necessary for example-driven training: by improving the latent representations of utterances, it is easier to measure similarity in the set of examples.
This section describes several experiments that were carried out to show the unique benefits of observers and example-driven training, as well as to validate our hypothesis regarding the two methods. First, we show that with the example-driven formulation for intent prediction, we can attain strong performance on intents unseen during training. Next, we show that the generalization to new intents transfers across datasets. Next, we carry out a probing experiment that demonstrates that the latent representation of the observers contains greater semantic information about the input. Finally, we discuss an ablation over the number of observers used which demonstrates that the benefit of observers is primarily a consequence of the disentangled attention.
4.1 Transfer to Unseen Intents
By formulating intent prediction as a sentence similarity task, the example-driven formulation allows for the potential to predict intents that are unseen at training time. We carry out experiments in the few-shot setting for each dataset, by (1) randomly removing 4 - 10 intent classes when training in an
Model	BANKING77	CLINC150	HWU64
Few	Full	Few	Full	Few	Full
Prior Work
USE* (Casanueva et al., 2020)	84.23	92.81	90.85	95.06	83.75	91.25
CONVERT* (Casanueva et al., 2020)	83.32	93.01	92.62	97.16	82.65	91.24
USE+CONVERT* (Casanueva et al., 2020)	85.19	93.36	93.26	97.16	85.83	92.62
BERT-BASE (Mehri et al., 2020)	79.87	93.02	89.52	95.93	81.69	89.97
CONVBERT (Mehri et al., 2020)	83.63	92.95	92.10	97.07	83.77	90.43
CONVBERT + MLM (Mehri et al., 2020)	83.99	93.44	92.75	97.11	84.52	92.38
Proposed Models
CONVBERT + MLM + Example	84.09	94.06	92.35	97.11	83.44	92.47
CONVBERT + MLM + Observers	83.73	92.83	92.47	96.76	85.06	92.10
CONVBERT + MLM + Example + Observers	85.95	93.83	93.97	97.31	86.28	93.03
Table 1: Accuracy scores  $(\times 100\%)$  on all three intent detection data sets with varying number of training examples (Few: 10 training utterances per intent; Full: full training data). The full data results of Casanueva et al. (2020) are trained on more data as they forego a validation set. We follow the setup of Mehri et al. (2020), wherein a portion of the training set is used as the validation set. Results in bold-face are statistically significant by t-test  $(p < 0.01)$ .
Model	BANKING77	CLINC150	HWU64
BERT-BASE (OFF-THE-SHELF)	19.50	26.50	26.56
CONVBERT (OFF-THE-SHELF)	19.50	26.50	26.56
CONVBERT + MLM + Example	67.36	79.69	62.24
CONVBERT + MLM + Example + Observers	84.87	94.35	85.32
BEST FULLY TRAINED MODEL	85.95	93.97	86.28
Table 2: Accuracy scores  $(\times 100\%)$  for transferring to unseen intents averaged over 30 runs wherein 4-10 intents are removed from the few-shot setting during training and added back in during evaluation. The last row corresponds to the best results that were trained with all of the intents, shown in Table 1. Note that the non example-driven models are incapable of predicting unseen slots, and their perform is equivalent to random chance.
example-driven manner, (2) adding the removed intents back to the set of examples during evaluation and (3) reporting results only on the unseen intents. We repeat this process 30 times for each dataset and the results are reported in Table 2. It should be noted that we do not perform MLM training on the utterances corresponding to the unseen intents.
These results demonstrate that the example-driven formulation generalizes to new intents, without having to re-train the model. The performance on the unseen intents approximately matches the performance of the best model which has seen all intents (denoted BEST FULLY TRAINED MODEL in Table 2). These results highlight a valuable property of the proposed formulation: namely, that new
intent classes can be added in an online manner without having to re-train the model. While the off-the-shelf BERT-base and ConvBERT models, which are not at all fine-tuned on the datasets, are able to identify similar sentences to some extent – training in an example-driven manner drastically improves performance.
The addition of observers, in combination with example-driven training, significantly improves performance on this experimental setting  $(+18.42\%)$ . This suggests that the observers generalize better to unseen intents, potentially because the observers are better able to emphasize words that are key to differentiating between intents (e.g., turn the volume up vs turn the volume down).
4.2 Transfer Across Datasets
While transferring to unseen intents is a valuable property, the unseen intents in this experimental setting are still from the same domain. To further evaluate the generalizability of our models, we carry out experiments evaluating the ability of models to transfer to other datasets. Using the full data setting with 10 training utterances per intent, we (1) train a model on a dataset and (2) evaluate the models on a new dataset, using the training set of the new dataset as examples during inference. In this manner, we evaluate the ability of the models to transfer to unseen intents and domains without additional training.