SlowGuess/NAACL_2021_OCR huggingface dataset:
Dian Yu*
Luheng He
luheng@google.com
Yuan Zhang
zhangyua@google.com
Xinya Du*
xd75@cornell.edu
Panupong Pasupat
ppasupat@google.com
Qi Li
qilqil@google.com
Few-shot learning arises in important practical scenarios, such as when a natural language understanding system needs to learn new semantic labels for an emerging, resource-scarce domain. In this paper, we explore retrieval-based methods for intent classification and slot filling tasks in few-shot settings. Retrieval-based methods make predictions based on labeled examples in the retrieval index that are similar to the input, and thus can adapt to new domains simply by changing the index without having to retrain the model. However, it is nontrivial to apply such methods on tasks with a complex label space like slot filling. To this end, we propose a span-level retrieval method that learns similar contextualized representations for spans with the same label via a novel batch-softmax objective. At inference time, we use the labels of the retrieved spans to construct the final structure with the highest aggregated score. Our method outperforms previous systems in various few-shot settings on the CLINC and SNIPS benchmarks.
Few-shot learning is a crucial problem for practical language understanding applications. In the few-shot setting, the model (typically trained on source domains with abundant data) needs to adapt to a set of unseen labels in the target domain with only a few examples. For instance, when developers introduce a new product feature, a query understanding model has to learn new semantic labels from a small dataset they manage to collect.
Few-shot learning is challenging due to the imbalance in the amount of data between the source and target domains. Traditional classification methods, even with the recent advancement of pretrained language models (Peters et al., 2018; Devlin et al., 2019), could suffer from over-fitting (Snell
et al., 2017; Triantafillou et al., 2019) or catastrophic forgetting (Wu et al., 2019) when incorporating the data-scarce target domain. On the other hand, metric learning methods (Weinberger et al., 2006; Vinyals et al., 2016; Snell et al., 2017) have been shown to work well in few-shot scenarios. These methods are based on modeling similarity between inputs, effectively allowing the model to be decoupled from the semantics of the output space. For example, a model would learn that the utterance "I'd like to book a table at black horse tavern at  $7pm$  " (from Figure 1) is similar to "make me a reservation at  $8$  " and thus are likely to have similar semantic representations, even without knowing the semantic schema in use. Unlike learning output labels, which is difficult when examples are scarce, learning a similarity model can be done on the abundant source domain data, making such models data-efficient even in few-shot settings.
While there are many instantiations of metric learning methods (see Section 3), we focus on retrieval-based methods, which maintain an explicit retrieval index of labeled examples. The most basic setting of retrieval-based model for few-shot learning is: after training a similarity model and encoding target domain data into the index, we can retrieve examples most similar to the given input, and then make a prediction based on their labels. Compared to methods that do not maintain an index, such as Prototypical Networks (Snell et al., 2017), retrieval-based methods are less sensitive to outliers with few data points, and are powerful when we have abundant data in the source domain (Triantafillou et al., 2019).
However, applying retrieval-based models on tasks with a structured output space is non-trivial. For example, even if we know that the utterance in Figure 1 is similar to "make me a reservation at 8", we cannot directly use its slot values (e.g., the time slot has value "8" which is not in the input), and not all slots in the input (e.g., "black
Association for Computational Linguistics: Human Language Technologies, pages 734-749
Figure 1: Illustration of span-level retrieval for slot filling. For each span (including spans that are not valid slots such as "book a table") in the input utterance, we retrieve its most similar span from the retrieval index, and then assign the slot name as the prediction with a similarity score. We use modified beam search to decode an output that maximizes the average similarity score. The gold slots are "black horse tavern" and "7 pm" in this example.
horse tavern") have counterparts in the retrieved utterance. While previous works have exploited token-level similarity methods in a BIO-tagging framework, they had to separately simulate the label transition probabilities, which might still suffer from domain shift in few-shot settings (Wiseman and Stratos, 2019; Hou et al., 2020).
In this work, we propose Retriever, a retrieval-based framework that tackles both classification and span-level prediction tasks. The core idea is to match token spans in an input to the most similar labeled spans in a retrieval index. For example, for the span "7 pm" in the input utterance, the model retrieves "8" as a similar span (given their surrounding contexts), thus predicting that "7 pm" has the same slot name time as "8". During training, we fine-tune a two-tower model with BERT (Devlin et al., 2019) encoders, along with a novel batch softmax objective, to encourage high similarity between contextualized span representations sharing the same label. At inference time, we retrieve the most similar span from the few-shot examples for every potential input span, and then decode a structured output that has the highest average span similarity score.
We show that our proposed method is effective on both few-shot intent classification and slot-filling tasks, when evaluated on CLINC (Larson et al., 2019) and SNIPS (Coucke et al., 2018) datasets, respectively. Experimental results show that Retriever achieves high accuracy on few-shot target domains without retraining on the target data. For example, it outperforms the strongest baseline by  $4.45\%$  on SNIPS for the slot-filling task.
2 Benefits of Retriever
In addition to being more robust against overfitting and catastrophic forgetting problems, which are essential in few-shot learning settings, our proposed method has multiple advantages over strong baselines. For instance, if the scheme is changed or some prediction bugs need to be fixed, there is minimum re-training required. More importantly, compared to classification or Prototypical Networks which require adding arbitrary number of instances to the training data and hope that the model will predict as expected (Yu et al., 2019; Liang et al., 2020), Retriever can guarantee the prediction when a similar query is encountered. At the same time, Retriever is more interpretable where the retrieved examples can serve as explanations. In addition, different from the simplified assumption that one utterance may only have one intent (Bunt, 2009; Yu and Yu, 2019), Retriever can be used to predict multiple labels. Lastly, because Retriever does not need to model transition probability, the decoding procedure can be paralleled and potentially be modified to be non-autoregressive for speedup. We can also tune threshold (explained in Section 5.2) to change precision and recall according to use case requirements.
Few-shot metric learning Metric learning methods target at learning representations through distance functions. Koch et al. (2015) proposed Siamese Networks which differentiated input examples with contrastive and triplet loss functions (Schroff et al., 2015) on positive and negative pairs. While they are more data efficient for new
classes than linear classifiers, Siamese Networks are hard to train due to weak pairs sampled from training batch (Gillick et al., 2019). In comparison, Prototypical Networks (Snell et al., 2017) proposed to compute class representations by averaging embeddings of support examples for each class. These methods have been mostly explored in computer vision and text classification (Geng et al., 2019; Yu et al., 2018), and consistently outperform Siamese Networks and retrieval-based methods such as  $k$ -nearest-neighbors, especially when there are more classes and fewer annotated examples (Triantafillou et al., 2019; Sun et al., 2019). However, newly added examples which are outliers may change the prototypical representations dramatically that can harm all predictions on the class. In addition, these methods do not perform well when there are more annotated data available per class (Triantafillou et al., 2019).
Recently, Wang et al. (2019) showed that a simple nearest neighbor model with feature transformations can achieve competitive results with the state-of-the-art methods on image classification. Inspired by their work, we train our retrieval-based model with a novel batch softmax objective.
Metric learning in language understanding
Utilizing relevant examples to boost model performance has been applied to language modeling (Khandelwal et al., 2020), question answering (Guu et al., 2020; Lewis et al., 2020), machine translation (Zhang et al., 2018), and text generation (Peng et al., 2019). Recently, metric learning has been applied to intent classification (Sun et al., 2019; Krone et al., 2020). Ren and Xue (2020) trained Siamese Networks before learning a linear layer for intent classification and showed competitive results with traditional methods in the full-data setting. Similar ideas are also extended to sequence labeling tasks such as named entity recognition (NER, Wiseman and Stratos, 2019; Fritzler et al., 2019) by maximizing the similarity scores between contextual tokens representations sharing the same label. Krone et al. (2020) utilized Prototypical Networks to learn intent and slot name prototype representations and classified each token to its closest prototype. They showed better results than meta-learning, another prevalent few-shot learning method (Finn et al., 2017; Mishra et al., 2018). In order to consider label dependencies that are essential in slot tagging tasks (Huang et al., 2015), Hou et al. (2020) proposed a collapsed dependency
transfer (CDT) mechanism by simulating transition scores for the target domain from transition probabilities among BIO labels in the source domain, outperforming previous methods on slot filling by a large margin. Yang and Katiyar (2020) further explored the transition probability by evenly distributing the collapsed transition scores to the target domain to maintain a valid distribution. However, this simulation is noisy and the difference between the source and target domains can result in biased transition probabilities.
The most similar approach to ours is a concurrent work from Ziyadi et al. (2020), which learns span boundaries and sentence similarities before retrieving the most similar span, inspired by question-answering models. Even though this approach predicts spans before retrieving on the span level and thus bypasses the problem of transition probability in previous research, it only achieves unsatisfactory results. Different from these researches, we propose to learn span representations using a batch softmax objective without having to explicitly learn span boundaries. Our method achieves more accurate slot and intent prediction than previous methods in the few-shot setting.
4 Setup
We consider two tasks where the input is an utterance  $\mathbf{x}$  with tokens  $x_{1},\ldots ,x_{n}$  and the output is some structure  $\mathbf{y}$ . For the slot filling task, the output  $\mathbf{y}$  is a set of non-overlapping labeled spans  $\{(r_i,\ell_i)\}_{i = 1}^m$  where  $r_i$  is a span of  $\mathbf{x}$  (e.g., "7 pm") and  $\ell_{i}$  is a slot name (e.g., time). For the intent classification task, the output  $\mathbf{y}$  is simply an intent label  $\ell$  for the whole utterance  $\mathbf{x}$ . For notational consistency, we view intent classification as predicting a labeled span  $(r,\ell)$  where  $r = \mathbf{x}_{1:n}$ .
In the few-shot setup, examples  $(\mathbf{x},\mathbf{y})$  are divided into source and target domains. Examples in the target domain may contain some labels  $\ell$  that are unseen in the source domain. The model will be given ample training data from the source domain, but only a few training examples from the target domain. For instance, the model receives only  $K = 5$  examples for each unseen label. The model can be evaluated on test data from both domains.
We propose a retrieval-based model, Retriever, for intent classification and slot filling in the few-shot setting. Figure 1 illustrates our approach. At
a high level, from examples  $(\mathbf{x},\mathbf{y})$  in the target training data (and optionally the source training data), we construct a retrieval index consisting of labeled spans  $(r,\ell)$  from  $\mathbf{y}$ . Given a test utterance  $\mathbf{x}$ , for each span of interest in  $\mathbf{x}$  (all spans  $\mathbf{x}_{i:j}$  for slot filling; only  $\mathbf{x}_{1:n}$  for intent classification), we retrieve the most similar labeled spans  $(r,\ell)$  from the index, and then use them to decode an output  $\mathbf{y}$  that maximizes the average span similarity score.
The use of retrieval provides several benefits. For instance, we empirically show in Section 7.1 that the model does not suffer from catastrophic forgetting because both source and target data are present in the retrieval index. Class imbalance can also be directly mitigated in the retrieval index. Additionally, since the trained model is nonparametric, we could replace the retrieval index to handle different target domains without having to retrain the model. This also means that the model does not need access to target data during training, unlike traditional classification methods.
5.1 Retriever
The retriever is the only trainable component in our model. Given a query span  $r' = \mathbf{x}_{i:j}$  from the input  $\mathbf{x}$ , the retriever returns a set of labeled spans  $(r, \ell)$  with the highest similarity scores  $s(\mathbf{z}, \mathbf{z}')$ , where  $\mathbf{z} = E(r)$  and  $\mathbf{z}' = E(r')$  are the contextualized embedding vectors of  $r$  and  $r'$ , respectively.
Similarity score To compute the contextualized embeddings  $\mathbf{z}$  and  $\mathbf{z}'$  of spans  $r$  and  $r'$ , we first apply a Transformer model initialized with pretrained BERT on the utterances where  $r$  and  $r'$  come from. For slot filling, we follow Toshniwal et al. (2020) and define the span embedding as the concatenated embeddings of the its first and last wordpieces. For intent classification, we use the embedding of the [CLS] token. We then define  $s(\mathbf{z},\mathbf{z}')$  as the dot product between  $\mathbf{z}$  and  $\mathbf{z}'$ .
Training with batch softmax We use examples from the source domain to train Retriever. Let  $\ell_1, \ldots, \ell_N$  be the  $N$  class labels (slot or intent labels) in the source domain. To construct a training batch, for each class label  $\ell_i$ , we sample  $B$  spans  $r_i^1, \ldots, r_i^B$  from the training data with that label, and compute their embeddings  $\mathbf{z}_i^1, \ldots, \mathbf{z}_i^B$ .
Then, for each query span  $r_i^j$ , we compute similarity scores against all other spans in the batch to form a  $B \times N$  similarity matrix:
$$
S _ {i} ^ {j} = \left[ \begin{array}{c c c c} s \left(\mathbf {z} _ {i} ^ {j}, \mathbf {z} _ {1} ^ {1}\right) & s \left(\mathbf {z} _ {i} ^ {j}, \mathbf {z} _ {2} ^ {1}\right) & \dots & s \left(\mathbf {z} _ {i} ^ {j}, \mathbf {z} _ {N} ^ {1}\right) \\ s \left(\mathbf {z} _ {i} ^ {j}, \mathbf {z} _ {1} ^ {2}\right) & s \left(\mathbf {z} _ {i} ^ {j}, \mathbf {z} _ {2} ^ {2}\right) & \dots & s \left(\mathbf {z} _ {i} ^ {j}, \mathbf {z} _ {N} ^ {2}\right) \\ \vdots & \vdots & \ddots & \vdots \\ s \left(\mathbf {z} _ {i} ^ {j}, \mathbf {z} _ {1} ^ {B}\right) & s \left(\mathbf {z} _ {i} ^ {j}, \mathbf {z} _ {2} ^ {B}\right) & \dots & s \left(\mathbf {z} _ {i} ^ {j}, \mathbf {z} _ {N} ^ {B}\right) \end{array} \right]. \tag {1}
$$
We now summarize the score between  $r_i^j$  and each label  $\ell_{i'}$  by applying a reduction function (defined shortly) along each column to get a  $1 \times N$  vector:
$$
\hat {S} _ {i} ^ {j} = \left[ s \left(\mathbf {z} _ {i} ^ {j}, \mathbf {z} _ {1} ^ {*}\right) \quad s \left(\mathbf {z} _ {i} ^ {j}, \mathbf {z} _ {2} ^ {*}\right) \quad \dots \quad s \left(\mathbf {z} _ {i} ^ {j}, \mathbf {z} _ {N} ^ {*}\right) \right] \tag {2}
$$
We use the softmax of  $\hat{S}_i^j$  as the model's probability distribution on the label of  $r_i^j$ . The model is then trained to optimize the cross-entropy loss on this distribution against the gold label  $\ell_i$ .
We experiment with three reduction functions, mean (Eq. 3), max (Eq. 4), and min-max (Eq. 5):
$$
s \left(\mathbf {z} _ {i} ^ {j}, \mathbf {z} _ {i ^ {\prime}} ^ {*}\right) = \frac {1}{B} \sum_ {j ^ {\prime} = 1} ^ {B} s \left(\mathbf {z} _ {i} ^ {j}, \mathbf {z} _ {i ^ {\prime}} ^ {j ^ {\prime}}\right) = s \left(\mathbf {z} _ {i} ^ {j}, \frac {1}{B} \sum_ {j ^ {\prime} = 1} ^ {B} \mathbf {z} _ {i ^ {\prime}} ^ {j ^ {\prime}}\right) \tag {3}
$$
$$
s(\mathbf{z}_{i}^{j},\mathbf{z}_{i^{\prime}}^{*}) = \max_{\substack{1\leq j^{\prime}\leq B;\\ j^{\prime}\neq j\text{if} i = i^{\prime}}}s(\mathbf{z}_{i}^{j},\mathbf{z}_{i^{\prime}}^{j^{\prime}}) \qquad (4)
$$
$$
s \left(\mathbf {z} _ {i} ^ {j}, \mathbf {z} _ {i ^ {\prime}} ^ {*}\right) = \left\{ \begin{array}{l l} \min  _ {1 \leq j ^ {\prime} \leq B} s \left(\mathbf {z} _ {i} ^ {j}, \mathbf {z} _ {i ^ {\prime}} ^ {j ^ {\prime}}\right), & \text {i f} i = i ^ {\prime} \\ \max  _ {1 \leq j ^ {\prime} \leq B} s \left(\mathbf {z} _ {i} ^ {j}, \mathbf {z} _ {i ^ {\prime}} ^ {j ^ {\prime}}\right), & \text {o t h e r w i s e} \end{array} \right. \tag {5}
$$
The mean reduction averages embeddings of the spans with the same label and is equivalent to Prototypical Networks. Similar to hard negative sampling to increase margins among classes (Schroff et al., 2015; Roth et al., 2020; Yang et al., 2019), max takes the most similar span to the query (excluding the query itself) as the label representation, while min-max takes the least similar span when considering spans with the same label as the query.
After training, we build a dense retrieval index where each entry  $(r,\ell)$  is indexed by  $\mathbf{z} = E(r)$ . The entries  $(r,\ell)$  come from examples  $(\mathbf{x},\mathbf{y})$  in the support set which, depending on the setting, could be just the target training data or a mixture of source and target data. For each query span  $r'$  of the input utterance  $\mathbf{x}$ , we embed the span and compute the similarity scores against all index entries.
<sup>1</sup>We experimented with affine transformation as well as cosine similarity but did not see any performance gain. For intent classification, using the [CLS] token achieves better results than averaging word embeddings.
Intent classification For intent classification, both index entries and query spans are restricted to the whole utterances. The entire process thus boils down to retrieving the most similar utterance based on the [CLS] token embedding. We simply output the intent label of the retrieved utterance.
Slot filling In contrast to BIO decoding for token-level similarity models (Hou et al., 2020), decoding with span retrieval results poses unique challenges as gold span boundaries are not known a priori. Hence, we use a modified beam search procedure with simple heuristics to compose the spans.
Specifically, for each of the  $n \times m$  spans in an utterance of length  $n$  (where the hyperparameter  $m$  is the maximum span length), we retrieve the most similar span from the retrieval index. Then we normalize the similarity scores by L2-norm so that they are within the range [0, 1]. Since we do not explicitly predict span boundaries, all  $n \times m$  spans, including non-meaningful ones (e.g., "book  $a$ "), will have a retrieved span. Such non-meaningful spans should be dissimilar to any labeled span in the retrieval index. We thus choose to filter the spans with a score threshold to get a smaller set of candidate spans. In addition, we adjust the threshold dynamically (by reducing the threshold for a few times) if no span is above the current threshold.
Once we get candidate spans with similarity scores, we use beam search to decode a set of spans with maximum average scores. We go through the list of candidate spans in the descending order of their similarity scores. For each candidate span, we expand beam states if the span does not overlap with the existing spans in the beam. The search beams are pruned based on the average similarity score of the spans included so far. Lastly, we add spans in the filtered set which do not overlap with the final beam.
Beam search can avoid suboptimal decisions that a greedy algorithm would make. For instance, if we greedily process the example in Figure 1, "black" and "tavern" would become two independent spans, even though their average similarity score is lower than the correct span "black horse tavern". Nevertheless, beam search is prone to mixing up span boundaries and occasionally predicts consecutive partial spans such as "black horse" and "tavern" as individual slots. Since consecutive spans of the same slot label are rare in slot filling
datasets, we merge the two spans if their retrieval scores are within a certain range:
$$
r _ {i: j}, r _ {j: k} = \left\{ \begin{array}{l l} r _ {i: k} & \text {i f} | s (\mathbf {z} _ {i: j}, \mathbf {z} ^ {\prime}) - s (\mathbf {z} _ {j: k}, \mathbf {z} ^ {\prime \prime}) | <   \lambda \\ r _ {i: j}, r _ {j: k} & \text {o t h e r w i s e} \end{array} \right.
$$
where  $r_{i:j}$  and  $r_{j:k}$  are two consecutive potential spans sharing the same label, and  $\mathbf{z}'$  and  $\mathbf{z}''$  are the embeddings of their retrieved spans, respectively ( $r_{i:k}$  indicates merging the two spans into one span;  $\lambda$  is the merge threshold where  $\lambda = 1$  means always merge and  $\lambda = 0$  means never merge).
We evaluate our proposed approach on two datasets: CLINC (Larson et al., 2019) for intent classification and SNIPS (Coucke et al., 2018) for slot filling. Note that we use max (Eq. 4) as the reduction function for both tasks since it empirically yields the best results. The effect of reduction functions will be analyzed later in Section 7.1.
6.1 Intent Classification
The CLINC intent classification dataset (Larson et al., 2019) contains utterances from 10 intent categories (e.g., "travel"), each containing 15 intents (e.g., "flight_status", "book_flight"). To simulate the few-shot scenario where new domains and intents are introduced, we designate  $n_c$  categories and  $n_i$  intents per category as the source domain (with all 100 training examples per intent), and use the remaining  $150 - n_c \times n_i$  intents as the target domain. We experiment with  $(n_c, n_i) = (10, 10)$ , (8, 10), and (5, 15). The target training data contains either 2 or 5 examples per target intent.
We compare our proposed method Retriever with a classification model BERT fine-tune and a Prototypical Network model Proto. The former learns a linear classifier on top of BERT embeddings (Devlin et al., 2019), and the latter learns class representations based on Prototypical Networks.4 We also show results with the initial BERT checkpoint without training (Protofrz, Retrieverfrz). We use the same batch size for all models, and tune other hyperparameters on the development set before testing.
$^3 (n_c,n_i) = (10,10)$  simulates the situation where all the categories are known, but we adapt to new intents in all 10 categories;  $(n_{c},n_{i}) = (5,15)$  simulates the situation where we adapt to 5 entirely new intent categories.
4 Previous work show that Prototypical Networks outperforms other optimization-based and metric-learning models such as MAML in (intent) classification tasks (Triantafillou et al., 2019; Krone et al., 2020).
2We use beam search for simplicity. Other search methods such as Viterbi algorithm (Forney, 1973) can also be used.