SlowGuess/NAACL_2022_OCR huggingface dataset:
Tingting Ma $^{1*}$ , Qianhui Wu $^{2}$ , Zhiwei Yu $^{2}$ , Tiejun Zhao $^{1}$ , Chin-Yew Lin $^{2}$
hittingtingma@gmail.com
{qianhuiwu, zhiwyu, cyl}@microsoft.com, tjzhao@hit.edu.cn
Recent studies on few-shot intent detection have attempted to formulate the task as a meta-learning problem, where a meta-learning model is trained with a certain capability to quickly adapt to newly specified few-shot tasks with potentially unseen intent categories. Prototypical networks have been commonly used in this setting, with the hope that good prototypical representations could be learned to capture the semantic similarity between the query and a few labeled instances. This intuition naturally leaves a question of whether or not a good sentence representation scheme could suffice for the task without further domain-specific adaptation. In this paper, we conduct empirical studies on a number of general-purpose sentence embedding schemes, showing that good sentence embeddings without any fine-tuning on intent detection data could produce a nontrivially strong performance. Inspired by the results from our qualitative analysis, we propose a frustratingly easy modification, which leads to consistent improvements over all sentence encoding schemes, including those from the state-of-the-art prototypical network variants with task-specific fine-tuning. $^{1}$
The task of intent detection aims at classifying user queries, typically in the form of short sentences, into intent categories. It has been widely adopted as one crucial component inside various applications such as dialogue systems, virtual assistants, and search engines. The domain-specific nature of those applications makes intent detection rather challenging because of the difficulty to acquire high-quality labeled data at scale. In particular, the sets of intents could vary a lot in different real-world scenarios. Such scenarios motivate the research of few-shot intent detection, which aims
to classify utterances with new intent labels given very few labeled examples.
Recently, there exists a popular stream of research efforts (Yu et al., 2018; Geng et al., 2019; Nguyen et al., 2020; Dopierre et al., 2021a,b) that models the few-shot intent detection task as a meta-learning problem - a general machine learning paradigm which has already been successfully applied in other tasks of natural language processing (Han et al., 2018; Gu et al., 2018; Chen et al., 2019; Hou et al., 2020, inter alia). Under this formulation, the target is to train a good meta-learner that could be used to quickly adapt to any few-shot intent classification task with very few labeled examples. One of the most popular methods of meta-learning is the prototypical network (Snell et al., 2017), which learns an embedding of the input data, and then constructs a prototypical representation for every class via averaging over the input embeddings. Each query will be classified as the class with the minimum distance between the query embedding and the class prototype. Earlier empirical findings (Dopierre et al., 2021a) suggest that the prototypical networks could reach the state-of-the-art performance on most intent detection datasets when a text encoder based on fine-tuned BERT (Devlin et al., 2019) is used for sentence representation.
However, one notable challenge for prototypical networks, or basically most of the current meta-learning approaches is that the models could easily overfit the sparse and biased data distribution from merely a few training instances (Yang et al., 2021). Given that the goal of prototypical networks is essentially to learn proper encoding functions for nearest-neighbor classification, one natural question arises: is it possible to utilize other general-purpose sentence representation schemes without fine-tuning on intent detection data? In this way, the cost of collecting and fine-tuning on domain-specific labeled data might be mitigated,
*Work during internship at Microsoft Research Asia.
Our code is available at https://github.com/microsoft/KC/tree/main/papers/IDML.
Human Language Technologies, pages 3806 - 3818
while reducing the risk of domain-specific overfitting.
In this paper, we conduct an empirical study to verify the utility and the effectiveness of recent popular sentence encoding schemes for intent detection meta-learning. Specifically, we make the following contributions:
- We empirically compare a number of popular sentence embedding methods on various intent detection benchmarks and observe non-trivial strong performance in the meta-learning setup for few-shot intent detection.
- We quantitatively verify the better capability for cross-dataset generalization from general-purpose sentence encoders, and conduct qualitative analysis on the behaviors of different sentence encoding schemes.
- Based on our analysis, we propose a frustratingly simple modification to utilize the label name information, with the hope to yield sentence representations more targeted at the intent detection task. Follow-up experiments show consistent and substantial improvements over all sentence encoding methods, making them stronger baselines for the task, while the modification could also improve the state-of-the-art system performance.
2.1 The meta-learning setup
The meta-learning setting aims at learning a good "meta-learner" that could quickly adapt to newly specified classification tasks with few labeled examples available. A few-shot task, called an episode, is denoted as  $\mathcal{T} = (\mathcal{S},\mathcal{Q},\mathcal{Y})$ , usually following an  $N$ -way  $K$ -shot setting: given a support set  $\mathcal{S} = \{(x_i,y_i)\}_{i=1}^{N\times K}$  containing a small number of  $K$  labeled examples for each class  $c\in \mathcal{V}$ $(|\mathcal{V}| = N)$ , the model is expected to assign a label from  $\mathcal{V}$  to each instance in the query set  $\mathcal{Q} = \{(x_j,y_j)\}_{j=1}^{N\times K}$ . At the meta-training phase, the meta-learner is trained from a series of episodes sampled from training data with classes  $\mathcal{V}_{train}$ , via updating based on the prediction loss on  $\mathcal{Q}$ . At the meta-testing phase, the meta-learner is evaluated on many episodes  $\mathcal{T}' = (\mathcal{S}',\mathcal{Q}',\mathcal{Y}')$  constructed from test data, with each  $\mathcal{V}'\subset \mathcal{V}_{test}$  from a non-overlapping label space to verify the meta-learning capability, i.e.,  $\mathcal{V}_{test}\cap \mathcal{V}_{train} = \emptyset$ .
2.2 Prototypical networks
Formally, a prototypical network (Snell et al., 2017, ProtoNet) learns an encoding function  $E_{\phi}$  (parameterized by  $\phi$ ) to embed a sentence  $x_{i}$  into a vector  $E_{\phi}(x_{i})$ . The class prototype  $e_{c}$  for each class  $c$  is obtained by taking the average embedding of sentences with the label  $c$  in the support set  $S$ :
$$
e _ {c} = \frac {1}{K} \sum_ {(x _ {i}, y _ {i}) \in \mathcal {S}: y _ {i} = c} E _ {\phi} (x _ {i}).
$$
With these prototype vectors, the predicted class distribution in the label space  $\mathcal{V}$  for a query  $x$  is calculated by
$$
p (y = c | x) = \frac {\exp (- d (E _ {\phi} (x) , e _ {c}))}{\sum_ {c \in \mathcal {Y}} \exp (- d (E _ {\phi} (x) , e _ {c}))},
$$
where  $d$  is a distance metric, usually set to be the Euclidean distance. The encoder is trained by minimizing the cross-entropy loss on the query set of the episodes from  $\mathcal{V}_{train}$ .
3 Our Empirical Study
3.1 Adapting generic sentence embedding
The main goal of this work is to explore the effectiveness of general-purpose sentence embedding methods without fine-tuning on intent data. A high-quality sentence embedding could be used to identify which instance in the few labeled examples is semantically close to an input query and henceforth expressing the same intent. This intuition makes it natural to directly adapt sentence encoding to ProtoNet. Specifically, for whatever pre-trained encoder  $E_{s}$  to produce sentence embedding, we replace the encoder  $E_{\phi}$  with  $E_{s}$  in the ProtoNet. Note that we take a pre-trained  $E_{s}$  as-is, in other words, there is no meta-training phase.
We experiment with a number of modern popular sentence embedding methods, covering sentence embeddings pre-trained from either large-scale unlabeled text data or with supervision from additional sentence pairs. Specifically, the following four typical methods yielding five specific model instances in total are used in our experiments:
Sentence-BERT Sentence-BERT (Reimers and Gurevych, 2019) takes BERT / RoBERTa (Liu et al., 2019b) as the basic encoder and uses Siamese and triplet network (Schroff et al., 2015) structure to derive sentence embeddings by comparing similarities between sentence pairs. Here we consider
two model instances pre-trained on different data: i) SBERT-paraphrase, a DistillRoBERTa (Sanh et al., 2019) based model trained on a broad range of paraphrase corpora; ii) SBERT-NLI, a RoBERTa based model trained on SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018), using a three-way classification objective to predict the relationship of a pair of sentences, i.e., entailment, neural, or contradiction. Both model instances utilize mean pooling over token representations in a sentence for sentence representation.
SimCSE SimCSE (Gao et al., 2021) learns sentence embeddings by contrastive learning. Specifically, it encodes a sentence with the RoBERTa model and takes the representation of the [CLS] token as the sentence representation. Given an anchor sentence, the model is trained to predict the "positive" example, i.e., the most semantic similar example, among the "negatives". Here we consider the situation that all anchors and their positive as well as negative examples are constructed from SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018), denoted as SimCSE-NLI.
DeCLUTR DeCLUTR (Giorgi et al., 2021) is an unsupervised sentence embedding method trained on documents from OpenWebText (Gokaslan and Cohen, 2019) and a subset of WebText (Radford et al., 2019). The mean pooling of contextual word representations obtained from RoBERTa is used as the sentence embedding. The sentence encoder is trained with a self-supervised contrastive loss to minimize the distance between the embeddings of textual segments randomly sampled from nearby in the same document.
SP-paraphrase Different from afore-mentioned models that are built upon large pre-trained language models, Wieting et al. (2019, 2021) propose a lightweight paraphrastic sentence embedding method, denoted as  $SP$ -paraphrase.  $SP$ -paraphrase first tokenizes a sentence into subwords with SentencePiece (Kudo and Richardson, 2018), then learns context-independent embeddings for sub-word tokens within a pre-defined vocabulary, and finally averages over all sub-word embeddings of a sentence for the sentence embedding. This model is trained on ParaNMT (Wieting and Gimpel, 2018) with a margin loss and carefully selected negative examples.
3.2 Meta-learning methods for reference
To study the different behaviors between general sentence encoding and meta-learning algorithms trained on intent datasets, we also compare with the representative and the start-of-art meta-learning algorithms as following:
ProtoNet As described before.
ProtoNet+MLM The unlabeled data of the target dataset is used for finetuning the pretrained language model with the masked language modeling (MLM) objective. This step is intentionally serving as a kind of domain adaptation, leading to a finetuned encoder to be used as the initial base encoder of ProtoNet (Dopierre et al., 2021a).
ProtAugment The current start-of-the-art framework in intent detection meta-learning proposed by Dopierre et al. (2021b). A paraphrasing model is trained to produce multiple diverse paraphrases for an unlabeled sentence from the training, development, or testing instance. Based on ProtoNet+MLM, the prototypical network is trained with an additional consistency loss to make the embedding of a sentence to be closer to the unlabeled prototype embedding of its paraphrases, and more distant away from other unlabeled prototypes.
We evaluate these methods on four datasets for a comprehensive analysis and fair comparison with Dopierre et al. (2021b).
Banking77 Banking77 (Casanueva et al., 2020) is a single-domain dataset that contains 77 fine-grained intents about banking. It consists of 13,083 customer service queries and many of the intents are partially overlapped (e.g. verify top up, top up limits, pending top up). 25 intent classes are used for training, 25 for development and the remaining classes are for testing.
HWU64 HWU64 (Liu et al., 2019a) contains 11,036 user-generated utterances about home robots covering 64 intents from 21 different domains such as alarm and calendar. This dataset is class-balanced and each intent has 190 instances. Intents are split into train, dev, and test by domains to minimize the label semantic sharing amongst splits.
Liu57 This is also a multi-domain intent dataset from Liu et al. (2019a) which contains 25,478 user
$^{2}$ https://www.sbert.net/examples/training/paragraphs/README.html
Method	Base Model	Banking77	HWU64	Liu57	Clinic150
K=1	K=5	K=1	K=5	K=1	K=5	K=1	K=5
Meta-learning models trained on intent dataset:
ProtoNet	RoBERTa-base	86.98±1.09	94.37±0.39	79.23±2.35	91.77±1.12	82.51±1.99	93.16±0.97	94.61±0.74	98.55±0.26
ProtoNet+MLM	RoBERTa-base	89.38±1.42	95.84±0.52	85.11±1.42	93.47±0.83	87.33±2.61	95.16±0.80	96.89±0.28	98.89±0.26
ProtAugment	RoBERTa-base	90.34±0.99	96.28±0.49	85.61±1.37	93.88±0.76	88.08±1.70	95.33±0.64	97.26±0.28	99.10±0.26
ProtAugment*	BERT-base-cased	89.56	94.71	84.34	92.55	86.11	93.70	96.49	98.74
Pre-trained general sentence encodings:
SBERT-paraphrase	DistilRoBERTa	81.32±1.43	94.35±0.55	77.01±1.24	91.73±1.22	77.90±1.46	93.84±0.97	90.87±0.81	98.55±0.18
SBERT-NLI	RoBERTa-base	78.97±1.33	93.69±0.58	78.18±1.41	92.31±1.01	79.45±2.49	93.81±1.07	91.24±0.65	98.55±0.29
SimCSE-NLI	RoBERTa-base	78.62±0.91	93.44±0.68	77.37±1.86	92.00±0.95	78.65±2.43	93.73±1.22	90.95±0.70	98.54±0.24
DeCLUTR	RoBERTa-base	71.75±1.29	91.26±0.90	71.13±2.85	90.33±1.02	71.07±2.63	91.93±1.17	85.71±1.13	98.32±0.30
SP-paraphrase	Sentence-Piece	78.44±1.47	92.81±0.53	73.45±2.03	89.00±1.19	74.80±1.87	89.50±1.49	86.11±2.26	96.68±0.68
Table 1: Performance comparison under the 5-way K-shot settings. ProtAugment* denotes results reported in Dopierre et al. (2021b) with BERT-base-cased model. The highest accuracies of meta-learning models are underlined while those of general sentence embedding methods are bolded.
utterances about home robots with a total number of 54 intent classes. The class distribution is highly imbalanced, with the most frequent intent (query) accounts for  $23\%$  of all utterances while the least frequent intent (volume_other) only occupies  $0.09\%$ . Each of the train, dev, and test splits has 18 classes.
Clink150 Larson et al. (2019) introduce a crowdsourced dataset containing 22,500 user queries covering 150 different intents in ten general domains. We randomly select 50 classes for training, development, and testing, respectively.
We evaluate all methods on the standard 5-way 1-shot and 5-way 5-shot settings as in Dopierre et al. (2021b). We compute the averaged accuracy on 600 episodes and there are five query examples in each episode. To reduce the performance variation, we run all experiments five times with five different class splits and report the averaged accuracy.
Table 1 shows the performance of all approaches on four benchmarks. We can see that general sentence embeddings without any task-specific finetuning achieve non-trivial performance. Compared with meta-learning models elaborately designed for few-shot learning, general sentence embedding can reach a rather strong performance on all datasets in the 5-way 5-shot setting, even outperform or on par with the representative ProtoNet on HWU64, Liu57, and Clinic150. For the 5-way 1-shot setting, general sentence embeddings yield less satisfactory results than meta-learning. We suspect that the finetuned meta-learning models
could be less sensitive to the distribution of support examples, especially in the 1-shot setting.
4.1 Sentence embedding visualization
Here we try to obtain a more intuitive understanding of the sentence embeddings obtained from different methods via low-dimensional projection using t-SNE (van der Maaten and Hinton, 2008). For meta-learning based models, we use the sentence encoders trained on the HUW64 training split in the 5-way 1-shot setting. Figure 1 shows the projection results of sentences from 10 randomly selected classes in the HWU64 test split.
1) Meta-learning based methods generally produce sentence embeddings with larger between-class distances and smaller within-class distances on most classes. Compared with general pre-trained sentence embeddings, embeddings of different classes derived from meta-learning methods are more compact and distinguishable, which enables these meta-learning based models to achieve more robust test performance when given different support examples for similarity comparison, especially in the 1-shot setting.
2) Meta-learning based methods are good at handling test classes that share similar patterns with training classes. As shown in Table 2, when only testing on the class remove_calendar and class query_calendar, all meta-learning models significantly outperform sentence embeddings since two similar classes query alarm and remove alarm exist in the training data.
3Training intent classes: query alarm, remove alarm, set alarm, email_addcontact, music_likeness, query_music, music_settings, query_news, query_TRANport, transport_taxi, transport-ticket, transportTraffic, query/weather, query_email, query_emailcontact, send_email.
(a) ProtoNet
(b) ProtoNet+MLM
(c) ProtAugment
(d) SBERT-paraphrase
(e) SBERT-NLI
(f) SimCSE-NLI
(g) DeCLUTR
Figure 1: T-SNE visualization of different sentence representations from 10 randomly selected classes in the HWU64 test split. Intent classes: query Calendar, remove Calendar, volume_mute, volume_up, volume_down, QA_defined, QA_math, iot_wemo_on, iot_coffee, iotCleaning.
(h) SP-paraphrase
Type Set	Meta-Learning Models trained on HWU64	General Sentence Embeddings	ProtAugment+SentEmb
ProtoNet	ProtoNet+MLM	ProtAugment	SBERT-para.	SBERT-NLI	SimCSE-NLI	DeCLUTR	SP-para.	PA-SBERT-NLI	PA-SimCSE-NLI
Calendar/*	83.77	95.90	95.18	76.55	78.77	77.88	72.12	68.68	-	-
Volume/*	47.43	50.73	48.46	51.73	62.19	60.32	45.42	57.60	55.09↓	53.09↓
3) General sentence embeddings may achieve superior performance than meta-learning based models on target tasks that require fine-grained information but share little knowledge with the training tasks. In Figure 1, all methods seem to struggle in differentiating the fine-grained intent classes volume_up, volume_down, and volume_mute, which are very dissimilar to classes in training tasks. To quantify the ability of different methods to discriminate them, we evaluate all methods only on these three classes in the 1-shot setting. Surprisingly, as shown in Table 2, meta-learning models lag behind almost all general sentence embedding methods. To investigate whether this inferiority is caused by overfitting, we take SBERT-NLI and SimCSE-NLI $^4$  as base encoders of ProtAugment, denoted as PA-SBERT-NLI and PA-SimCSE-NLI. Table 2 indicates that compared SBERT-NLI and SimCSE-NLI, finetuning will lead to a performance drop of 7.10 points and 7.23 points respectively, which verifies that meta-learning based models may have a bias towards categorizing intent classes similar to training
tasks, which, on the debit side, restraints their capabilities on distinguishing fine-grained test classes that share little knowledge with training classes.
Table 2: Accuracy on particular type sets from HWU64 test classes under 1-shot setting. Volume/\* denotes {volume_up, volume_down, volume_mute}. while Calendar/\* denotes {query Calendar, remove Calendar}. PA- denotes applying the meta-learning procedure as in ProtAugment.
Model	Source dataset	5-way 1-shot	5-way 5-shot
ProtAugment	HWU64	73.20±2.40	90.81±1.02
ProtAugment	Liu57	71.70±3.92	90.16±1.35
ProtAugment	Clinic150	79.35±1.49	92.86±0.57
SBERT-para.	paraphrase data	81.32±1.43	94.35±0.55
SBERT-NLI	NLI data	78.97±1.33	93.69±0.58
SimCSE-NLI	NLI data	78.62±0.91	93.44±0.68
DeCLUTR	unlabeled data	71.75±1.29	91.26±0.90
SP-para.	ParaNMT	78.44±1.47	92.81±0.53
PA-SBERT-para.	HWU64	80.97±1.67 ↓	93.72±0.68 ↓
PA-SBERT-para.	Liu57	79.56±2.52 ↓	93.96±0.87 ↓
PA-SBERT-para.	Clinic150	82.77±1.39 ↑	94.75±0.54 ↑
Table 3: Accuracy on Banking77 dataset while the models/sentence encodings are trained on source datasets. PA-SBERT-para. denotes training ProtAugment on intent data by taking SBERT-para. as initial encoder.
4.2 Cross-dataset generalization
To further verify our hypothesis that meta-learning methods could fail when the target tasks require fine-grained information but share little knowledge with the training tasks, we test models on a more challenging setting: directly transfer ProtAugment trained on other datasets to the single-domain Bank-
4We choose these two models because they achieve best performance in Table 2.
Query:	[create_or_addlists] add new item to list
Support #1:	[create_or_addlists] add to my groceries
Support #2:	[removeLists] delete list
Predict:	ProtAugment: [create_or_addlists] GeneralSentEmb: [removeLists]
Query:	[receiving-money] How can my friend transfer money to me?
Support #1:	[receiving-money] How can my friend pay me?
Support #2:	[beneficiary.notAllowed] I tried to transfer cryptocurrency into my account but was denied
Predict:	ProtAugment: [beneficiary.notAllowed] GeneralSentEmb: [receiving-money]
Query:	[meaning_of_life] would you let me know what the meaning of life is
Support #1:	[meaning_of_life] is there a reason people exist
Support #2:	[are_you_a_robot] can you tell me know what kind of life form you are
Predict:	ProtAugment: [are_you_a_robot] GeneralSentEmb: [are_you_a_robot]
Table 4: Case study under 1-shot setting. Green color means the method predicts correctly while red color means wrong prediction. GeneralSentEmb means all general sentence embedding methods.
ing77 dataset.5 From Table 3, we find:
1) General sentence embeddings have better cross-dataset generalization performance. ProtAugment, which shows better in-domain accuracy, lags behind most general sentence embeddings under the challenging cross-dataset generalization test. ProtAugment trained on HWU64 and Liu57 performs the worst, even underperforms the sentence embedding baselines since their intents are about home robot, which is obviously different from Banking77. ProtAugment trained on Clinic150 achieves better performance since Clinic150 has several intents $^{6}$  from bank domain, but still inferior to SBERT-paraphrase on 5-way 1-shot setting, SBERT-paraphrase, SBERT-NLI, SimCSE-NLI on 5-way 5-shot setting. This indicates the meta-learning methods could overfit the task distribution from training datasets. Benefiting from the supervision signal and diverse data distribution coverage provided by labeled sentence pairs (paraphrase, or NLI), such tasks guided the general sentence encoding to encode more fine-grained information which might relevant to target intent labels in any granularity, and moreover, reducing the risk of overfitting to a small intent dataset.
2) Finetuning the strongest general sentence embedding by meta-learning algorithm on intent datasets struggles to bring significant improvement compared with raw sentence embedding.
PA-SBERT-para. even drops performance compared to SBERT-para. when finetuning on HWU64 and Liu57 datasets. This indicates that finetuning sentence embedding on a small intent corpus dis
similar to test data may cause overfitting and harm the cross-dataset generalization. When finetuning on a more similar corpus Clinic 150, the limited improvement questions the necessity of current meta-learning algorithms in practical large-domain gap scenarios when a high-quality general sentence embedding is available.
4.3 Case study
We conduct qualitative analysis to get a better understanding of the behaviors from different methods. A few examples have been listed in Table 4.