SlowGuess/NAACL_2022_OCR huggingface dataset:
1) General sentence encoding captures semantic relatedness instead of pure intent similarity between query and support, which may sometimes be misleading. For the first example, all general sentence embedding methods fail due to the sharing part "list" between the query example and the support example of the wrong label. This indicates that the sentence embeddings actually capture the relatedness between two sentences instead of intents. So the general sentence embeddings tend to be misled by irrelevant parts in the sentences which do not convey the real intent. However, ProtAugment can focus on the key parts such as verb phrases for identifying intents by domain specific finetuning.
2) Patterns learned by meta-learning models could overfit and fail in cross-dataset settings. The second example from Banking77 shows an interesting case where ProtAugment trained on HWU64 fails in the cross-dataset scenario. The support example of the category receiving-money and the query are semantically close, therefore no surprise for our sentence embedding baselines to get it correct. However, ProtAugment makes the wrong prediction by incorrectly focusing on the same verb "transfer" between
<sup>5</sup>We choose Banking77 as test dataset since it contains the most fine-grained intents (bank domain) and is significantly different from other datasets. HWU64, Clinic150, and Liu57 share some intents, like "recipe", "query weather", "traffic".
${}^{6}$  such as "transaction", "exchange_rate", "credit_limit"
Method	Banking77	HWU64	Liu57	Clinic150
K=1	K=5	K=1	K=5	K=1	K=5	K=1	K=5
L-ProtoNet+MLM	94.00 (+4.62)	95.97 (+0.13)	91.30 (+6.19)	94.09 (+0.62)	92.00 (+4.67)	95.26 (+0.10)	98.36 (+1.47)	99.08 (+0.19)
L-ProtAugment	93.42 (+3.08)	96.11 (-0.17)	91.73 (+6.12)	94.15 (+0.27)	92.79 (+4.71)	95.34 (+0.01)	98.43 (+1.17)	99.19 (+0.09)
L-SBERT-paraphrase	88.94 (+7.62)	94.53 (+0.18)	87.09 (+10.08)	92.43 (+0.70)	88.03 (+10.40)	94.11 (+0.27)	96.49 (+5.62)	98.67 (+0.12)
L-SBERT-NLI	88.30 (+9.33)	94.06 (+0.37)	87.28 (+9.10)	92.90 (+0.59)	88.31 (+8.86)	94.06 (+0.25)	96.11 (+4.87)	98.70 (+0.15)
L-SimCSE-NLI	88.19 (+9.57)	93.44 (0.00)	87.14 (+9.77)	92.70 (+0.70)	89.09 (+10.44)	94.29 (+0.56)	96.35 (+5.40)	98.74 (+0.20)
L-DeCLUTR	82.33 (+10.58)	91.91 (+0.65)	80.95 (+9.82)	91.22 (+0.89)	80.61 (+9.54)	92.40 (+0.47)	93.05 (+7.34)	98.50 (+0.18)
L-SP-paraphrase	89.00 (+10.56)	93.29 (+0.48)	83.20 (+9.75)	90.19 (+1.19)	82.09 (+7.28)	89.90 (+0.40)	91.81 (+5.70)	96.49 (-0.19)
Table 5: Accuracy with improvement value obtained after adding label name under 5-way 1-shot setting and 5-way 5-shot setting. We denote all methods adding the label names in the support set with a “L-” prefix. The absolute improvement accuracy is shown in (parentheses) compared the baselines without label names.
the query and support example for the wrong label, while ignoring the overall semantic meaning in the sentence. Such shortcut leads to better accuracy on the HWU64 dataset since most of its intents contain only verbs and object nouns, however it could lead to failure in Banking77 since intents in Banking77 need capture more fine-grained semantic information.
3) Uninformative support examples make all methods fail. For the last instance, the support example for meaning_of_life (asking what is the meaning of life) is not a usual expression for that intent, and all methods fail by predicting a label with more content similarity.
5 An Easy Modification
5.1 Label names as support
Inspired by previous analysis revealing the need for sentence encoders to capture more of the key phrases, we propose the following frustratingly simple modification: adding the label names as instances into the support set. Denoting the label name of an intent category  $c$  as  $l_{c}$ , then the prototype of class  $c$  after adding the label name becomes
$$
e _ {c} = \frac {1}{K + 1} (E _ {s} (l _ {c}) + \sum_ {(x _ {i}, y _ {i}) \in \mathcal {S}: y _ {i} = c} E _ {s} (x _ {i})).
$$
Note that the label names are always available for free at both meta-training and meta-testing phases. The label name can be seen as a discriminative and representative example for the corresponding intent. It is discriminative in the sense that adding label names to the support set is equivalent to putting more weights on words similar to the intent phrase when calculating the prototypes since words in the intent label usually are key words. For the first example in Table 4, adding the label name create_or_addlists and removeLists could make the model paying
more attention to discriminate the act of “add” and “delete”. The label name is also representative in that it sometimes could directly convey a relatively abstract concept. For the third example in Table 4, the label name meaning_of_life is more informative than the support set examples.
From Table 5, we can observe that the label name support has consistently and substantially improved all methods. Specifically, in the 5-way 1-shot setting, the results improve about  $3\%$  to  $11\%$  on Banking77,  $6\%$  to  $10\%$  on HWU64,  $1\%$  to  $8\%$  on Clinc150. Moreover, adding the label name as support could also improve the state-of-the-art ProtAugment framework, as shown in the results of L-ProtoNet+MLM and L-ProtAugment.
Meaningful labels improve baselines. As shown in Table 6, for the first example, adding label names correct the prediction of sentence embedding baselines. Adding share_location as a support example, the prototype for this class may contain more information about "know" and "location" compared to the irrelevant word "miranda". This illustrates the discriminative effectiveness of label names. For the second example, all methods incorrectly predict removeLists because the bad example given in the support set for the label create_or_addLists. Adding the label names corrects all methods with better representation for the class create_or_addLists.
Limitation: negative effect from misleading or vague labels. We also observe slightly negative effects in some cases. For example, when adding the label name general_negate (this intent means a person does not agree with something) for the third example in Table 6, "negate" is not a usual expression for the intent general_negate, and the association between "not" in the query and
Episode	ProtAugment	SBERT-Para.
w/o L	w/ L	w/o L	w/ L
Query:	[spelling]i need to know how to spell "miranda"
Support #1:	[spelling]i can't figure out how to spell superficial	✓	✓	×	✓
Support #2:	[share_location]have miranda know about my current location
Query:	[create_or_addlists]i need to create a new to do list
Support #1:	[create_or_addlists]we need milk	×	✓	×	✓
Support #2:	[removeLists]delete butterfly clips from my wish list
Query:	[general_negate]sorry but that is not the right answer.
Support #1:	[general_negate]please rectify the command.	✓	×	✓	×
Support #2:	[general_don't_care]any one would be okay with me, olly.
Table 6: Case study on HWU64, Clinic150 datasets after adding label names. [Intent]: the true intent label;  $\checkmark$  : correct predictions;  $X$  : wrong predictions;  $w/o$  L: without label name;  $w/$  L: after adding label name.
the word "don't" in the label name general don't_care makes the model prefer this wrong intent. We also find some label names in the Liu57 dataset to be rather vague or ambiguous, making them difficult to bring any useful information. For example, the label name likeness actually corresponds to utterances expressing the likeness to music, while the label name music means listening to or playing music. Adding these label names usually leads to confusion between these two intents.
6.1 Few-Shot Intent Detection
Studies on few-shot intent detection usually focus on two settings: (1) only a handful of annotated examples for each intent are available during training (Casanueva et al., 2020; Mehri and Eric, 2021; Zhang et al., 2020, 2021b; Qu et al., 2021); (2) in addition to the few-shot examples of target intents, rich labeled examples of other intents are available for training (Xia et al., 2021, 2020; Nguyen et al., 2020; Li and Zhang, 2021; Dopierre et al., 2021b; Yu et al., 2021; Zhang et al., 2021a). In this paper, we focus on the second setting, where the problem is typically formulated as the meta-learning problem and various approaches have been proposed (Yu et al., 2018; Geng et al., 2019; Nguyen et al., 2020; Li and Zhang, 2021; Dopierre et al., 2021b). Dopierre et al. (2021b) propose to use diverse paraphrases to improve the ProtoNet and achieve the SoTA performance in the semi-supervised intent detection meta-learning setting. Zhang et al. (2021a) study the effectiveness of pre-training with labeled intent data for this problem. Instead of developing a new framework or a new algorithm, this work conducts an empirical study on the effectiveness of using general pre-trained sentence encodings for
this task.
6.2 Label Semantics for Low-Resource Text Classification
There also exists more complex usage of label names in the related literature on zero-shot or few-shot text classification tasks (Yazdani and Henderson, 2015; Chen et al., 2016; Wang et al., 2018; Yan et al., 2020; Luo et al., 2021; Hou et al., 2021), typically involving learnable label embeddings with crafted encoder modules or a more clever usage of pre-trained language models.
In zero-shot text classification, Chang et al. (2008) embed label descriptions and texts into a shared Wikipedia concept space and then measure their similarities for classification. Similarly, Chen et al. (2016) jointly embed label names and utterances into one distributed embedding space. Yazdani and Henderson (2015) leverage the structure of intent labels to produce a classification hyperplane for zero-shot intent classification.
For few-shot text classification, prompt-based methods (Schick and Schütze, 2021a,b) use label names to construct verbalizers to map each label into cloze-style phrases, which enables the utilization of powerful pretrained language models. More recently, Müller et al. (2022) propose label tuning which only finetunes the label embeddings but freezes the sentence encoder for few-shot text classification. Mueller et al. (2022) explore label semantics by performing pre-training on a mix of gold and weakly annotated sentence-label pairs. Another line of works (Luo et al., 2021; Hou et al., 2021) incorporate label names into the meta-learning models. Luo et al. (2021) append label names to the utterances to help the model extract discriminative features. Hou et al. (2021) take a linear combination of the prototype calculated
by support examples and the label embedding as the anchored label representation to separate different labels in a multi-label intent classification scenario. Different from previous works, our simple modification aims to enhance general sentence embeddings towards key parts that express an intent. Besides, the proposed modification doesn't introduce any parameters, so it can be adapted to any pretrained sentence encoders with ease.
Motivated by the nature of prototypical networks for intent detection meta-learning, in this paper, we empirically compare some modern popular sentence encoders on multiple intent detection benchmarks, observing non-trivially strong performance with better cross-dataset generalization capability than the fine-tuned sentence encoders. Inspired by our follow-up analysis, we propose a simple modification that has consistently and substantially brought performance gain over all systems: adding the intent label names into the support set. This strategy not only improves over the performance from general-purpose sentence encoding, but also the state-of-the-art results from the fine-tuned ProtAugment framework.
One limitation of our study for now is that the sentence representation in use is mostly based on BERT variants. It could be technically interesting to experiment with models pre-trained in a sequence-to-sequence fashion (Lewis et al., 2020; Raffel et al., 2020), which might have better captured semantic paraphrase representations via denoising or other training objectives.
We would like to thank Jin-Ge Yao for helpful discussion and valuable feedback on this study. We thank all the anonymous reviewers for their suggestions to improve the initial draft.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics.
Inigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient
intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 38-45, Online. Association for Computational Linguistics.
Ming-Wei Chang, Lev Ratinov, Dan Roth, and Vivek Srikumar. 2008. Importance of semantic representation: Dataless classification. In AAAI'08.
Mingyang Chen, Wen Zhang, Wei Zhang, Qiang Chen, and Huajun Chen. 2019. Meta relational learning for few-shot link prediction in knowledge graphs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pages 4217-4226. Association for Computational Linguistics.
Yun-Nung Chen, Dilek Hakkani-Tür, and Xiaodong He. 2016. Zero-shot learning of intent embeddings for expansion by convolutional deep structured semantic models. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016, Shanghai, China, March 20-25, 2016, pages 6045-6049. IEEE.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Thomas Dopierre, Christophe Gravier, and Wilfried Logerais. 2021a. A neural few-shot text classification reality check. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 935-943, Online. Association for Computational Linguistics.
Thomas Dopierre, Christophe Gravier, and Wilfried Logerais. 2021b. ProtAugment: Intent detection meta-learning through unsupervised diverse paraphrasing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2454-2466, Online. Association for Computational Linguistics.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Empirical Methods in Natural Language Processing (EMNLP).
Ruiying Geng, Binhua Li, Yongbin Li, Xiaodan Zhu, Ping Jian, and Jian Sun. 2019. Induction networks for few-shot text classification. In Empirical Methods in Natural Language Processing (EMNLP), pages 3904-3913, Hong Kong, China. Association for Computational Linguistics.
John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. 2021. DeCLUTR: Deep contrastive learning for unsupervised textual representations. In Proceedings
of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 879-895, Online. Association for Computational Linguistics.
Aaron Gokaslan and Vanya Cohen. 2019. Openwebtext corpus.
Jiatao Gu, Yong Wang, Yun Chen, Victor O. K. Li, and Kyunghyun Cho. 2018. Meta-learning for low-resource neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3622-3631, Brussels, Belgium. Association for Computational Linguistics.
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. FewRel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In Empirical Methods in Natural Language Processing (EMNLP), pages 4803-4809, Brussels, Belgium. Association for Computational Linguistics.
Yutai Hou, Wanxiang Che, Yongkui Lai, Zhihan Zhou, Yijia Liu, Han Liu, and Ting Liu. 2020. Few-shot slot tagging with collapsed dependency transfer and label-enhanced task-adaptive projection network. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1381-1393, Online. Association for Computational Linguistics.
Yutai Hou, Yongkui Lai, Yushan Wu, Wanxiang Che, and Ting Liu. 2021. Few-shot learning for multi-label intent detection. In Thirty-Fifth AAAI Conference on Artificial Intelligence, pages 13036-13044. AAAI Press.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of-scope prediction. In Empirical Methods in Natural Language Processing (EMNLP), pages 1311-1316. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy,
Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Yue Li and Jiong Zhang. 2021. Semi-supervised meta-learning for cross-domain few-shot intent classification. In Proceedings of the 1st Workshop on Meta Learning and Its Applications to Natural Language Processing, pages 67-75, Online. Association for Computational Linguistics.
Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. 2019a. Benchmarking natural language understanding services for building conversational agents. In IWSDS.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized BERT pretraining approach. ArXiv, abs/1907.11692.
Qiaoyang Luo, Lingqiao Liu, Yuhao Lin, and Wei Zhang. 2021. Don't miss the labels: Label-semantic augmented meta-learner for few-shot text classification. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2773-2782, Online. Association for Computational Linguistics.
Shikib Mehri and Mihail Eric. 2021. Example-driven intent prediction with observers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2979-2992, Online. Association for Computational Linguistics.
Aaron Mueller, Jason Krone, Salvatore Romeo, Saab Mansour, Elman Mansimov, Yi Zhang, and Dan Roth. 2022. Label semantic aware pre-training for few-shot text classification. In ACL 2022, Online. Association for Computational Linguistics.
Thomas Müller, Guillermo Pérez-Torró, and Marc Franco-Salvador. 2022. Few-shot learning with siamese networks and label tuning. In ACL 2022, Online. Association for Computational Linguistics.
Hoang Nguyen, Chenwei Zhang, Congying Xia, and Philip Yu. 2020. Dynamic semantic matching and aggregation network for few-shot intent detection. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 1209-1218, Online. Association for Computational Linguistics.
Jin Qu, Kazuma Hashimoto, Wenhao Liu, Caiming Xiong, and Yingbo Zhou. 2021. Few-shot intent classification by gauging entailment relationship between utterance and semantic label. In Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI, pages 8-15, Online. Association for Computational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108.
Timo Schick and Hinrich Schütze. 2021a. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255–269, Online. Association for Computational Linguistics.
Timo Schick and Hinrich Schütze. 2021b. It's not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339-2352, Online. Association for Computational Linguistics.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 815-823.
Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, volume 30, pages 4080-4090. Curran Associates, Inc.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86):2579-2605.
Guoyin Wang, Chunyuan Li, Wenlin Wang, Yizhe Zhang, Dinghan Shen, Xinyuan Zhang, Ricardo Henao, and Lawrence Carin. 2018. Joint embedding of words and labels for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 2321-2331, Melbourne, Australia. Association for Computational Linguistics.