SlowGuess/NAACL_2021_OCR huggingface dataset:
support_set=all	support_set=balance	support_set=tgt
tgt	src	avg	tgt	src	avg	tgt
Initial BERT
Protofrz	14.07	25.02	21.37	-	-	-	-
Retrieverfrz	8.24	54.76	39.25	22.09	25.29	24.22	37.93
Pre-train on src domain
BERT fine-tune	-	96.51	-	-	-	-	-
Proto	75.02	95.73	88.83	-	-	-	-
Retriever	62.69	97.08	85.62	75.93	95.44	88.94	88.53
Retriever min-max	66.00	96.64	86.43	71.82	95.14	87.37	86.38
Fine-tune on tgt domain
BERT fine-tune	78.89	43.91	55.57	-	-	-	-
Proto	80.44	95.57	90.53	-	-	-	90.35
Retriever	66.76	96.95	86.89	79.20	95.50	90.07	91.16
Retriever min-max	67.64	96.84	87.11	77.60	95.35	89.43	89.56
Fine-tune on tgt domain with src data
BERT fine-tune	72.00	95.18	87.45	-	-	-	-
Proto	83.33	94.82	90.99	-	-	-	90.22
Retriever	69.51	97.04	87.86	84.95	95.41	91.92	90.78
Retriever min-max	71.35	96.96	88.42	81.00	94.55	90.03	89.82
Table 1: Intent accuracy on CLINC for  $n_c = 10, n_i = 10$  with 5-shots. Our retrieval-based method outperforms BERT fine-tune and Prototypical Networks in both target and source domains. We report results for our method when the support set consists of all examples in the source and target domains (all), when the support set consists of balanced few-shot number of examples for intents in both source and target domains (balance), and when the support set consists of examples of the target domain only (tgt) which serves as an upper-bound.
Evaluation We sample domains and intents three times for each  $(n_c, n_i)$  setting, and report average prediction accuracy. We report accuracy on intents from the target domain (tgt), source domain (src), and the macro average across all intents (avg).
In addition to applying the model to the target domain after pre-training on the source domain without re-training (Pre-train on src domain), we also evaluate the model performance with fine-tuning. We re-train the model with either target domain data only (Fine-tune on tgt domain) or a combination of source and target domain data (Fine-tune on tgt domain with src data).
Moreover, we evaluate the models with the following support set variations: with target domain data and all data in the source domain (support_set = all), with equal number of examples (same as the few-shot number) per intent (support_set = balance), and with only examples from the target domain (support_set = tgt). The last one serves as an upper-bound for the target domain accuracy.
Results Table 1 shows the results for  $(n_c, n_i) = (10, 10)$  and 5 examples per target intent; results on other settings exhibit the same patterns (See Appendix A.3). We observe that Retriever per
forms the best on the source domain (97.08%) before fine-tuning. Retriever also achieves the highest accuracy on the target domain (84.95%) after fine-tuning, while maintaining competitive performance on the source domain (95.41%) among all the methods.
6.2 Slot Filling
SNIPS (Coucke et al., 2018) is a slot filling dataset containing 39 slot names from 7 different domains: GetWeather (GW), PlayMusic (PM), AddToPlaylist (ATP), RateBook (RB), FindScreeningEvent (FSE), BookRestaurant (BR), and SearchCreativeWork (SCW). Following Hou et al. (2020), we train models on five source domains, use a sixth one for development, and test on the remaining domain. We directly use the  $K$ -shot split provided by Hou et al. (2020), where the support set consists of the minimum number of utterances such that at least  $K$  instances exist for each slot name. We also set  $K = 5$  in our experiment. Appendix A.2 contains further details about the setup.
We compare against two baselines and three models from the previous work. BERT Tagging is a BERT-based BIO tagging model (Devlin et al., 2019) fine-tuned on the testing domain
GW	PM	ATP	RB	FSE	BR	SCW	Average F1
Classification-based
BERT Tagging	59.41	42.00	46.07	20.74	28.20	67.75	58.61	46.11
Token-level
SimilarTokenfrz	53.46	54.13	42.81	75.54	57.10	55.30	32.38	52.96
MatchingToken	36.67	33.67	52.60	69.09	38.42	33.28	72.10	47.98
ProtoToken	67.82	55.99	46.02	72.17	73.59	60.18	66.89	63.24
L-TapNet+CDT+Proto	-	-	-	-	-	-	-	67.27
L-Proto+CDTpw*	74.68	56.73	52.20	78.79	80.61	69.59	67.46	68.58
L-TapNet+CDT+Protopw*	71.64	67.16	75.88	84.38	82.58	70.05	73.41	75.01
Span-level (ours)
Protofrz	39.47	38.35	47.68	69.36	38.60	42.39	19.90	42.25
Proto	64.47	53.97	54.64	73.37	42.89	62.48	27.76	54.23
Retrieverfrz	63.39	46.01	51.11	79.65	62.42	62.13	33.85	56.94
Retriever	82.95	61.74	71.75	81.65	73.10	79.54	51.35	71.72
Table 2: Results on SNIPS test data with 5-shot support sets. Our span-based retrieval model outperforms previous classification-based and token-level retrieval models even without label semantics. Classification-based and token-level results are reported in Hou et al. (2020). *Pair-wise embeddings (marked with  ${}^{\mathrm{{pw}}}$  ) are expensive at inference time,so we do not compare our method with these directly.
after training on the source domains, while SimilarToken $^{frz}$  uses BERT embeddings to retrieve the most similar token based on cosine similarity without any training. MatchingToken and ProtoToken are two token-level methods that leveraged Matching Networks (Vinyals et al., 2016) and Prototypical Networks (Snell et al., 2017) respectively. L-TapNet+CDT+proto (Hou et al., 2020) is an adaptation of TapNet (Yoon et al., 2019) with label semantics, CDT transition probabilities, and Prototypical Networks.
We experiment with several variants of our proposed method. Proto trains Prototypical Networks to compute span class representations. Retriever retrieves the most similar slot example for each span. Both methods use the same decoding method. Similar to SimilarToken $^{frz}$ , Proto $^{frz}$  and Retriever $^{frz}$  use the original BERT embeddings without any training. All models are trained on source domains and early stopped based on performance on the development domains.
Evaluation We report F1 scores for each testing domain in a cross-validation episodic fashion. Following Hou et al. (2020), we evaluate each testing domain by sampling 100 different support sets and ten exclusive query utterances for each support set. We calculate F1 scores for each episode and report average F1 scores across 100 episodes.
Results Table 2 summarizes the experiment results on the SNIPS dataset. Our span-level method
(Retrever) achieves higher averaged F1 than all five baselines, outperforming the strongest token-level method (L-TapNet+CDT+proto) by  $4.45\%$ . This shows that our model is effective at span-level predictions. More importantly, the better performance suggests that our span-level Retriever model is more efficient at capturing span structures compared to simulated dependencies as our method does not suffer from the potential discrepancy in the transition probabilities between the target and source domains.
Although Hou et al. (2020) showed that adding pairwise embeddings with cross-attention yielded much better performance, this method is expensive both in memory and computation at inference time, especially when the support set is large (Humeau et al., 2019). For fair comparison, we do not directly compare with methods using pairwise embeddings (methods with  $^{\text{pw}}$  in Table 2). Note that our method with pre-computed support example embeddings even outperforms L-Proto+CDT $^{\text{pw}}$  with less memory and computation cost.
7.1 Intent Classification
Models without re-training The pre-train on src domain section in Table 1 shows the results of models that are only pre-trained on the source domains but not fine-tuned on the target domains. Classification models such as BERT fine-tune cannot make predictions on target domains in this
setting. In contrast, even without seeing any target domain examples during training, retrieval-based models can still make predictions on new domains by simply including new examples in the support sets. With support_set=all, Retriever achieves  $97.08\%$  on the source domain while Proto performs worse than BERT fine-tune, consistent with previous findings (Triantafillou et al., 2019). Retriever achieves the best accuracy  $(75.93\%)$  on target domains with a balanced support set on all intents (support_set=balance). More importantly, Retriever also achieves competitive accuracy on source domains  $(95.44\%)$ , demonstrating that our proposed model achieves the best of both worlds even without re-training on new domains.
Varying the support set at inference time The construction of the support set is critical to retrieval-based methods. In Table 1, we present the model performances under different support settings (all, balance, tgt). The support_set = tgt setting serves as an upper bound for the target domain accuracy for both Retriever and Proto methods. In general, Retriever achieve the best performance on the source domain intents when we use full support sets (support_set = all). In comparison, if we use a balanced support set (support_set = balance), we can achieve much higher accuracy on the target domain while having a slight degradation on the source domain intents prediction. This is because full support sets have more source domain examples to increase confusion over target domains.
Data for fine-tuning The Fine-tune on tgt domain section in Table 1 shows different model behaviors when fine-tuned on the target domain data directly. While BERT fine-tune achieves high accuracy (78.89%) on the target domain, it suffers from catastrophic forgetting on the source domain (43.91%). On the other hand, Proto and Retriever can get high accuracy on the target domain (80.44% and 79.20%) while maintaining high performance on the source domain.
When we combine data from the source domain, we observe performance gains in all the models under the Fine-tune on tgt domain with src data section. Specifically, we add few-shot source domain examples as contrastive examples for the models to learn better utterance/class representations for Retriever and Proto. Results show that accuracy on the target domain increases by over  $3\%$  compared to only using target domain data. This
tgt	src	avg
BERT fine-tune	-	-	-
Proto	+12.89	-0.51	+5.18
Retriever	+14.60	-0.14	+6.11
Retriever min-max	+10.79	-0.20	+4.47
Table 3: Improvement  $(\%)$  over BERT fine-tune on target (tgt), source (src), and average (avg), after fine-tuning on the 5-shot support sets. Numbers are averaged over different  $(n_c, n_i)$  data samples.
suggests that unlike other retrieval-based methods such as kNN, Retriever does not require a large support set to guarantee prediction accuracy.
Impact of reduction functions We compare the reduction functions proposed in Section 5.1 and found that max performs the best. Since mean is equivalent to Prototypical Networks, we compared to Proto directly in the experiments. min-max is more intuitive in contrasting with least similar examples within the same class compared to max. However, its performance is worse than max. We speculate the reason to be that we retrieve the example with the maximum score at inference time so that the boundary margin may not be utilized.
Performance over different settings Table 3 shows the average improvement of our methods over the BERT fine-tune baseline, where all models are fine-tuned on the target domain with a balanced few-shot dataset after training on the source domain (same as Fine-tune on tgt domain with src data section in Table 1). Both Proto and Retriever outperforms the baseline on the target domains with a large margin, and Retriever has the best improvement on all intents on average.
7.2 Slot Filling
We note that Retriever outperforms the strongest baselines but reaches a low score on the SCW domain. This may be due to the bigger difference between the test (SCW) and the development domain (GW) including the size of the support set and their respective slot names. We also found that from all the correctly predicted slot spans,  $96.73\%$  predicted the correct slot names. This shows that the majority of the errors come from querying with invalid spans. We believe that span-based pretraining such as Span-BERT (Joshi et al., 2020) could make our proposed method achieve better results.
beam size	merge threshold	avg. F1
1	0.99	70.10
5	0.99	71.47
10	0.99	71.72
10	0	70.43
10	1	72.10
Table 4: Ablation study on beam size and merge condition. Merge threshold of 0 means never merge and 1 means always merge. Using larger beam and merging consecutive spans improve F1 scores.
Analyzing Proto From Table 2, Retriever outperforms Proto by  $17\%$  when training the span representations. We conjecture that this is caused by Proto learning noisy prototype. Compared to Retriever, the similarity scores between the spans and their corresponding class representations are low, indicating that the span-level prototypes may not be clearly separated.
Ablation on decoding method Table 4 compares beam search to greedy search. Results suggest that beam search with larger beam sizes achieve better F1 scores. As discussed in Section 5.2, we merge same-label spans during inference based on a score threshold. As shown in Table 4, merging spans results in a  $1.67\%$  F1 gain  $(70.43\%)$  vs  $72.10\%$  ) under the same beam size.
Error Analysis We find that the main problem of our proposed model is that tokens surrounding the gold span may contain excessive contextual information so that these surrounding invalid spans retrieve corresponding spans with high similarities. For instance, in the query "add my track to old school metal playlist", the token "playlist" retrieves an actual playlist span with a high similarity score. Another major issue is that the similarity score retrieved by a partial of the gold span sometimes is higher than that retrieved by the whole span. Our ablation results on merge threshold shown in Table 4 also suggest that partial spans may retrieve complete spans individually so that if we merge consecutive spans with the same slot name, we can achieve higher F1 scores.
In this paper, we propose a retrieval-based method, Retriever, for few-shot intent classification and slot filling. We conduct extensive experiments to compare different model variants and baselines, and show that our proposed approach is effective
in the few-shot learning scenario. We believe that our method can also work on open domain dialog tasks where annotations may be more scarce and other text classification tasks. In the future, we plan to extend our method to predict more complex structures with span-based retrieval.
Our intended use case is few-shot domain adaption to new classes. Our experiments are done on English data, but the method is not English-specific. We use 8 Cloud TPUs V2 cores<sup>5</sup> for training and one V100 GPU for inference. Since our model does not have to be retrained for the new domains, it can reduce the resources needed when applying such systems. We claim that our proposed method outperforms baselines on few-shot slot filling and intent classification examples. Our experiments mainly focus on the 5-shot setting and the 2-shot setting, which are typical testing scenarios applied by previous work with the same claim.
We thank Terry Koo and Emily Pitler from Google Research, and anonymous reviewers for their constructive suggestions.
Harry Bunt. 2009. The dit++ taxonomy for functional dialogue markup. In AAMAS 2009 Workshop, Towards a Standard Markup Language for Embodied Dialogue Acts, pages 13-24.
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, Maël Primet, and Joseph Dureau. 2018. Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of
<sup>5</sup>https://cloud.google.com/tpu
deep networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of ICML'17, pages 1126-1135. JMLR.org.
G. D. Forney. 1973. The viterbi algorithm. Proc. of the IEEE, 61:268 - 278.
Alexander Fritzler, Varvara Logacheva, and Maksim Kretov. 2019. Few-shot classification in named entity recognition task. In Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing, SAC '19, page 993-1000, New York, NY, USA. Association for Computing Machinery.
Ruiying Geng, Binhua Li, Yongbin Li, Xiaodan Zhu, Ping Jian, and Jian Sun. 2019. Induction networks for few-shot text classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3904-3913, Hong Kong, China. Association for Computational Linguistics.
Daniel Gillick, Sayali Kulkarni, Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene Ie, and Diego Garcia-Olano. 2019. Learning dense representations for entity retrieval. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 528-537, Hong Kong, China. Association for Computational Linguistics.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Realm: Retrievalaugmented language model pre-training.
Yutai Hou, Wanxiang Che, Yongkui Lai, Zhihan Zhou, Yijia Liu, Han Liu, and Ting Liu. 2020. Few-shot slot tagging with collapsed dependency transfer and label-enhanced task-adaptive projection network. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1381-1393, Online. Association for Computational Linguistics.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. CoRR, abs/1508.01991.
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2019. Real-time inference in multi-sentence tasks with deep pretrained transformers. CoRR, abs/1905.01969.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations.
Gregory R. Koch, Richard Zemel, and Ruslan Salakhutdinov. 2015. Siamese neural networks for one-shot image recognition. In ICML Deep Learning workshop.
Jason Krone, Yi Zhang, and Mona Diab. 2020. Learning to classify intents and slot labels given a handful of examples. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 96-108, Online. Association for Computational Linguistics.
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1311-1316, Hong Kong, China. Association for Computational Linguistics.
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks.
Kai-Hui Liang, Austin Chau, Yu Li, Xueyuan Lu, Dian Yu, Mingyang Zhou, Ishan Jain, Sam Davidson, Josh Arnold, Minh Nguyen, and Zhou Yu. 2020. Gunrock 2.0: A User Adaptive Social Conversational System. *Alexa Prize Proceedings*.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. 2018. A simple neural attentive metal-learner. In International Conference on Learning Representations.
Hao Peng, Ankur Parikh, Manaal Faruqui, Bhuwan Dhingra, and Dipanjan Das. 2019. Text generation with exemplar-based adaptive decoding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2555-2565, Minneapolis, Minnesota. Association for Computational Linguistics.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
F. Ren and S. Xue. 2020. Intention detection based on siamese neural network with triplet loss. IEEE Access, 8:82242-82254.
Karsten Roth, Timo Milbich, Samarth Sinha, Prateek Gupta, Björn Ommer, and Joseph Paul Cohen. 2020. Revisiting training strategies and generalization performance in deep metric learning.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 4077-4087. Curran Associates, Inc.
Shengli Sun, Qingfeng Sun, Kevin Zhou, and Tengchao Lv. 2019. Hierarchical attention prototypical networks for few-shot text classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 476-485, Hong Kong, China. Association for Computational Linguistics.
Shubham Toshniwal, Haoyue Shi, Bowen Shi, Lingyu Gao, Karen Livescu, and Kevin Gimpel. 2020. A cross-task analysis of text span representations. In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 166-176, Online. Association for Computational Linguistics.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. 2019. Meta-dataset: A dataset of datasets for learning to learn from few examples. CoRR, abs/1903.03096.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, k—oray kavukcuoglu, and Daan Wierstra. 2016. Matching networks for one shot learning. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 3630-3638. Curran Associates, Inc.
Yan Wang, Wei-Lun Chao, Kilian Q. Weinberger, and Laurens van der Maaten. 2019. Simpleshot: Revisiting nearest-neighbor classification for few-shot learning.
Kilian Q Weinberger, John Blitzer, and Lawrence Saul. 2006. Distance metric learning for large margin nearest neighbor classification. In Advances in Neural Information Processing Systems, volume 18, pages 1473-1480. MIT Press.
Sam Wiseman and Karl Stratos. 2019. Label-agnostic sequence labeling by copying nearest neighbors. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages