SlowGuess/NAACL_2021_OCR huggingface dataset:
5363-5369, Florence, Italy. Association for Computational Linguistics.
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 808–819, Florence, Italy. Association for Computational Linguistics.
Yi Yang and Arzoo Katiyar. 2020. Simple and effective few-shot named entity recognition with structured nearest neighbor learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6365-6375, Online. Association for Computational Linguistics.
Yinfei Yang, Gustavo Hernandez Abrego, Steve Yuan, Mandy Guo, Qinlan Shen, Daniel Cer, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2019. Improving multilingual sentence embedding using bidirectional dual encoder with additive margin softmax. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 5370-5378. International Joint Conferences on Artificial Intelligence Organization.
Sung Whan Yoon, Jun Seo, and Jaekyun Moon. 2019. TapNet: Neural network augmented with task-adaptive projection for few-shot learning. In Proceedings of Machine Learning Research, volume 97 of Proceedings of Machine Learning Research, pages 7115-7123, Long Beach, California, USA. PMLR.
Dian Yu, Michelle Cohn, Yi Mang Yang, Chun Yen Chen, Weiming Wen, Jiaping Zhang, Mingyang Zhou, Kevin Jesse, Austin Chau, Antara Bhowmick, Shreenath Iyer, Giritheja Sreenivasulu, Sam Davidson, Ashwin Bhandare, and Zhou Yu. 2019. Gunrock: A social bot for complex and engaging long conversations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, pages 79-84, Hong Kong, China. Association for Computational Linguistics.
Dian Yu and Zhou Yu. 2019. Midas: A dialog act annotation scheme for open domain human machine spoken conversations.
Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald Tesauro, Haoyu Wang, and Bowen Zhou. 2018. Diverse few-shot text classification with multiple metrics. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1206-1215, New Orleans, Louisiana. Association for Computational Linguistics.
Jingyi Zhang, Masao Utiyama, Eiichro Sumita, Graham Neubig, and Satoshi Nakamura. 2018. Guiding neural machine translation with retrieved translation pieces. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1325-1335, New Orleans, Louisiana. Association for Computational Linguistics.
Morteza Ziyadi, Yuting Sun, Abhishek Goswami, Jade Huang, and Weizhu Chen. 2020. Example-based named entity recognition.
We use the public uncased BERT-base model from https://github.com/google-research/bert for embedding spans. Our implementation is adapted from https://github.com/google-research/bert/blob/master/run-classifier.py. Since the span embeder in the retriever is the only trainable component in our model, the number of parameters is the same as the initial BERT model.
On SNIPS, we set the initial learning rate to be  $2 \times 10^{-5}$  with  $10\%$  data for warmup. We set per-class batch size to be 5 for 5-shot experiments. We use F1 score on the development domain as the metric for early stopping. For decoding, we set  $m = 7$  to be the maximum span length and  $\lambda = 0.99$  as the merging threshold. For dynamic threshold, we decrease the threshold by 0.05 each time for 10 times until at least one span is above the current threshold. We also use the development domain results to choose individual threshold for each target domain to filter invalid spans. We use grid search between [0.85, 0.97] with a step of 0.05 to search for the best threshold on the development domain. Our span-level evaluation is modified from conlleval script: https://www.clips.uantwerpen.be/conll2000/chunking/conlleval.txt.
On CLINC, we set the initial learning rate to be  $5 \times 10^{-5}$  and  $1 \times 10^{-5}$  for fine-tuning on the target domain. We set per-class batch size to be 8 for training on the source domain, and 5 and 2 for 5-shot and 2-shot fine-tuning.
A.2 SNIPS Data Details
Test Domain	Dev. Domain	Avg. |S|
GetWeather	PlayMusic	28.91
PlayMusic	AddTo Playlist	34.43
AddTo Playlist	RateBook	13.84
RateBook	FindScreeningEvent	19.83
FindScreeningEvent	BookRestaurant	19.27
BookRestaurant	SearchCreativeWork	41.58
SearchCreativeWork	GetWeather	5.28
Table 5: Corresponding development domain and average support set size for each testing domain.
A.3 CLINC Results on Different Data Constructions
Table 6: Intent accuracy on CLINC for  ${n}_{c} = {10},{n}_{i} = {10}$  with 5-shots.
support_set=all	support_set=balance	support_set=tgt
tgt	src	avg	tgt	src	avg	tgt
Initial BERT
Protofrz	8.70	24.06	18.94	-	-	-	-
Retrieverfrz	3.91	54.96	37.94	17.38	16.83	17.01	27.96
Pre-train on src domain
BERT fine-tune	-	96.51	-	-	-	-	-
Proto	76.40	95.70	89.27	-	-	-	-
Retriever	53.73	97.02	82.59	73.13	94.32	87.26	86.47
Retriever min-max	53.47	96.87	82.40	68.47	95.29	86.35	81.76
Fine-tune on tgt domain
BERT fine-tune	75.57	50.91	59.13	-	-	-	-
Proto	76.36	95.06	88.82	-	-	-	86.67
Retriever	55.20	97.05	83.10	74.89	94.57	88.01	87.74
Retriever min-max	55.96	96.92	83.27	71.09	95.25	87.19	83.91
Fine-tune on tgt domain with src data
BERT fine-tune	64.97	95.15	85.09	-	-	-	-
Proto	77.02	95.29	89.20	-	-	-	86.31
Retriever	56.36	97.17	83.56	76.87	94.11	88.36	88.18
Retriever min-max	58.82	96.85	84.17	74.31	94.32	87.56	83.98
Table 7: Intent accuracy on CLINC for  ${n}_{c} = {10},{n}_{i} = {10}$  with 2-shots.
support_set=all	support_set=balance	support_set=tgt
tgt	src	avg	tgt	src	avg	tgt
Initial BERT
Protofrz	12.57	21.67	17.42	-	-	-	-
Retrieverfrz	9.46	56.96	34.79	23.37	25.87	24.70	32.72
Pre-train on src domain
BERT fine-tune	-	96.92	-	-	-	-	-
Proto	73.17	94.96	84.79	-	-	-	-
Retriever	66.56	96.76	82.67	76.56	94.99	86.39	83.72
Retriever min-max	66.08	96.64	82.38	73.14	95.13	84.87	79.52
Fine-tune on tgt domain
BERT fine-tune	76.92	44.17	59.45	-	-	-	-
Proto	77.53	94.88	86.78	-	-	-	85.21
Retriever	68.35	97.11	83.69	77.54	95.67	87.21	85.72
Retriever min-max	70.65	96.76	84.58	76.74	94.90	86.43	85.41
Fine-tune on tgt domain with src data
BERT fine-tune	70.48	95.11	83.61	-	-	-	-
Proto	79.89	94.32	87.59	-	-	-	85.41
Retriever	71.46	96.88	85.02	79.90	94.61	87.75	89.67
Retriever min-max	71.73	96.86	85.13	78.43	95.31	87.43	85.32
Table 8: Intent accuracy on CLINC for  ${n}_{c} = 8,{n}_{i} = {10}$  with 5-shots.
support_set=all	support_set=balance	support_set=tgt
tgt	src	avg	tgt	src	avg	tgt
Initial BERT
Protofrz	7.24	20.99	14.57	-	-	-	-
Retrieverfrz	4.76	57.24	32.75	17.64	18.29	17.99	24.17
Pre-train on src domain
BERT fine-tune	-	96.92	-	-	-	-	-
Proto	70.59	94.42	83.30	-	-	-	-
Retriever	54.46	97.22	77.27	68.06	93.90	81.84	78.89
Retriever min-max	56.36	96.79	77.92	65.43	94.46	80.91	74.19
Fine-tune on tgt domain
BERT fine-tune	68.94	56.41	62.25	-	-	-	-
Proto	73.19	94.70	84.66	-	-	-	79.83
Retriever	57.58	97.01	78.61	70.98	93.56	83.02	80.35
Retriever min-max	58.71	96.89	79.07	68.71	94.64	82.54	76.51
Fine-tune on tgt domain with src data
BERT fine-tune	61.53	95.10	79.43	-	-	-	-
Proto	73.57	94.59	84.78	-	-	-	80.08
Retriever	59.06	97.07	79.33	72.33	93.11	83.42	80.14
Retriever min-max	60.49	96.89	79.91	70.92	94.68	83.59	79.21
Table 9: Intent accuracy on CLINC for  ${n}_{c} = 8,{n}_{i} = {10}$  with 2-shots.
support_set=all	support_set=balance	support_set=tgt
tgt	src	avg	tgt	src	avg	tgt
Initial BERT
Protofrz	15.02	17.58	16.30	-	-	-	-
Retrieverfrz	10.65	55.97	33.31	21.16	25.85	23.51	26.71
Pre-train on src domain
BERT fine-tune	-	96.87	-	-	-	-	-
Proto	67.05	96.37	81.71	-	-	-	-
Retriever	64.19	97.18	80.69	68.40	95.85	82.13	72.02
Retriever min-max	61.79	96.71	79.25	65.11	95.85	80.48	68.21
Fine-tune on tgt domain
BERT fine-tune	69.32	64.76	67.04	-	-	-	-
Proto	73.63	96.09	84.86	-	-	-	77.45
Retriever	68.65	97.05	82.85	74.59	96.24	85.42	77.96
Retriever min-max	67.94	97.04	82.49	71.88	96.27	84.07	75.93
Fine-tune on tgt domain with src data
BERT fine-tune	67.93	95.95	81.94	-	-	-	-
Proto	74.43	95.63	85.03	-	-	-	76.61
Retriever	71.51	96.91	84.21	76.37	95.81	86.09	78.62
Retriever min-max	70.50	96.92	83.71	73.76	95.79	84.78	76.65
Table 10: Intent accuracy on CLINC for  ${n}_{c} = 5,{n}_{i} = {15}$  with 5-shots.
support_set=all	support_set=balance	support_set=tgt
tgt	src	avg	tgt	src	avg	tgt
Initial BERT
Protofrz	9.39	16.19	12.79	-	-	-	-
Retrieverfrz	5.18	56.25	30.72	14.99	18.36	16.68	19.63
Pre-train on src domain
BERT fine-tune	-	96.87	-	-	-	-	-
Proto	63.50	95.72	79.61	-	-	-	-
Retriever	56.19	97.14	76.67	63.75	95.36	79.55	67.82
Retriever min-max	55.88	96.94	76.41	60.32	95.50	77.91	63.60
Fine-tune on tgt domain
BERT fine-tune	61.91	77.11	69.51	-	-	-	-
Proto	66.38	95.72	81.05	-	-	-	70.40
Retriever	58.83	97.01	77.92	66.90	95.30	81.10	71.26
Retriever min-max	57.97	97.07	77.52	63.23	95.78	79.50	66.44
Fine-tune on tgt domain with src data
BERT fine-tune	60.32	96.16	78.24	-	-	-	-
Proto	66.98	95.77	81.38	-	-	-	70.28
Retriever	59.88	96.96	78.42	67.23	94.85	81.04	70.52
Retriever min-max	60.08	97.08	78.58	65.45	95.73	80.59	68.06
Table 11: Intent accuracy on CLINC for  $n_c = 5, n_i = 15$  with 2-shots.