SlowGuess/NAACL_2022_OCR huggingface dataset:
John Wieting and Kevin Gimpel. 2018. ParaNMT-50M: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 451-462. Association for Computational Linguistics.
John Wieting, Kevin Gimpel, Graham Neubig, and Taylor Berg-Kirkpatrick. 2019. Simple and effective paraphrastic similarity from parallel translations. In Proceedings of the Association for Computational Linguistics.
John Wieting, Kevin Gimpel, Graham Neubig, and Taylor Berg-Kirkpatrick. 2021. Paraphrastic representations at scale. arXiv preprint arXiv:2104.15114.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics, pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Congying Xia, Caiming Xiong, and Philip Yu. 2021. Pseudo Siamese Network for Few-Shot Intent Generation, page 2005-2009. Association for Computing Machinery, New York, NY, USA.
Congying Xia, Caiming Xiong, Philip Yu, and Richard Socher. 2020. Composed variational natural language generation for few-shot intents. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 3379–3388, Online. Association for Computational Linguistics.
Guangfeng Yan, Lu Fan, Qimai Li, Han Liu, Xiaotong Zhang, Xiao-Ming Wu, and Albert Y.S. Lam. 2020. Unknown intent detection using Gaussian mixture model with an application to zero-shot intent classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1050-1060, Online. Association for Computational Linguistics.
Shuo Yang, Lu Liu, and Min Xu. 2021. Free lunch for few-shot learning: Distribution calibration. In International Conference on Learning Representations (ICLR).
Majid Yazdani and James Henderson. 2015. A model of zero-shot learning of spoken language understanding. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 244-249, Lisbon, Portugal. Association for Computational Linguistics.
Dian Yu, Luheng He, Yuan Zhang, Xinya Du, Panupong Pasupat, and Qi Li. 2021. Few-shot intent classification and slot filling with retrieved examples. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 734-749, Online. Association for Computational Linguistics.
Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald Tesauro, Haoyu Wang, and Bowen Zhou. 2018. Diverse few-shot text classification with multiple metrics. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1206-1215, New Orleans, Louisiana. Association for Computational Linguistics.
Haode Zhang, Yuwei Zhang, Li-Ming Zhan, Jiaxin Chen, Guangyuan Shi, Xiao-Ming Wu, and Albert Y.S. Lam. 2021a. Effectiveness of pre-training for few-shot intent classification. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1114-1120, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jianguo Zhang, Trung Bui, Seunghyun Yoon, Xiang Chen, Zhiwei Liu, Congying Xia, Quan Hung Tran, Walter Chang, and Philip Yu. 2021b. Few-shot intent detection via contrastive pre-training and fine-tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1906-1912, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jianguo Zhang, Kazuma Hashimoto, Wenhao Liu, Chien-Sheng Wu, Yao Wan, Philip Yu, Richard Socher, and Caiming Xiong. 2020. Discriminative nearest neighbor few-shot intent detection by transferring natural language inference. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5064-5082, Online. Association for Computational Linguistics.
Extended Analysis for Label Semantics Since we add label names as additional support examples, the embedding space of sentences does not change in general sentence embedding based methods. For meta-learning based methods, we can take label names as support examples only in the meta-testing phase, or leverage them in both meta-training and meta-testing stages. The latter will change the embedding space. As shown in Table A.1, we find that adding label names at both stages generally yields slightly better performance than only using it during meta-testing.
To further validate the effectiveness of using label names, we sample fifty 5-way 1-shot episodes from five classes in Liu57 dataset. We visualize the query instances and prototype representations in the same embedding space. As shown in Figure A.1, in the 1-shot setting, the prototypes of the same class spread out and it is difficult to distinguish them from other classes. This indicates that the model struggles to extract the crucial information relevant to the intent class due to the limited information contained in one support example. However, adding label names helps centralize the prototypes of the same class and separate them away from those of other classes. This suggests that label names may be regarded as high-quality free examples which could help distinguish different classes.
Implementation Details We use Euclidean distance as the distance metric in ProtoNet for all sentence encoders, except for SP-paraphrase, in which cosine distance leads to much better performance. For meta-learning based methods, we use ROBERTa-base from Huggingface (Wolf et al., 2020) as the encoder. For the optimizer, we use Adam (Kingma and Ba, 2015) with a learning rate of 2e-5. We set the batch size to 32 and train the model for 10,000 episodes. We use the validation split to evaluate the model every 600 episodes and select the checkpoint with the best performance. The ProtAugment model has 123M parameters and the training lasts around one hour on four Tesla V100 GPUs.
Method	Banking77	HWU64	Liu57	Clinic150
K=1	K=5	K=1	K=5	K=1	K=5	K=1	K=5
L†-ProtoNet+MLM	93.86±0.44	96.21±0.48	89.81±0.96	93.99±0.96	91.09±2.15	95.28±0.74	97.64±0.27	98.93±0.19
L†-ProtAugment	94.05±0.54	96.45±0.44	90.79±0.88	94.11±0.72	92.21±1.32	95.49±0.55	98.15 ±0.24	99.17±0.22
L-ProtoNet+MLM	94.00±0.66	95.97±0.58	91.30±1.72	94.09±0.43	92.00±1.42	95.26±0.67	98.36±0.22	99.08±0.21
L-ProtAugment	93.42±1.42	96.11±0.75	91.73±1.23	94.15±0.58	92.79±1.28	95.34±0.88	98.43±0.17	99.19±0.22
Table A.1: Performance comparison under the 5-way K-shot settings. L represents leveraging label names in both meta-training and meta-testing stages.  $\mathrm{L}^{\dagger }$  represents using label names during meta-testing only. The better performance between  ${\mathrm{L}}^{ \dagger  }$  and  $\mathrm{L}$  is underlined.
(a) SBERT-paraphrase
(b) L-SBERT-paraphrase
Figure A.1: T-SNE visualization of prototypes and query instances for SBERT-paraphrase and L-SBERT-paraphrase. We randomly select five classes from the Liu57 dataset for exemplification.