SlowGuess/NAACL_2022_OCR huggingface dataset:
Huffpost Dataset Samples of the same class are more diverse and scattered on Huffpost. For instance, "green streets are healthy streets", "the real heroes of Pakistan" and "what next for Kurdistan ?" are from the same class:"WORLD NEWS". In this scenario, a single class-wise prototype is difficult
Methods	Huffpost
5-way	10-way	GFSL
Proto	51.57	36.74	16.47
Matching	49.77	34.28	14.18
Induction	44.69	29.35	10.40
Proto-HATT	51.23	36.65	16.06
MLMAN	52.76	38.22	16.78
MGIMN(ours)	54.98	40.12	19.61
Table 4: Experiment results of standard FSL and generalized FSL settings on Huffpost datasets, while the FSL setting is same with Table 2.
to represent the entire class semantic. Interestingly, our approach improves more significantly than other datasets,  $2.22\%$  of 5-way 5-shot standard FSL metric,  $1.9\%$  of 10-way 5-shot standard FSL metric and  $2.83\%$  of generalized FSL metric. In our approach, richer matching features gained through interacting from low level with multi-grained interaction, are effective on the dataset with diverse expressed samples.
To further validate the effect of different interaction levels and instance matching vector, we make some ablation studies on both the datasets Liu and Huffpost. The settings are totally same with the main experiments.
4.3.1 Different Interaction Levels
We respectively take out the single-level interaction layer and see how the specific alignment feature affects the performance. As shown in the Table 5, when taking out the specific interaction layer, the performance decreases in varying degrees, which explains that each alignment layer has positive effect on the performance and can complement each other. It is noted that class-level interaction layer has the greatest impact. The model can pay attention to the whole class context through class-level interaction, which makes the model encode more precise class semantic information. It is the key to judge the relationship between query and class.
4.3.2 Instance-wise Matching Vector
We remove all interaction layers in our model, named 'w/o instance & class & episode' in Table 5. Then it is the same as matching network except that the scalar matching score is replaced by instance-wise matching. We make comparison with matching network. As given in the Table 5,
2 and 4, it performs better than matching network, e.g. Liu dataset improves  $3.49\%$  of 10-way 5-shot FSL score and Huffpost dataset improves  $2.30\%$  of GFSL score. Unlike scalar comparison in matching network, our approach can make fine-grained instance vector comparisons in fine-grained feature dimensional level.
4.4 Number of Classes and Inference Speed
As shown in Table 6, the inference speed increases linearly with the increase of the number of classes, from  $315ms/Query$  to  $1630ms/Query$  when  $c$  increases from 50 to 318. It is challenging for deploying the model to the online application. To address the problem of inference speed, motivated by the idea of retrieval in traditional search system, we propose the retrieval-then-classify method(RTC for short). (1)Stage1-retrieval: We construct the class-wise representation by averaging the vector of each support instance, produced by MGIMN encoder and then calculate the similarity between query and class-wise representation. In our experiments, we retrieve top  $N = 10$  classes with  $K = 5$  instances per class. (2)Stage2-classify: Retrieved support instances by stage1 are classified by MGIMN proposed in this paper. The  $C$ -way 5-shot task is reduced to a 10-way 5-shot which can greatly save the inference time and computation cost.
In addition to the generalized FSL metric score, we also report the inference speed (processing time per query) to show the effectiveness of retrieval-then-classify. We can see that the inference speed of retrieval-then-classify is greatly increased by  $5 \times$  to  $23 \times$ , with a small amount of performance loss. At the same time, comparing with other retrieval methods (e.g. BM25 and original bert), our approach can further improve the performance
5.1 Few-shot Learning
Intuitively, the few-shot learning focus on learn a classifier only using a few labeled training examples, which is similar to human intelligence. Since the aim of few-shot learning is highly attractive, researchers have proposed various few-shot learning approaches.
Generally, the matching network encodes query text and all support instances independently (Vinyals et al., 2016), then computes the cosine similarity between query vector and each support
Methods	Liu	Huffpost
5-way	10-way	GFSL	5-way	10-way	GFSL
MGIMN(ours)	87.84	80.60	57.66	54.98	40.12	19.61
w/o episode	86.22	78.99	56.67	54.14	39.53	18.69
w/o class	84.56	76.89	54.62	54.09	39.10	17.53
w/o instance	87.74	79.93	57.39	53.65	38.86	18.67
w/o instance&class&episode	80.53	70.94	42.54	51.81	37.10	16.48
Table 5: Ablation study results on Liu and Huffpost datasets.
Methods	Liu(c=50)	OOS(c=150)	Amzn(c=318)
score	speed	score	speed	score	speed
MGIMN-overall	57.66	315	76.23	757	49.46	1630
RTC-BM25	54.97	55	74.80	56	44.76	58
RTC-oribert	52.93	60	70.55	65	31.09	70
RTC-mgimnbert	56.21	60	75.58	65	46.80	70
Table 6: The generalized FSL accuracy(%) and inference speed. Speed is reported by averaging for processing 100 queries and the value is the processing time per query(ms/query)
sample vector, and finally computes average of all scores for per class. The prototypical network basically encodes query text and support instances independently (Snell et al., 2017), then computes class-wise vector by averaging vectors of all support instances in each class, and finally computes euclidean distances as final scores. Sung et al. (2018) introduced relation network (Sung et al., 2018) that exploits a relation module to model relationship between query vector and class-wise vectors rather than leveraging the distance metric (e.g., euclidean and cosine). Geng et al. (2019) introduced the induction network and leverage the induction module that take the dynamic routing as a vital algorithm to induce and generalize class-wise representations. In the few-shot scenario, the model-agnostic manner usually viewed as the improved version of few-shot, and defined as MAML (Finn et al., 2017), which can be exploited in different fields MT (Gu et al., 2018; Li et al., 2020b), dialog generation (Qian and Yu, 2019; Huang et al., 2020).
For few-shot text classification, researchers have also proposed various techniques to improve the existing approaches for few-shot learning. Basically, the one of our strong baselines Proto-HATT is introduced by Gao et al. (2019), that leverages the attention with instance-level and feature-level then highlight both the vital features and support point. Ye and Ling (2019) also tries to encode both query and per support set by leveraging the interactive way at word level with taking the matching information into account.
5.2 Text Matching
Text matching model aims to predict the score of text pair dependent on massive labeled data. Before BERT, related work focuses on deriving the matching information between two text sequences based on the matching aggregation framework. It performs matching in low-level and aggregates matching results based on attention mechanism. Many studies are proposed to improve performance. The attend-compare-aggregate method (Parikh et al., 2016) which has an effectiveness on alignment, meanwhile aggregates the aligned feature by using feed-forward architecture. The previous work extracts fine-grained matching feature with bilateral matching operation by considering the multiperspective case(Wang et al., 2017). Tay et al. (2017) exploit the factorization layers to enhance the word representation via scalar features with an effective and strong compressed vectors for alignment. Yang et al. (2019) present a straightforward but efficient text matching model using strong alignment features. After the PTM (e.g., BERT) is presented (Devlin et al., 2019), it has became commonly used approach on the various fields of NLP. Thus, many text matching methods are also leveraging the PTM. For example, Reimers and Gurevych (2019) use the sentence embeddings of BERT to conduct text matching, and Gao et al. (2021) use contrastive learning to train text matching models. Additionally, to handle the issue of few-shot learning architecture, we employ the similar idea of
comparison, aggregation and introduce new architecture multi-grained interactive matching network.
In this research investigation, we introduce the Multi-grained Interactive Matching Network(MGIMN) for the text classification task with few-shot manner. Meanwhile, we introduce a two-stage method retrieval-then-classify (RTC) to solve the inference performance in realistic scenery. Experiment results illustrate that the presented method obtains the best result in all five different kinds of datasets with two evaluation metrics. Moreover, RTC method obviously make the inference speed getting faster. We will make further investigations on the the task of domain adaptation problem by extending our proposed method.
Trapit Bansal, Rishikesh Jha, Tsendsuren Munkhdalai, and Andrew McCallum. 2020. Self-supervised meta-learning for few-shot natural language classification tasks. arXiv preprint arXiv:2009.08445.
Yujia Bao, Menghua Wu, Shiyu Chang, and Regina Barzilay. 2019. Few-shot text classification with distributional signatures. arXiv preprint arXiv:1908.06039.
Jonathan Bragg, Arman Cohan, Kyle Lo, and Iz Beltagy. 2021. Flex: Unifying evaluation for few-shot nlp. arXiv preprint arXiv:2107.07170.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171-4186.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, pages 1126-1135.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2020. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723.
Tianyu Gao, Xu Han, Zhiyuan Liu, and Maosong Sun. 2019. Hybrid attention-based prototypical networks for noisy few-shot relation classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6407-6414.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821.
Ruiying Geng, Binhua Li, Yongbin Li, Xiaodan Zhu, Ping Jian, and Jian Sun. 2019. Induction networks for few-shot text classification. arXiv preprint arXiv:1902.10482.
Jiatao Gu, Yong Wang, Yun Chen, Victor O. K. Li, and Kyunghyun Cho. 2018. Meta-learning for low-resource neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3622-3631.
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. arXiv preprint arXiv:1810.10147.
Yi Huang, Junlan Feng, Shuo Ma, Xiaoyu Du, and Xiaoting Wu. 2020. Towards low-resource semi-supervised dialogue generation with meta-learning. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 4123-4128.
Mladen Karan and Jan Šnajder. 2016. Faqir-a frequently asked questions retrieval test collection. In International Conference on Text, Speech, and Dialogue, pages 74-81. Springer.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1746-1751.
Jason Krone, Yi Zhang, and Mona Diab. 2020. Learning to classify intents and slot labels given a handful of examples. arXiv preprint arXiv:2004.10793.
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1311-1316, Hong Kong, China. Association for Computational Linguistics.
Aoxue Li, Weiran Huang, Xu Lan, Jiashi Feng, Zhenguo Li, and Liwei Wang. 2020a. Boosting few-shot learning with adaptive margin loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12576-12584.
Rumeng Li, Xun Wang, and Hong Yu. 2020b. Metamt, a meta learning method leveraging multiple domain data for low resource machine translation. In The Thirty-Fourth AAAI Conference on Artificial Intelligence.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pretrain, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586.
Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. 2019. Benchmarking natural language understanding services for building conversational agents. arXiv preprint arXiv:1903.05566.
Qiaoyang Luo, Lingqiao Liu, Yuhao Lin, and Wei Zhang. 2021. Don't miss the labels: Label-semantic augmented meta-learner for few-shot text classification. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP* 2021, pages 2773-2782.
Rishabh Misra. 2018. Huffman news category dataset.
Hoang Nguyen, Chenwei Zhang, Congying Xia, and Philip S Yu. 2020. Dynamic semantic matching and aggregation network for few-shot intent detection. arXiv preprint arXiv:2010.02481.
Ankur P Parikh, Oscar Tackstrom, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. arXiv preprint arXiv:1606.01933.
Kun Qian and Zhou Yu. 2019. Domain adaptive dialog generation via meta learning. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 2639-2649.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pages 3982-3992.
Timo Schick and Hinrich Schütze. 2020. It's not just size that matters: Small language models are also few-shot learners. arXiv preprint arXiv:2009.07118.
Jake Snell, Kevin Swersky, and Richard S Zemel. 2017. Prototypical networks for few-shot learning. arXiv preprint arXiv:1703.05175.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. 2018. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1199-1208.
Yi Tay, Luu Anh Tuan, and Siu Cheung Hui. 2017. Compare, compress and propagate: Enhancing neural architectures with alignment factorization
for natural language inference. arXiv preprint arXiv:1801.00102.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. 2016. Matching networks for one shot learning. arXiv preprint arXiv:1606.04080.
Zhiguo Wang, Wael Hamza, and Radu Florian. 2017. Bilateral multi-perspective matching for natural language sentences. arXiv preprint arXiv:1702.03814.
Runqi Yang, Jianhai Zhang, Xing Gao, Feng Ji, and Haiqing Chen. 2019. Simple and effective text matching with richer alignment features. arXiv preprint arXiv:1908.00300.
Zhi-Xiu Ye and Zhen-Hua Ling. 2019. Multi-level matching and aggregation network for few-shot relation classification. arXiv preprint arXiv:1906.06678.
Wenpeng Yin. 2020. Meta-learning for few-shot natural language processing: A survey. arXiv preprint arXiv:2007.09604.
Kashnitsky Yury. 2020. Hierarchical text classification of amazon product reviews.
Peng Zhou, Zhenyu Qi, Suncong Zheng, Jiaming Xu, Hongyun Bao, and Bo Xu. 2016. Text classification improved by integrating bidirectional LSTM with two-dimensional max pooling. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3485-3495.