SlowGuess/NAACL_2021_OCR huggingface dataset:
C1n	C2n	C3n	C4n	C5n	C0
C1n	DNNC	55.50±2.27	72.29±0.20
ENTAILMENT	65.17±1.36	75.43±0.41
HYBRID	70.08±0.77	78.25±0.19
C2n	DNNC	64.58±0.42	77.75±1.08	61.72±0.90
ENTAILMENT	64.08±2.04	76.33±1.01	64.68±0.71
HYBRID	74.25±1.34	86.67±1.01	64.39±0.27
C3n	DNNC	65.25±1.67	79.58±1.50	64.67±1.93	50.25±0.52
ENTAILMENT	75.50±1.63	83.83±0.62	75.25±1.24	56.56±2.43
HYBRID	74.25±1.08	85.92±1.05	76.58±1.05	53.09±1.73
C4n	DNNC	66.75±0.54	79.08±0.51	60.50±2.35	62.25±1.08	42.56±0.76
ENTAILMENT	68.33±1.16	72.67±0.77	68.58±1.90	69.50±1.34	53.92±0.75
HYBRID	73.75±1.41	85.50±1.06	71.67±1.53	75.83±2.44	52.75±0.63
C5n	DNNC	65.33±0.62	76.75±1.59	62.83±3.17	59.75±2.83	57.25±2.32	36.66±1.07
ENTAILMENT	67.58±0.82	73.50±1.24	67.83±0.47	71.83±0.66	73.75±0.74	50.95±0.68
HYBRID	70.75±1.27	82.50±1.27	72.42±0.96	76.67±1.05	71.00±0.41	47.05±1.60
Table 2: System performance without base classes on the benchmark IFS-INTENT. Horizontal direction: different groups of testing classes (base classes  $C_b$ , five rounds of novel classes ( $C_n^1, \dots, C_n^5$ ) and the OOD classes  $C_o$ ); vertical direction: timeline of incremental learning over new rounds of novel classes. Numbers are averaged over results of three random seeds.
C1n	C2n	C3n	C4n	C5n	C0
C1n	DNNC	12.17±0.88	28.89±13.39
ENTAILMENT	67.17±1.20	82.03±6.36
C2n	DNNC	6.47±1.02	5.28±0.75	73.97±4.83
ENTAILMENT	51.67±5.02	53.00±3.01	81.61±4.71
C3n	DNNC	3.5±1.26	3.83±0.51	2.28±1.09	74.56±5.56
ENTAILMENT	52.83±0.66	36.50±6.82	56.33±3.50	44.06±20.17
C4n	DNNC	1.67±0.89	2.39±0.45	2.64±0.92	4.31±0.41	43.1±13.97
ENTAILMENT	40.58±3.71	42.17±5.87	47.17±7.74	34.92±4.09	78.57±2.15
C5n	DNNC	1.47±0.39	2.44±0.7	2.64±1.44	1.08±1.12	2.42±0.35	20.03±7.29
ENTAILMENT	32.08±7.00	34.75±2.16	37.67±7.29	24.58±2.63	22.50±3.18	22.29±14.49
Table 3: System performance without base classes on the benchmark IFS-RELATION.
Details for two datasets are reported in Table 1. For both benchmarks, we first split the classes into different rounds according to the setting illustrated in Table 1. Then we split the train/test examples provided by the original dataset into different rounds according to the split classes. For the new classes in each round, we randomly split 10 new classes into 5 groups (each with 2 classes) and intentionally let the 5 groups have different sizes of k-shot examples  $(k\in [1,5])$
Baselines. Since this is the first work that studies the incremental few-shot text classification problem, there is no prior system that deals with exactly the same task. In the setting without base classes, most few-shot learning models didn't work. We compare our proposed model ENTAILMENT with another work (Zhang et al., 2020) which also solves text classification as a textual entailment problem and use large-scale entailment datasets for pretraining. Together, their hybrid model, HYBRID, is also compared. In the setting with additional base
classes, we further compare two few-shot learning models (Snell et al., 2017; Gidaris and Komodakis, 2018) adapted from the computer vision field. For these two baselines, we replace their encoders with RoBERTa to fit into the text classification task.
- DNNC. Zhang et al. (2020) proposed a discriminate nearest neighbor classifier. They decide whether two utterances are in the same class or not and make predictions by assigning the label of the nearest neighbor among all the examples.
- Prototypical Network (Snell et al., 2017). We train the Prototypical Network on base classes with the episode training method. For each round  $C_n^i$ , representations for new classes are calculated as the average embedding of  $k$ -shot examples. Given a query example, the label is predicted with its nearest neighbor among all the class representations.
- DyFewShot (Gidaris and Komodakis, 2018). We introduced this baseline in Section 2. For this baseline, we extend this baseline to address multi-round few-shot classes: for the present round  $C_n^t$ ,
(a)IFS-INTENT.
(b) IFS-RELATION.
all the preceding classes, including that in  $C_b$  and  $\{C_n^1\cdots,C_n^{t - 1}\}$ , are viewed as "base classes".
Implementation and setting. For DNNC, EN-TAILMENT, and HYBRID, we use the MNLI (Williams et al., 2018) dataset to pre-train these models. All systems are implemented through the Huggingface Transformers package. For both pretraining and fine-tuning, we set the learning rate as 1e-6, the batch size is 16. We run 20 epochs for the pre-training. For the fine-tuning process, we run 5 epochs on IFS-INTENT and 50 epochs on IFS-RELATION. We run the same program with 3 different seeds and report the average performance. Accuracy is reported for  $\{C_b, C_n^1, \dots, C_n^5\}$  and F1 score for  $C_o$ .
As the problem formulation presented in Section 3, we want to investigate two questions.  $\mathcal{Q}_1$ : can our system get better performance on each round?  $\mathcal{Q}_2$ : can our system hold more stable performance during the incremental learning process? We answer these questions separately under the incremental learning setting with or without base classes.
Incremental learning without base classes. Tables  $2\sim 3$  list the results on two benchmarks, IFS-INTENT and IFS-RELATION, for the setting without base classes, respectively. For the IFS-INTENT benchmark, we compare ENTAILMENT with DNNC, together with their hybrid model HYBRID for 5 rounds. For the IFS-RELATION benchmark, we only compare ENTAILMENT with DNNC since HYBRID is not applicable for this dataset. The label (relation type) is not compatible with the input instance (an utterance
Figure 1: Average performance on new classes in different rounds. The x axis is the number of round and y is the average accuracy on new classes in this round.
(a) Average Performance.
Figure 2: Performance analysis on IFS-INTENT with base classes. In Figure (a), X axis is the number of round, where 0 indicates the round of base classes; Y axis is the average performance on all the seen classes in this round (including base classes  $C_b$  and new classes  $C_n^1$ , ...,  $C_n^i$ ). In Figure (b), X axis indicates a subset of classes, where 0 indicates base classes and 1 indicates new classes in  $C_n^1$ ; Y axis is the performance drop rate  $d$  for different subsets.
(b) Performance drop rate.
with an entity pair). Therefore, we can not mix the pairs from these two models (ENTAILMENT and DNNC) to train a hybrid model.
As for question  $\mathcal{Q}_1$ , we find that ENTAILMENT and HYBRID outperform all the baselines. These results show the effectiveness of formalizing text classification as a textual entailment problem. For the benchmark IFS-INTENT, the hybrid model, HYBRID, achieves the best performance since it has the largest number of entailment pairs (by combining pairs from two models) for the training. It shows in the extreme case that no base classes are available, the more data the better. For the benchmark IFS-RELATION, this task is much more difficult compared to intent detection due to the complicity of the training examples (utterances with entity pairs). DNNC does not perform well for this task since comparing two complex examples can not benefit from the pre-training entailment model.
As for  $\mathcal{Q}_2$ , we show the average performance change on new classes in Figure 1. For IFS-INTENT, the average performance of new classes increases in the beginning then drops for the remaining rounds. This might due to the lack of training data in the first found. For IFS-RELATION, the average performance drops dramatically due to this task is much more difficult.
Incremental learning with base classes. The results on IFS-INTENT with base classes are shown
\( C_b \)	\( C_n^1 \)	\( C_n^2 \)	\( C_n^3 \)	\( C_n^4 \)	\( C_n^5 \)	\( C_o \)
\( C_b \)	ProtoNet	87.25±0.10	53.4±10.68
DyFewShot	81.04±1.91	55.01±2.52
DNNC	95.96±0.68	61.89±4.78
ENTAILMENT	96.42±0.41	64.73±3.84
HYBRID	96.12±0.12	58.92±1.22
\( C_1^n \)	ProtoNet	85.83±1.94	31.67±1.48	43.66±3.08
DyFewShot	81.29±1.56	00.00±0.00	39.33±1.25
DNNC	95.75±0.41	74.83±1.64	64.54±2.02
ENTAILMENT	94.42±0.21	75.42±1.56	56.38±5.29
HYBRID	95.62±1.00	77.75±0.25	58.41±5.10
\( C_2^n \)	ProtoNet	83.92±0.33	24.92±5.54	38.83±3.43	31.14±9.83
DyFewShot	81.29±1.56	00.00±0.00	00.50±0.71	33.94±1.42
DNNC	95.42±0.62	72.92±4.37	75.08±3.30	49.02±3.23
ENTAILMENT	94.29±0.16	71.92±1.45	84.83±1.33	48.12±3.20
HYBRID	96.44±0.19	76.75±2.75	75.00±1.00	42.11±0.30
\( C_3^n \)	ProtoNet	81.08±2.06	24.33±5.54	30.67±6.17	22.50±1.34	23.62±6.99
DyFewShot	81.29±1.56	00.00±0.00	00.50±0.71	00.00±0.00	27.48±1.24
DNNC	95.67±0.33	68.17±2.37	66.33±5.02	71.25±3.78	45.69±1.73
ENTAILMENT	92.71±0.41	70.75±0.54	82.83±2.16	73.92±2.52	29.34±3.31
HYBRID	95.44±0.44	73.62±0.62	71.62±2.62	73.50±0.75	33.69±3.66
\( C_4^n \)	ProtoNet	81.17±2.52	17.83±2.58	31.75±0.94	24.92±1.90	22.25±3.19	28.19±4.78
DyFewShot	81.54±1.71	00.25±0.35	00.17±0.24	00.00±0.00	00.00±0.00	23.52±1.51
DNNC	95.29±0.16	68.75±2.35	66.75±3.82	67.00±3.40	57.75±1.41	42.09±3.72
ENTAILMENT	91.67±0.36	65.92±2.18	79.92±1.78	73.75±0.74	69.08±0.12	45.73±2.80
HYBRID	95.69±0.06	72.12±0.62	67.75±1.25	70.25±0.25	72.62±1.38	38.85±0.89
\( C_5^n \)	ProtoNet	80.00±2.65	21.83±5.45	29.17±3.70	24.67±3.12	23.17±3.60	30.33±4.17	29.24±2.96
DyFewShot	81.50±1.27	00.08±0.12	00.83±0.62	00.00±0.00	00.00±0.00	00.50±0.71	21.23±1.34
DNNC	95.12±0.47	67.50±0.89	67.92±4.70	64.42±4.17	52.42±1.20	53.33±2.09	30.46±5.92
ENTAILMENT	89.17±0.60	65.08±2.45	78.50±0.94	69.08±1.12	68.25±0.35	70.67±1.30	39.48±1.45
HYBRID	95.56±0.06	68.75±2.75	67.38±0.62	63.75±1.75	65.12±3.62	61.62±2.38	37.65±0.44
Table 4: System performance with base classes on the benchmark IFS-INTENT.
in Table 4. We compare our systems ENTAILMENT and HYBRID with three baselines: DNNC, ProtoNet, and DyFewShot. This setting is evaluated incrementally on base classes, five rounds of new classes, and OOD.
As for question  $\mathcal{Q}_1$ , we summarize our observations as follows. (i) Pre-trained models (ENTAILMENT, HYBRID, and DNNC) work much better than few-shot learning methods (ProtoNet and DyFewShot) which means pre-training from a large-scale entailment dataset helps a lot in this setting. (ii) Our proposed models, ENTAILMENT and HYBRID obtain comparable performances and they outperform all the other baselines consistently in all test classes for the whole timeline. This shows the effectiveness of our proposed method of generating (utterance, label) entailment pairs.
To answer  $\mathcal{Q}_2$  in this setting, we propose a new evaluation metric, performance drop rate  $d$ , to evaluate the performance change along the timeline, i.e., how fast the performance drops when adding new rounds of classes into the system. For example, the performances on base classes decrease when incrementally adding five rounds of new classes into the system. Given a list of performance results for a
certain subset of classes (for example, base classes) on  $m$  rounds,  $r = (r_1, r_2, \dots, r_m)$ , we calculate the performance drop rate as the average drop rate of different rounds  $d = \frac{1}{m - 1} \sum_{i=0}^{m - 1} (r_i - r_{i + 1}) / r_i$ . In the experiments, we calculate  $d$  for four methods on base classes, new classes in round1 and round2 separately. The average drop rate of DyFewShot is not reported since there are 0.0 values in the performance.
In Figure 2, we show the average performance on all the seen classes in different rounds (including base classes and all the seen new classes) in (a) and the performance drop rate  $d$  in (b). As shown in Figure 2 (a), the average performance drops with the increase of round numbers. We can also observe that our proposed models ENTAILMENT and HYBRID achieve the best performance on the average performance on all the seen classes, including base classes and new classes. Figure 2 (b) shows the performance drop rate  $d$  for different models. ProtoNet and ENTAILMENT have higher drop rate than DNNC and HYBRID on base classes. For new classes in round1 and round2, the drop rate on ProtoNet is much higher than all the entailment methods. In summary, ENTAILMENT achieves the
best performance on the average accuracy of all seen classes, while DNNC is more stable and has a lower performance drop rate. HYBRID combines the advantages of both models by combining these two models together.
In this work, we define a new challenge in the NLP domain, incremental few-shot text classification with multi-round new classes in two settings: with or without base classes. In addition to the problem formulation, we also release two benchmark datasets for this particular challenge: IFS-INTENT and IFS-RELATION. Two approaches, ENTAILMENT and HYBRID are proposed to solve this problem. They convert the text classification problem into textual entailment and make the maximum utilization of the pre-training textual entailment model. Extensive experiments are conducted and the results consistently show the effectiveness of our proposed models.
We thank the reviewers for their valuable comments. This work is supported in part by NSF under grants III-1763325, III-1909323, and SaTC-1930941.
Inigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. arXiv preprint arXiv:2003.04807.
Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing textual entailment: Models and applications. Synthesis Lectures on Human Language Technologies, 6(4):1-220.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Ruiying Geng, Binhua Li, Yongbin Li, Xiaodan Zhu, Ping Jian, and Jian Sun. 2019. Induction networks for few-shot text classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3895-3904.
Spyros Gidaris and Nikos Komodakis. 2018. Dynamic few-shot visual learning without forgetting. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pages 4367-4375.
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4803-4809.
Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid O Seaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenzo Romano, and Stan Szpakowicz. 2019. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. arXiv preprint arXiv:1911.10422.
Kamran Kowsari, Kiana Jafari Meimandi, Mojtaba Heidarysafa, Sanjana Mendu, Laura Barnes, and Donald Brown. 2019. Text classification algorithms: A survey. Information, 10(4):150.
Qian Li, Hao Peng, Jianxin Li, Congying Xia, Renyu Yang, Lichao Sun, Philip S. Yu, and Lifang He. 2020. A Survey on Text Classification: From Shallow to Deep Learning. arXiv preprint arXiv:2008.00364.
Bing Liu and Ian Lane. 2016. Attention-based recurrent neural network models for joint intent detection and slot filling. In INTERSPEECH, pages 685-689.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Hoang Nguyen, Chenwei Zhang, Congying Xia, and S Yu Philip. 2020. Semantic matching and aggregation network for few-shot intent detection. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 1209-1218.
Hang Qi, Matthew Brown, and David G. Lowe. 2018. Low-shot learning with imprinted weights. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pages 5822-5830.
Mengye Ren, Renjie Liao, Ethan Fetaya, and Richard S. Zemel. 2019. Incremental few-shot learning with attention attractor networks. In NeurIPS, pages 5276-5286.
Timo Schick and Hinrich Schütze. 2020. Exploiting cloze questions for few-shot text classification and natural language inference. CoRR, abs/2001.07676.
Jake Snell, Kevin Swersky, and Richard S. Zemel. 2017. Prototypical networks for few-shot learning. In NeurIPS, pages 4077-4087.
Denny Vrandecic and Markus Krötzsch. 2014. Wiki-data: a free collaborative knowledgebase. Communications of the ACM, 57(10):78-85.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122.
Congying Xia, Caiming Xiong, S Yu Philip, and Richard Socher. 2020a. Composed variational natural language generation for few-shot intents. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 3379-3388.
Congying Xia, Chenwei Zhang, Hoang Nguyen, Jiawei Zhang, and Philip Yu. 2020b. Cg-bert: Conditional text generation with bert for generalized few-shot intent detection. arXiv preprint arXiv:2004.01881.
Congying Xia, Chenwei Zhang, Xiaohui Yan, Yi Chang, and S Yu Philip. 2018. Zero-shot user intent detection via capsule neural networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3090-3099.
Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. In EMNLP-IJCNLP, pages 3912-3921.
Wenpeng Yin, Nazneen Fatema Rajani, Dragomir Radev, Richard Socher, and Caiming Xiong. 2020. Universal natural language processing with limited annotations: Try few-shot textual entailment as a start. arXiv preprint arXiv:2010.02584.
Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald Tesauro, Haoyu Wang, and Bowen Zhou. 2018. Diverse few-shot text classification with multiple metrics. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1206-1215.
Xiaodong Yu, Wenpeng Yin, and Dan Roth. 2020. Paired representation learning for event and entity coreference. CoRR, abs/2010.12808.
Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via convolutional deep neural network. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 2335-2344.
Jianguo Zhang, Kazuma Hashimoto, Wenhao Liu, Chien-Sheng Wu, Yao Wan, Philip Yu, Richard Socher, and Caiming Xiong. 2020. Discriminative nearest neighbor few-shot intent detection by transferring natural language inference. In EMNLP, pages 5064-5082.