SlowGuess/NAACL_2021_OCR huggingface dataset:
Congying Xia $^{1*}$ , Wenpeng Yin $^{2*}$ , Yihao Feng $^{3}$ , Philip Yu $^{1}$
<sup>1</sup>University of Illinois at Chicago, Chicago, IL, USA
$^{2}$ Salesforce Research, Palo Alto, CA, USA
<sup>3</sup>University of Texas at Austin, Austin, TX, USA
{cxia8, psyu}@uic.edu,wyin@salesforce.com,yihao@cs.utexas.edu
Text classification is usually studied by labeling natural language texts with relevant categories from a predefined set. In the real world, new classes might keep challenging the existing system with limited labeled data. The system should be intelligent enough to recognize upcoming new classes with a few examples. In this work, we define a new task in the NLP domain, incremental few-shot text classification, where the system incrementally handles multiple rounds of new classes. For each round, there is a batch of new classes with a few labeled examples per class. Two major challenges exist in this new task: (i) For the learning process, the system should incrementally learn new classes round by round without re-training on the examples of preceding classes; (ii) For the performance, the system should perform well on new classes without much loss on preceding classes. In addition to formulating the new task, we also release two benchmark datasets  ${}^{1}$  in the incremental few-shot setting: intent classification and relation classification. Moreover, we propose two entailment approaches, ENTAILMENT and HYBRID, which show promise for solving this novel problem.
Text classification has achieved great success in the past decades with the development of deep learning techniques (Kowsari et al., 2019; Li et al., 2020). However, decent performance highly relies on the availability of large-scale task-specific training data. Recently, few-shot text classification (Yu et al., 2018; Geng et al., 2019; Xia et al., 2020a) has attracted increasing attention from the NLP community since it is unlikely to have large-scale labeled data for new classes in the real world.
Typically, few-shot text classification is formulated like this: the system first sees a set of base classes  $C_b$  that have a large number of labeled examples, then a group of new classes  $C_n$  is provided with  $k$  examples per class. For a testing instance, the system is required to search for its label in the space of  $C_b \cup C_n$  or merely  $C_n$ . However, this setting might not be suitable for real scenarios. First, base classes with rich annotations might not be available at the beginning. It happens whenever you want to build a system from scratch. Second, take the bank's customer service system as an example, queries with new intents are continuously appearing (e.g., by a sequence of rounds) without enough labeled data. The system should be able to keep learning and recognizing new intents round by round. For each query, the system needs to pick up the most appropriate intent in the incrementally increasing label space or return "none of them".
In this work, we propose a more realistic and challenging task in the low resource scenarios: incremental few-shot text classification. In this new task, the system is provided with  $m$  rounds of new classes (i.e.,  $C_n^1, C_n^2, \dots, C_n^m$ ) without any base classes that have enough annotations. For each round, there are a group of new classes,  $C_n^i$  ( $i = 1, \dots, m$ ), and each class has  $k$  labeled examples ( $k$  is in the range of [1, 5] and varies for different classes). During testing, the system is required to either select the best class from  $C_n^1 \cup C_n^1 \cdots \cup C_n^m$  or output "none of them" which means no existing class applies to the input. As far as we know, this is the first work that studies incremental few-shot learning without base classes. All previous few-shot learning models (Snell et al., 2017; Gidaris and Komodakis, 2018; Yin et al., 2020; Xia et al., 2020b; Nguyen et al., 2020) fail to solve this problem since they relied on the large-scale labeled data of base classes to train a robust system. To provide a complete vision about incremental few-shot text classification, we also conduct experiments with
*Indicates Equal Contribution.
$^{1}$ Code & Data are available at https://github.com/ congyingxia/IncrementalFSTC.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1351-1360 June 6-11, 2021. ©2021 Association for Computational Linguistics
additional base classes to compare with these baselines.
To evaluate the performance for different models, we build two benchmark datasets for this new problem. One is intent detection that aims at understanding the intents under user queries (Liu and Lane, 2016; Xia et al., 2018). This benchmark simulates a task like a bank's customer service as we mentioned. The other is relation classification which needs to determine the correct relation between two entities in a given sentence (Zeng et al., 2014). In reality, the relation types might be unlimited. For example, there are fine-grained relations or implicit relations that need entailment. In open-domain or open-form relation tasks, there always exists the problem of lack of annotations.
Another important feature of our benchmark datasets is that we do not provide dev sets. Existing systems are commonly evaluated on the dev set to choose the best training model. We claim that in real-world (incremental) few-shot applications, we cannot expect extra labeled data other than the  $k$  examples. This is in line with the observation in Schick and Schütze (2020). If a system has to rely on the dev set to find the best parameters, it is not suitable for the incremental few-shot setting.
Furthermore, we propose a novel approach, ENTAILMENT, to solve this new problem. ENTAILMENT models the text classification problem in a textual entailment (Dagan et al., 2013) framework. To figure out if an input  $x$  belongs to a class  $y$ , ENTAILMENT tries to infer the truth value of  $y$  (i.e., a hypothesis), given the  $x$  (i.e., the premise). The main benefit of this formulation is that the system learns this task not only from label-specific examples, but more importantly, from the large-scale entailment datasets. In other words, we make use of indirect supervision from textual entailment datasets to address the target few-shot task.
In summary, our contribution lies in three aspects. 1) We propose a new task named Incremental Few shot Text Classification with multi-round new classes. This task is more challenging and realistic for low resource scenarios. 2) We create and release two benchmark datasets to evaluate the performance of this new task. 3) We propose two novel models, ENTAILMENT and HYBRID, to solve this novel problem. Extensive experiments on these two datasets show the effectiveness of our proposed models.
Incremental few-shot learning. As far as we know, there is no prior work in the NLP domain that studies incremental few-shot text classification. In this section, we mainly introduce some work in the computer vision domain. These works only assume that a single round of new classes  $C_n$  is appended to the base classes  $C_b$ . Generally, they will learn class representations for classification. Different approaches differ in the way of representing base classes and new classes. Hereafter, we use  $W_b$  and  $W_n$  as the representations for  $C_b$  and  $C_n$ , respectively.
Snell et al. (2017) proposes the Prototypical Network, in which both  $W_{b}$  and  $W_{n}$  are stored as the average embedding of the few-shot support images for a certain class. Although Prototypical Network was not designed for incremental few-shot learning, it can be easily adapted to the incremental setting by providing the representations for all the classes. It trains a nearest neighbor algorithm on the base classes and tests directly on the union of base and new classes. Qi et al. (2018) proposes an "imprinting" mechanism: the base representations  $W_{b}$  are learned through supervised pre-training (e.g., the weight matrix in a softmax classifier), and  $W_{n}$  are computed using the averaged representations like Prototypical Network.
In Gidaris and Komodakis (2018), the base representations  $W_{b}$  are learned through supervised pre-training. The representation of the  $i^{th}$  novel class  $(W_{n,i})$  comes from two origins: (i) the prototypical averaging,  $w_{avg}$ ; (ii) attention-weighted sum over base representations:  $w_{att}$ . Namely,  $W_{n,i} = \phi_{avg} \odot w_{avg} + \phi_{att} \odot w_{att}$ , where  $\phi_{avg}$  and  $\phi_{att}$  are learnable weight vectors. In the few-shot training stage, the original base classes  $C_{b}$  are split into "new base classes" and "fake novel classes" for each episode. In testing, the representations of novel classes,  $W_{n}$ , are constructed based on the  $k$  examples and  $W_{b}$ .
In Ren et al. (2019), both  $W_{b}$  and  $W_{n}$  are learned through supervised training:  $W_{b}$  are classifier parameters pre-trained on base classes,  $W_{n}$  are classifier parameters learned in new classes. During the training, the support set and the query set are constructed differently for new classes. The support set consists of examples only from new classes; the query set contains examples from both new classes and base classes (because the training goal is to maximize the performance of all classes). The
training in this literature has two phases. The first phase is few-shot episode training which learns  $W_{n}$ , the second phase (called meta-learning training) optimizes the performance on the query set and regularizes the representations for new classes.
To summarize, compared with Snell et al. (2017) and Qi et al. (2018), both Gidaris and Komodakis (2018) and Ren et al. (2019) build connections between the representations of base classes and the new classes. However, these methods cannot be directly applied to our problem for the following reasons. (i) Despite the claims in some literature that they are dealing with incremental or dynamic few-shot problems, they only considered a single round of new classes (Qi et al., 2018; Gidaris and Komodakis, 2018; Ren et al., 2019). It is unclear if the system can keep the performance when multi-round new classes are considered. (ii) During the training for the new classes, they often rely on extra labeled data other than the  $k$  examples, such as the query set in Ren et al. (2019). (iii) Different from their setting, we have an extra label "none-of-them" in incremental few-shot text classification. It's not guaranteed that the input, such as the customer's utterance, always falls into the range of seen labels.
Using textual entailment for text classification. Zhang et al. (2020) is a state-of-the-art paper for few-shot text classification. They propose a clustering-based classifier named discriminative nearest neighbor classification (DNNC). DNNC compares whether two examples are in the same class or not. A matching model  $\mathrm{S}(x_i, x_j)$  is trained as a binary classifier, such that  $\mathrm{S}(x_i, x_j)$  is close to 1.0 if  $x_i$  and  $x_j$  belong to the same class, otherwise close to 0.0. Thus, their model can be pre-trained with a large-scale textual entailment dataset. Given a test query  $x$ , they compare the test query with all the previous examples. The final prediction is made by searching the nearest neighbor which has the highest matching score  $\mathrm{S}(x, x_i)$  with the query example. Their computation cost is high due to the comparison between all the utterance pairs.
Moreover, comparing whether two examples are in the same class is different from textual entailment. In textual entailment, a person reads a premise to infer that the hypothesis is true or not. The fact that two examples are in the same class does not mean they can entail each other. Thus, they cannot fully utilize the pre-trained entailment model. Instead, our proposed model, ENTAILMENT, entails the label with a given utterance,
which is much more efficient and maximizes the utilization of the pre-trained entailment model.
Yin et al. (2019) is another work that utilizes textual entailment for zero-shot text classification. They convert the zero-shot text classification as a problem of filling a label for a hypothesis. For example, they combine "emotion" labels with the question "this text expresses ?", and ask the model if this hypothesis is true, given the text. This work more focuses on zero-shot learning and they need to propose different questions for different labels.
In this section, we give a formal description of the problem "incremental few-shot text classification" without base classes. Furthermore, we extend the problem with additional base classes.
Training data. In the incremental few-shot text classification setting, the system is provided with  $m$  rounds of new classes sequentially:  $\{C_n^1,\dots ,C_n^m\}$ . Each round  $C_n^i$  has  $h$  new classes, namely  $C_n^i = \{C_{n,1}^i,\dots ,C_{n,h}^i\}$ . Each new class only has  $k$  examples ( $k\in [1,5]$ ). The value of  $k$  is not fixed and varies for different new classes in the same round, i.e.,  $k_{C_{n,s}^i}\neq k_{C_{n,t}^i}$ , where  $s,t\in [1,\ldots ,h]$ . For the setting with additional base classes, the system can access a set of base classes  $C_b = \{C_{b,1},C_{b,2},\dots ,C_{b,g}\}$ . All the base classes  $C_b$  have enough labeled examples for training.
We create the multi-round setting to mimic the real-world scenario where there is a sequence of new classes coming to the system. Since we can only collect a handful of examples for the upcoming classes and the number of examples cannot be guaranteed, we set  $k \in [1,5]$  and allow the flexibility that  $k_{C_{n,s}^i} \neq k_{C_{n,t}^i}$  in each round.
Development data. In the incremental few-shot setting, there are only  $k$  examples available for each new class. Thus, our formulation does not provide any development set to help select the best model. It is recommended to select hyperparameters based on experience or related tasks. In the experiments, we choose hyper-parameters like batch size based on the suggestions by Huggingface and other papers like Devlin et al. (2018) and Zhang et al. (2020).
Testing data. To evaluate the system, the test data consists of examples across all the classes. For the setting without base classes, the potential label space is  $C_n^1 \cup \dots \cup C_n^m \dots \cup C_o$ . For the setting with additional base classes, we search among all the classes in  $C_b \cup C_n^1 \cup \dots \cup C_n^m \dots \cup C_o$ .  $C_o$  is an extra out-of-distribution (OOD) class that consists of examples falling outside of all the seen classes. It gives us a chance to check the system's ability to detect instances that reject all the known classes. This is crucial for an open-set problem like incremental learning since there are always examples from upcoming classes that do not belong to any existing class.
Requirements. (i) For the training of  $i^{th}$  round  $C_n^i$ , the system can only access the newly added few-shot examples and label names in this round. The system is not allowed to re-train on the (full or partial) examples of preceding classes. (ii) For the evaluation, we care about the performance in different types of classes, including base classes, different rounds of new classes, and OOD classes in  $C_o$ . We expect a system that can continuously recognize new classes with few-shot examples. In the meantime, the performance of preceding classes should be maintained. A system showing severer catastrophic forgetting is less preferred.
4 Our Model: ENTAILMENT
Our approach ENTAILMENT casts the text classification problem into textual entailment: the input text acts as a premise, the class name, such as "open a bank account" in intent detection, acts as a hypothesis. Then the question that if the input belongs to a class is equivalent to ask if the hypothesis is true given the premise. There are two benefits of transforming the text classification problem to entailment. First, we can make use of indirect supervision from a large-scale entailment dataset (Williams et al., 2018) to benefit the few-shot settings. Second, this enables us to utilize the few-shot examples as well as the information of the class names. Typical text classification approaches treat classes as indices. In fact, class names usually contain informative signals.
Entailment pairs. To transfer the text classification problem into textual entailment, we construct positive and negative entailment pairs for the training. Positive entailment pairs  $(x_{i},y_{i})$  are constructed with utterance  $x_{i}$  and its gold label name
$y_{i}$ , where  $y_{i} \in C_{b}$  for base classes and  $y_{i} \in C_{n}^{i}$  for new classes. Negative entailment pairs consist of  $(x_{i}, y_{j})$ , where  $y_{j}$  is an incorrect label in the current round. For base classes,  $y_{j} \in C_{b}$  but  $y_{j} \neq y_{i}$ ; for new classes,  $y_{j} \in C_{n}^{i}$  but  $y_{j} \neq y_{i}$ .
For each entailment pair  $(x,y)$  whether it is positive or negative, we concatenate its utterance  $x$  with the label  $y$  and fed it into the RoBERTa (Liu et al., 2019) encoder. Given an utterance  $x = (X_{1},X_{2},\dots,X_{T_{2}})$  with  $T_{1}$  words and a label  $y = (Y_{1},Y_{2},\dots,Y_{T_{2}})$  with  $T_{2}$  words, we add a special start-of-sequence ([CLS]) token at the beginning of the input and a special end-of-sequence ([SEP]) token at the end of each sentence. The whole input is ([CLS],  $X_{1},X_{2},\dots,X_{T_{1}}$ , [SEP],  $Y_{1}, Y_{2},\dots,Y_{T_{2}}$ , [SEP]). We use the [CLS] embedding output from the RoBERTa encoder with a fully connected layer for binary textual entailment:
$$
e = \operatorname {R o B E R T a} (x, y), \tag {1}
$$
$$
p = \operatorname {s o f t m a x} (W e + z), \tag {2}
$$
where  $h\in \mathbb{R}^d$  is the embedding for the [CLS] token,  $W\in \mathbb{R}^{2\times d}$  and  $z\in \mathbb{R}^2$  are parameters.
Compared to Zhang et al. (2020), they discriminate whether two utterances  $(x_{i}, x_{j})$  are in the same class or not.  $(x_{i}, x_{j})$  is a positive pair if they belong to the same class, otherwise, it is a negative pair. To explore the potential of different combinations, we also propose a hybrid entailment model, HYBRID, that uses both (utterance, label) pairs  $(x_{i}, y_{i})$  and (utterance, utterance) pairs  $(x_{i}, x_{j})$ . In other words, we train HYBRID with pairs from both ENTAILMENT and DNNC (Zhang et al., 2020). In round  $C_{n}^{i}$  which contains  $h$  new classes and  $k$  examples for each class, ENTAILMENT generates  $h * k$  positive entailment pairs and  $(h - 1) * h * k$  negative entailment pairs, while DNNC generates  $h * k * (k - 1)$  positive pairs and  $h * (h - 1) * k^2$  negative pairs. HYBRID utilizes pairs from both models. For simplicity, we use the same  $k$  value for all new classes here; in real datasets, different new classes may have different numbers of few-shot examples. In that case, the number of generated pairs will change accordingly.
Training strategy. Both ENTAILMENT and HYBRID are binary classification models that can utilize indirect supervision from textual entailment. Firstly, we pre-train these models with a large-scale entailment dataset (Williams et al., 2018). For each round, models are fine-tuned on the new classes in
$C_n^i$ . For the setting with additional base classes, we fine-tune the models on base classes first. Then we continuously fine-tune the models on new classes.
Inference strategy. After the training, we use the model to infer the class for a test input. For each input utterance, we generate entailment pairs by accompanying the utterance with all classes except  $C_o$ . Each pair will get a score  $\lambda \in [0,1]$  indicating whether this input belongs to the particular class or not.  $\lambda > 0.5$  indicates "YES", "No" otherwise. If there is at least one class labeled with "YES", the class with the maximal  $\lambda$  score is returned; otherwise, the system returns  $C_o$ . We choose the threshold as 0.5 because entailment recognition is a binary classification problem.
Next, we compare our model with some related systems that can be potentially applied to the incremental few-shot text classification.
ENTAILMENT vs. Prototypical Network. Prototypical Network (Snell et al., 2017) tries to solve few-shot target tasks given a collection of training tasks. The few-shot learning problem solved in Prototypical network is slightly different from our incremental few-shot setting. In Prototypical Network, the label space for target tasks only contains the new classes. However, in the incremental few-shot setting, the target label space is continuously increasing by adding new classes. Due to this essential distinction, applying Prototypical Network to incremental few-shot are very likely to have performance drop on base classes when fine-tuning on new classes.
ENTAILMENT vs. Incremental few-shot approaches in computer vision. In Related Work, we introduced some typical approaches in computer vision that deal with the incremental few-shot problem. Those methods consistently try to learn representations for classes and examples separately (i.e., the  $W_{b}$  and  $W_{n}$  in Section 2). In our model, there are no individual representation vectors for classes or examples. Instead, the model learns an overall representation vector for the whole (input, class) pair. Our solution enables the learning of the input and the class to interact with each other, which has widely demonstrated its superiority in modeling the relations of two elements (Yu et al., 2020; Zhang et al., 2020).
In addition, the approaches in computer vision mostly rely on large-scale labeled data for base classes to train a robust system. We would argue
IFS-INTENT	IFS-RELATION
#class	#train	#test	#class	#train	#test
Cb	20	2088	800	10	5000	400
Cn1	10	30	400	10	30	400
Cn2	10	30	400	10	30	400
Cn3	10	30	400	10	30	400
Cn4	10	30	400	10	30	400
Cn5	10	30	400	10	30	400
Co	7	-	280	10	-	400
Table 1: Statistics of two datasets: IFS-INTENT and IFS-RELATION.  $C_b$ : base classes;  $\{C_n^1, \dots, C_n^5\}$ : five rounds of new classes;  $C_o$ : OOD classes. Note that  $C_o$  is never used for training.
that the base classes with rich annotations may not be available in real-world applications. Our system which can be pre-trained with entailment dataset, instead, does not rely on base classes. This makes our system more applicable to various scenarios.
IFS-INTENT. This is our benchmark for incremental few-shot intent detection. IFS-INTENT is converted from BANKING77 $^3$  (Casanueva et al., 2020), which is a single-domain intent detection dataset comprising 13,083 annotated examples over 77 intents (average: 170 examples per intent). Each intent class is described by a short name, such as "get physical card", "lost or stolen card", etc. We randomly split the 77 intents into a base group (i.e.,  $C_b$ , 20 base classes), 5 rounds of new intents (i.e.,  $\{C_n^1, \dots, C_n^5\}$ , each round has 10 new classes), and a group of out-of-distribution intents (i.e.,  $C_o$ , 7 ood classes).
IFS-RELATION. This is the benchmark for incremental few-shot relation classification. IFS-RELATION is converted from  $\mathrm{FewRel}^4$  (Han et al., 2018), which is a large-scale relation classification dataset. FewRel contains relations from different domains, including Wikipedia (Vrandecic and Krötzsch, 2014), SemEval-2010 (Hendrickx et al., 2019) and Pubmed. For classes in  $C_b$ ,  $C_n^1$ ,  $C_n^2$ ,  $C_n^3$ ,  $C_n^4$ , we randomly sample 10 classes from Wikipedia. Classes in  $C_n^5$  come from SemEval-2010 and classes in  $C_o$  come from Pubmed.
$^{3}$ https://github.com/PolyAI-LDN/  
task-specific-datasets  
 $^{4}$ https://github.com/thunlp/FewRel  
 $^{5}$ https://www.ncbi.nlm.nih.gov/pubmed/