SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qi-Wei Wang, Da-Wei Zhou, Yi-Kai Zhang, De-Chuan Zhan, Han-Jia Ye*
{wangqiwei,zhoudw,zhangyk,zhandc,yehj}@lamda.nju.edu.cn
Real-world scenarios are usually accompanied by continuously appearing classes with scarce labeled samples, which require the machine learning model to incrementally learn new classes and maintain the knowledge of base classes. In this Few-Shot Class-Incremental Learning (FSCIL) scenario, existing methods either introduce extra learnable components or rely on a frozen feature extractor to mitigate catastrophic forgetting and overfitting problems. However, we find a tendency for existing methods to misclassify the samples of new classes into base classes, which leads to the poor performance of new classes. In other words, the strong discriminability of base classes distracts the classification of new classes. To figure out this intriguing phenomenon, we observe that although the feature extractor is only trained on base classes, it can surprisingly represent the semantic similarity between the base and unseen new classes. Building upon these analyses, we propose a simple yet effective Training-frEE calibration (TEEN) strategy to enhance the discriminability of new classes by fusing the new prototypes (i.e., mean features of a class) with weighted base prototypes. In addition to standard benchmarks in FSCIL, TEEN demonstrates remarkable performance and consistent improvements over baseline methods in the few-shot learning scenario. Code is available at: https://github.com/wangkiw/TEEN
Deep Neural Networks (i.e., DNNs) have achieved impressive success in various applications [16, 19, 36, 55, 54, 5], but they usually rely heavily on static and pre-collected large-scale datasets (e.g., ImageNet [11]) to achieve this success. However, the data in real-world scenarios usually arrive continuously. For example, the face recognition system is required to authenticate existing users, and meanwhile, new users are continually added [30]. The scene where the model is required to continually learn new knowledge and maintain the ability on old tasks is referred to as Class-Incremental Learning (CIL) [41]. The main challenge of CIL is the notorious catastrophic forgetting problem [14], where the model forgets old knowledge as it learns new ones.
Many methods have been designed to overcome catastrophic forgetting in CIL from different perspectives, e.g., knowledge distillation [35], parameter regularization [22, 53], and network expansion [50, 47, 46]. These methods require new tasks to contain sufficient labeled data for supervised training. However, collecting enough labeled data in some scenarios is challenging, making the conventional CIL methods hard to deploy [30]. For example, the face recognition system can only collect very few facial images of a new user due to privacy reasons. Therefore, a more realistic and practical incremental learning paradigm, Few-Shot Class-Incremental Learning (FSCIL) [42], is
*Han-Jia Ye is the corresponding author.
proposed to address the problem of class-incremental learning with limited labeled data. Figure 1a gives a detailed illustration of FSCIL.
In addition to the catastrophic forgetting problem [14], FSCIL will suffer the overfitting problem because the model can easily overfit the very few labeled data of new tasks. Previous works [57, 58, 61] have adopted prototype-based methods [40, 33] to conquer the limited data problem. These methods freeze the feature extractor trained on base classes when dealing with new classes and use the prototypes of new classes as the corresponding classifier weights. The frozen feature extractor can alleviate the catastrophic forgetting problem and the prototype classifier can circumvent the overfitting problem. However, the inherent problem of learning with few-shot data is challenging to depict precisely the semantic information of a new category with limited data, making prototypes of new classes inevitably biased [27]. An intuitive reason for the biased prototypes is that the feature extractor has not been optimized for the new classes. For example, the feature extractor trained on "cat" and "dog" can not precisely depict the feature of a new class "bird" especially when the instances of "bird" are incredibly scarce. Many prototype adjustment methods [27, 57, 58, 15, 61, 1, 24] are dedicated to rectifying the biased representations of the new classes. These methods usually focus on designing complex pre-training algorithms to enhance the compatibility of representations [58, 59, 10], or complicated trainable modules that better adapt the representation of the new classes [57, 15, 61, 1], all of which rely
on significant training costs in exchange for improved model performance.
(a) An illustration of FSCIL setting.
(b) An illustration of the semantic similarity.
Figure 1: (a) The base session contains sufficient samples of base classes for training. The incremental sessions (i.e., sessions after the base session) only contain few-shot samples of new classes. FSCIL aims to obtain a unified classifier over all seen classes. Notably, the data from previous sessions are unavailable. (b) For example, considering the new classes "Plain", "Plate" and "Porcupine" in CIFAR100 [23], the corresponding most similar base classes selected by the similarity can depict the really semantic similarity. The three most similar base classes are selected by computing the cosine similarity between base prototypes and novel prototypes. The results show the feature extractor only trained on the base classes can also well represent the semantic similarity between the base and new classes.
However, we find that although existing methods perform well on the widely used performance measure (i.e., the average accuracy across all classes), they usually exhibit poor performance on new classes, which suggests that the calibration ability of existing methods could be improved. In other words, existing methods neglect the performance of new classes. A direct reason for this negligence is the current unified performance measure is easily overwhelmed by the dominated base classes (e.g., 60 base classes in the CIFAR100 dataset). However, the more recent tasks are usually more crucial in some real-world applications. For example, the new users in the recommendation system are usually more important and need more attention [45]. The important yet neglected new classes' performance inspires us to pay more attention to it.
To better understand the performance of current FSCIL methods [42, 6, 9, 58, 57], we explicitly evaluate and analyze the low performance of existing methods on new classes and empirically demonstrate the instances of new classes are prone to be predicted into base classes. To further understand this intriguing phenomenon, we revisit the representation of the feature extractor and find that the feature extractor trained only on base classes can already represent the semantic similarity between the base and new classes well. As shown in Figure 1b, although the novel classes are unavailable during the training of the feature extractor, the semantic similarity between the base and novel classes can also be well-represented. However, existing methods have been obsessed with designing complex learning modules and disregard this off-the-shelf semantic similarity.
2 "new class" and "novel class" in this paper mean the classes emerging after the base session.
Based on the above analysis, we propose to leverage the overlooked semantic similarity to explicitly enhance the discriminability of new classes. Specifically, we propose a simple yet effective Training-frEE prototype calibration (i.e., TEEN) strategy for biased prototypes of new classes by fusing the biased prototypes with weighted base prototypes, where the weights for the base prototypes are class-specific and semantic-aware. Notably, TEEN does not rely on any extra optimization procedure or model parameters once the feature extractor is trained on the base classes with a vanilla supervised optimization objective. Besides, the excellent calibration ability and training-free property make it a plug-and-play module.
- We empirically find that the lower performance of new classes is due to misclassifying the samples of new classes into corresponding similar base classes, which is intriguing and missing in existing studies.
- We propose a simple yet effective training-free calibration strategy for new prototypes, which not only achieved a higher average accuracy but also improved the accuracy of new classes  $(10.02\% \sim 18.40\%$  better than the runner-up FSCIL method).
- We validate TEEN on benchmark datasets under the standard FSCIL scenario. Besides, TEEN shows competitive performance under the few-shot learning scenario. The consistent improvements demonstrate TEEN's remarkable calibration ability.
2.1 Class Incremental Learning
The model in the Class-Incremental Learning (i.e., CIL) scenario is required to learn new classes without forgetting old ones. Save representative instances in old tasks (i.e., exemplars) and replay them in new tasks is a simple and effective way to maintain the model's discriminability ability on old tasks [20, 2, 37]. Furthermore, knowledge distillation [17, 35, 25, 49] is widely used to maintain the knowledge of the old model by enforcing the output logits between the old model and the new one to be consistent. iCaRL [35] uses the knowledge distillation as a regularization item and replays the exemplars when learning the feature representation. Many methods follow this line and design more elaborate strategies to replay exemplars [29] and distill knowledge [21, 12]. Recently, model expansion [28, 50, 47, 46, 39, 60] have been confirmed to be effective in CIL. The most representative method [50] saves a single backbone and freezes it for each incremental task. The frozen backbone substantially alleviates the catastrophic forgetting problem.
2.2 Few-Shot Learning
The model in Few-Shot Learning (i.e., FSL) [48] scenario is required to learn new classes with limited labeled data. Existing methods usually achieve this goal either from the perspective of optimization [13, 31, 3] or metric learning [40, 43, 26, 27, 56]. The core thought of optimization-based methods is to equip the model with the ability to fast adaptation with limited data. The metric-based methods focus on learning a unified and general distance measure to depict the semantic similarity between instances. Besides, [51] proposes to use Gaussian distribution to model each feature dimension of a specific class and sample augmented features from the calibrated distribution. Based on these augmented features, [51] train a logistic regression and achieve competitive performance. However, TEEN can outperform [51] in most settings and without any training cost when recognizing new classes.
2.3 Few-Shot Class-Incremental Learning
The model in Few-Shot Class-Incremental Learning (i.e., FSCIL) [42] scenario is required to incrementally learn new knowledge with limited labeled data. Many existing methods are dedicated to designing learning modules to train a more powerful feature extractor [58] or adapting the representation of the instances of new classes [57, 9, 61, 1]. Besides, TOPIC [42] utilizes a neural gas network to alleviate the challenging problems in FSCIL. [8] adopts word embeddings as semantic information and introduces a distillation-based FSCIL method. IDLVQ [6] proposes to utilize quantized reference vectors to compress the old knowledge and improve the performance in FSCIL. However, all these
methods overlook the abundant semantic information in base classes and the poor performance in new classes. In this study, we aim to take a small step toward filling this gap. The most related work to TEEN is [1]. However, it relies on optimizing a regularization-based objective function to implicitly utilize the semantic information. As opposed to [1], TEEN takes advantage of the empirical observation and gets rid of the optimization procedure and is thus more efficient and effective.
3.1 Definition and notations
In FSCIL [42], we assume there exists  $T$  sessions in total, including a base session (i.e., the first session) and  $T - 1$  incremental sessions (i.e., sessions after the first session). We denote the training data in the base session as  $\mathcal{D}_0$  and the training data in the incremental sessions as  $\{\mathcal{D}_1,\mathcal{D}_2,\ldots ,\mathcal{D}_{T - 1}\}$ . For the training data  $\mathcal{D}_i$  in the  $i$ -th session, we further denote it with  $\{(x_j,y_j)\}_{j = 1}^{N_i}$  and corresponding label space with  $\mathcal{C}_i$ . Note that only training data  $\mathcal{D}_i$  is available in the  $i$ -th session. Accordingly, the testing data and testing label space in session  $i$  can be denoted as  $\mathcal{D}_i^{\text{test}}$  and  $\mathcal{C}_i^{\text{test}}$ . To better evaluate the model's discriminability on all seen tasks, the testing label space  $\mathcal{C}_i^{\text{test}}$  of  $i$ -th session contains all seen classes during training, i.e.,  $\mathcal{C}_i^{\text{test}} = \bigcup_{j = 0}^i\mathcal{C}_j$ . An incremental session can also be denoted as a  $N$ -way  $K$ -shot classification task, i.e.,  $N$  classes and  $K$  labeled examples for each class. Note that the training label spaces between different sessions are disjoint, i.e., for any  $i,j\in [0,T - 1]$  and  $i\neq j,\mathcal{C}_i\cap \mathcal{C}_j = \emptyset$ .
Compared to conventional CIL, FSCIL only requires the model to learn new classes with limited labeled data. On the other hand, compared to conventional Few-Shot Learning (FSL), FSCIL requires the model to continually learn the knowledge of new classes while retaining the knowledge of previously seen classes. We introduce the related works on CIL, FSL and FSCIL in supplementary due to the space limitation.
The model in FSCIL can be decoupled into a feature encoder  $\phi_{\theta}(\cdot)$  with parameters  $\theta$  and a linear classifier  $W$ . Given a sample  $x_{j}\in \mathbb{R}^{D}$ , the feature of  $x_{j}$  can be denoted as  $\phi_{\theta}(x_j)\in \mathbb{R}^d$ . For a  $N$ -class classification task, the output logits of a sample  $x_{j}$  can be denoted as  $\mathcal{O}_j = W^\top \phi_\theta (x_j)\in \mathbb{R}^N$  where  $W\in \mathbb{R}^{d\times N}$ .
3.2 Prototypical Network
ProtoNet [40] is a widely used method in few-shot learning problems. It computes the mean feature  $c_{k}$  of a class  $k$  (e.g., class prototype) and uses the class prototype to represent the corresponding class:
$$
c _ {k} = \frac {1}{\operatorname {N u m} _ {k}} \sum_ {y _ {j} = k} \phi_ {\theta} \left(x _ {j}\right) \tag {1}
$$
$\mathrm{Num}_k$  denotes the number of samples in class  $k$ . For a classification task with  $N$  classes, the classifier can be represented by the  $N$  prototypes, i.e.,  $W = [c_1, c_2, \ldots, c_N]$ . Following the [58, 57, 61, 1], we freeze the feature extractor  $\phi_{\theta}$  trained on the base task and plug the class prototype into the classifier while dealing with a new class. The frozen feature extractor can alleviate catastrophic forgetting and the plug-in updating of the classifier can circumvent the overfitting problem relatively.
4 A Closer Look at FSCIL
In this session, we comprehensively analyze current FSCIL methods from a decoupled perspective. Although the previous updating paradigm of extractor-frozen and prototypes-plugged can achieve adequate average accuracy in all classes, there also exist some shortcomings in it. In this section, we empirically show that the current methods (e.g., [57, 58]) are generally not effective in new classes. Furthermore, we take a step toward understanding the reason for the low performance in new classes.
4.1 Understanding the reason for poor performance in new classes
To better understand the performance of existing methods, we first measure the performance by average accuracy on all classes, base classes and new classes, respectively. As illustrated in Figure 2,
Table 1: Detailed prediction results of False Negative Rate/False Positive Rate (%) on CIFAR100 [23] dataset. The analysis results are from session 1 because new classes do not exist in session 0. Exceedingly high FPR and relatively low FNR show the instances of new classes are easily misclassified into base classes and the instances of base classes are also easily misclassified into base classes. TEEN can achieve relatively lower FPR than baseline methods, which demonstrates the validity of the proposed calibration strategy. Please refer to the supplementary for results on miniImageNet and CUB200.
Session	1	2	3	4	5	6	7	8
FNR/FPR	FNR FPR	FNR FPR	FNR FPR	FNR FPR	FNR FPR	FNR FPR	FNR FPR	FNR FPR
ProtoNet [40]	2.13 67.40	4.42 67.40	6.68 60.67	8.28 58.90	10.25 56.68	11.70 54.47	12.57 51.66	13.78 51.80
CEC [57]	2.32 70.40	4.38 66.20	6.18 62.20	7.50 58.65	9.72 56.00	11.30 53.60	12.12 51.40	13.48 51.78
FACT [58]	2.05 66.60	3.88 61.70	5.58 56.80	7.23 55.05	8.85 53.64	9.83 51.13	10.45 48.83	11.75 49.27
TEEN	4.03 57.40	7.40 52.50	9.35 45.00	11.58 40.75	14.00 40.80	15.78 39.23	16.33 36.91	18.75 36.65
Table 2: Detailed prediction results of TNR/TBR (\%) on CIFAR100 [23] dataset. The analysis results are from session 1 because new classes do not exist in session 0. For new classes, we only consider the 10 most similar base classes out of 60 base classes. For base classes, we suppose  $C_i$  new classes exist in the current incremental session  $i$ . We only consider the most similar  $\lfloor 20\% \times C_i \rfloor$  new classes. Class similarity adopts cosine similarity between different class prototypes. TEEN can achieve relatively lower TBR than baseline methods, which demonstrates the validity of the proposed calibration strategy. Please refer to the supplementary for results on miniImageNet and CUB200.
Session	1	2	3	4	5	6	7	8
TNR/TBR	TNR TBR	TNR TBR	TNR TBR	TNR TBR	TNR TBR	TNR TBR	TNR TBR	TNR TBR
ProtoNet [40]	3.47	73.01	5.02	61.79	7.06	55.05	8.08	52.95
CEC [57]	2.87	71.70	4.53	61.08	5.66	58.26	6.35	55.16
FACT [58]	2.33	70.00	3.32	61.01	5.50	55.17	6.88	50.55
TEEN	2.16	66.77	2.53	55.14	3.55	44.87	3.85	37.68
3.57	39.47
4.02	37.07
4.76	33.83
4.85	35.04
our first observation is that the average accuracy across base classes is extremely higher than the accuracy in new classes. The inconsistent performance between the base and new classes is caused by the frozen feature extractor and biased new prototypes. The former forces the model to overfit base classes and the latter causes the model to underfit the new classes.
To determine the real cause of the low performance on new classes, we further investigate "What base classes are the new classes incorrectly predicted into?" and get our second observation.
Our second observation is that the prototype-based classifier misclassifies the new classes to their corresponding most similar base classes with high probability, i.e., many instances of new classes are closer to their nearest base prototype than corresponding target prototypes
To verify this observation, we first analyze the detailed prediction results from a decoupled perspective. Specifically, we treat all base classes as a "positive class" and all new classes as a "negative class" and transform the FSCIL problem into a two-class classification task. We compute the false negative rate (i.e., FNR) and false positive rate (i.e., FPR) of each binary prediction task on each incremental session. The FNR and FPR in the confusion matrix is defined as follows:
$$
\mathrm{FNR} = \frac{\mathrm{FN}}{\mathrm{TP} + \mathrm{FN}}\times 100\% ,\mathrm{FPR} = \frac{\mathrm{FP}}{\mathrm{FP} + \mathrm{TN}}\times 100\% \tag{2}
$$
Figure 2: The performance of FACT [58] on CIFAR100 dataset. NAcc,BAcc,AvgAcc mean the average accuracy on new classes, base classes, and all classes, respectively.
As shown in Table 1, the FPR is far greater than FNR illustrating new classes are generally misclassified into base classes but the base classes are generally misclassified into base classes. On the basis of this conclusion, we further explore the details of misclassification. To better illustrate the analysis, we define "misclassified To most similar Base classes Ratio" (i.e., TBR) for new classes and "misclassified To most similar New classes Ratio" (i.e., TNR) for base classes. Specifically, considering the misclassified instances in base classes and new classes respectively, the TBR and
(a) An qualitative illustration of the calibration
Incremental Session|	1	2
Sample type	UC	W→R	R→W	UC	W→R	R→W
Base Class	93.08	8.51	76.92	87.51	7.95	68.03
New Class	6.92	91.49	23.08	12.49	92.05	31.97
(b) An quantitative illustration of the calibration
Figure 3: (a) The  $\bigcirc$  represents the samples of new classes and the  $\triangle$  represents the samples of base classes. The dotted lines between samples and prototypes (i.e.,  $c_{n}, c_{n}^{base}$ ,  $\bar{c}_{n}$ ) represent the corresponding classification results. Blue and purple dotted lines represent samples that are correctly classified, and red dotted lines represent misclassification. The yellow circles (i.e.,  $\mathbf{W} \rightarrow \mathbf{R}$  samples) and green triangles (i.e.,  $\mathbf{R} \rightarrow \mathbf{W}$  samples) in the right figure are samples with prediction changes after calibration. (b) The detailed ratio (\%) of base and new classes with regard to three types of samples (i.e., UC samples,  $\mathbf{W} \rightarrow \mathbf{R}$  samples,  $\mathbf{R} \rightarrow \mathbf{W}$  samples). Only two incremental sessions (i.e., session 1 and session 2) of CIFAR100 are listed here for convenience. Please refer to the supplementary for results on more incremental sessions and more datasets.
TNR are defined as follows:
$$
\mathrm{TBR} = \frac {M _ {b}}{N _ {n}} \times 100 \%, \mathrm{TNR} = \frac {M _ {n}}{N _ {b}} \times 100 \% \tag{3}
$$
$N_{n}$  means the number of misclassified instances of new classes.  $M_{b}$  is the number of instances misclassified into most similar base classes in  $N_{n}$  samples.  $M_{n}$  and  $N_{b}$  have similar meaning. As shown in Table 2, the TBR is higher and the TNR is relatively low, which strongly underpins our second observation. To our knowledge, we are the first to explain the reason for the low performance of new classes in existing methods and observe that the samples of new classes are easily misclassified into the most similar base classes.
To summarize, we empirically demonstrate that existing methods perform poorly on new classes and find that this is because the model tends to misclassify new class samples into the most similar base classes. Combining existing analysis (i.e., the samples of new classes tend to be classified into the most similar base classes) and observations (i.e., the well-trained feature extractor on base classes can also well represent the semantic similarity between the base and new classes), we propose to use the frozen feature extractor as a bridge and calibrate the biased prototypes of new classes by fusing the biased new prototypes with the well-calibrated base prototypes.
5 Similarity-based Prototype Calibration
Based on the common assumptions in FSCIL [42], sufficient instances of base classes are available during the training of the feature extractor. We argue the sufficient data in the base session contains abundant semantic information (i.e., a sufficient number of classes) and the base prototypes are well-calibrated (i.e., a sufficient number of instances for each class). The above analysis leads to a natural question:
Can we leverage the well-calibrated prototypes in the base session for a new prototype calibration?
As the previous methods overlook the low performance of new classes caused by the corresponding biased prototypes, we propose to explicitly calibrate these biased prototypes with the help of the well-calibrated base prototypes. The off-the-shelf semantic similarity serves as a bridge between the base prototypes and the new ones. In the following sections, we introduce the details of fusing the well-calibrated base prototypes and ill-calibrated new prototypes to calibrate the biased ones. Afterwards, we analyze the effects of this training-free prototype calibration on new classes and base classes, respectively.
5.1 Fusing the biased prototypes with calibration item