SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dipam Goswami $^{1,2}$  Yuyang Liu $^{3,4,5,\dagger}$  Bartlomiej Twardowski $^{1,2,6}$  Joost van de Weijer $^{1,2}$
<sup>1</sup>Department of Computer Science, Universitat Autònoma de Barcelona
$^{2}$ Computer Vision Center, Barcelona  $^{3}$ University of Chinese Academy of Sciences
$^{4}$ State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences
$^{5}$ Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences  $^{6}$ IDEAS-NCBR {dgoswami, btwardowski, joost}@cvc.uab.es, sunshineleiuyuyang@gmail.com
Exemplar-free class-incremental learning (CIL) poses several challenges since it prohibits the rehearsal of data from previous tasks and thus suffers from catastrophic forgetting. Recent approaches to incrementally learning the classifier by freezing the feature extractor after the first task have gained much attention. In this paper, we explore prototypical networks for CIL, which generate new class prototypes using the frozen feature extractor and classify the features based on the Euclidean distance to the prototypes. In an analysis of the feature distributions of classes, we show that classification based on Euclidean metrics is successful for jointly trained features. However, when learning from non-stationary data, we observe that the Euclidean metric is suboptimal and that feature distributions are heterogeneous. To address this challenge, we revisit the anisotropic Mahalanobis distance for CIL. In addition, we empirically show that modeling the feature covariance relations is better than previous attempts at sampling features from normal distributions and training a linear classifier. Unlike existing methods, our approach generalizes to both many- and few-shot CIL settings, as well as to domain-incremental settings. Interestingly, without updating the backbone network, our method obtains state-of-the-art results on several standard continual learning benchmarks. Code is available at https://github.com/dipamgoswami/FeCAM.
In Continual Learning (CL), the learner is expected to accumulate knowledge from the ever-changing stream of new tasks data. As a result, the model only has access to the data from the current task, making it susceptible to catastrophic forgetting of previously learned knowledge [46, 37]. This phenomenon has been extensively studied in the context of Class Incremental Learning (CIL) [36, 61, 9, 74], where the objective is to incrementally learn new classes and achieve the highest accuracy for all classes encountered so far in a task-agnostic way without knowing from which tasks the evaluated samples are [56]. While one of the simplest approaches to mitigate forgetting is storing exemplars of each class, it has limitations due to storage and privacy concerns, e.g., in medical images. Hence, the focus has shifted towards more challenging exemplar-free CIL methods [80, 63, 43, 41, 35, 2].
In exemplar-free CIL methods, the challenge is to discriminate between old and new classes without access to old data. While some methods [79, 51, 78, 80] trained the model on new classes favoring plasticity and used knowledge distillation [20] to preserve old class representations, other methods [43, 3, 13, 41, 35] froze the feature extractor after the first task, thus favoring stability and incrementally
†The corresponding author is Yuyang Liu.
(a) Joint Training
(b) CIL - High Stability
(c) CIL - High Plasticity
Figure 1: Illustration of feature representations in CIL settings. In Joint Training (a), deep neural networks learn good isotropic spherical representations [17] and thus the Euclidean metric can be used effectively. However, it is challenging to learn isotropic representations of both old and new classes in CIL settings. When the model is too stable in (b), it is unable to learn good spherical representations of new classes and when it is too plastic in (c), it learns spherical representations of new classes but loses the spherical representations of old classes. Thus, it is suboptimal to use the isotropic euclidean distance. We propose FeCAM in (d) which models the feature covariance relations using Mahalanobis metric and learns better non-linear decision boundaries for new classes.
(d) CIL - FeCAM
learned the classifier. One of the drawbacks is the inability to learn new representations with a frozen feature extractor. Inspired by transfer learning [49], the objective of these classifier-incremental methods is to make best use of the learned representations from the pretrained model and continually adapt the classifier to learn new tasks. Recently, pretrained feature extractors have been used in exemplar-free CIL by prompt-based methods [63], using linear discriminant analysis [41] and with a simple nearest class mean (NCM) classifier [22]. These methods use a transformer model pretrained on large-scale datasets like ImageNet-21k [45] and solely focus on classifier-incremental learning.
This paper investigates methods to enhance the representation of class prototypes in CIL, aiming to improve plasticity within the stability-favoring classifier-incremental setting. A standard practice in few-shot CIL [31, 42, 76, 70, 72] is to obtain the feature embeddings of new class samples and average them to generate class-wise prototypes. The test image features are then classified by computing the Euclidean distance to the mean prototypes. The Euclidean distance is used in the NCM classifier, following [17], which claims that the highly non-linear nature of learned representations eliminates the need to learn the Mahalanobis [65, 11] metric previously used [38]. Our analysis shows that this holds true for classes that are considered during training, however, for new classes, the Euclidean distance is suboptimal. To address this problem, we propose to use the anisotropic Mahalanobis distance. In Fig. 1, we explain how the feature representations vary in CIL settings. Here, the high-stability case in CIL is explored, where the model does not achieve spherical representations for new classes in the feature space, unlike joint training. Thus, it is intuitive to take into account the feature covariances while computing the distance. The covariance relations between the feature dimensions better captures the more complex class structure in the high-dimensional feature space. Additionally, in Fig. 3, we analyze singular values for old and new class features to observe the changes in variances in their feature distributions, suggesting a shift towards more anisotropic representations.
While previous methods [38] proposed learning Mahalanobis metrics, we propose using an optimal Bayes classifier by modeling the covariance relations of the features and employing class prototypes. We term this approach Feature Covariance-Aware Metric (FeCAM). We compute the covariance matrix for each class from the feature embeddings corresponding to training samples and perform correlation normalization to ensure similar variances across all class representations, which is crucial for distance comparisons. We investigate various ways of using covariance relations in continual settings. We posit that utilizing a Bayes classifier enables better learning of optimal decision boundaries compared to previous attempts [67] involving feature sampling from Gaussian distributions and training linear classifiers. The proposed approach is simple to implement and requires no training since we employ a Bayes classifier. The Bayes classifier FeCAM can be used for both many-shot CIL and few-shot CIL, unlike existing methods. Additionally, we achieve superior performance with pretrained models on both class-incremental and domain-incremental benchmarks.
Many-shot class-incremental learning (MSCIL) is the conventional setting, where sufficient training data is available for all classes. A critical aspect in many-shot CIL methods is the semantic drift in
the feature representations [68] while training on new tasks. While recent methods use knowledge distillation [20] or regularization strategies [25, 68] to maintain the representations of old classes, these methods are dependent on storing images [30, 7, 21, 14, 16, 6, 5, 24], representations [23] or instances [33] from old tasks and becomes ineffective in practical cases where data privacy is required. Another set of methods [43, 3, 13, 41, 35] proposed freezing the feature extractor after the first task and learning only the classifier on new classes. We follow the same setting and do not violate privacy concerns by storing exemplars.
On the other hand, Few-Shot Class-Incremental Learning (FSCIL) considers that very few (1 or 5) samples per class are available for training [31, 53, 58]. Generally, these settings assume a big first task and freezes the network after training on the initial classes. To address the challenges of FSCIL, various techniques have been proposed. One approach is to use meta-learning to learn how to learn new tasks from few examples [39, 48, 4, 54]. Another approach is to incorporate variational inference to learn a distribution over models that can adapt to new tasks [40]. Most FSCIL methods [1, 8, 31, 53, 42, 76, 70, 72] obtains feature embeddings of a small number of examples and averages them to get the class-wise prototypes. These methods use the Euclidean distance to classify the features assuming equally spread classes in the feature space. We explore using the class prototypes in both MSCIL and FSCIL settings.
Since the emergence of deep neural networks, Euclidean distance is used in NCM classifier following [17], instead of Mahalanobis distance [38]. Mahalanobis distance has been recently explored for out-of-distribution detection with generative classifiers [29] and an ensemble using Mahalanobis [24] and also within the context of cross-domain continual learning [50].
For the classification of hand-crafted features, Mensink et al. [38] proposed the nearest class mean (NCM) classifier using (squared) Mahalanobis distance  $\mathcal{D}_M$  instead of Euclidean distance to assign an image to the class with the closest mean (see also illustration in Fig. 2):
$$
y ^ {*} = \underset {y = 1, \dots , Y} {\operatorname {a r g m i n}} \mathcal {D} _ {M} (x, \mu_ {y}), \quad \mathcal {D} _ {M} (x, \mu_ {y}) = (x - \mu_ {y}) ^ {T} M (x - \mu_ {y}) \tag {1}
$$
where  $Y$  is the number of classes,  $x, \mu_y \in \mathbb{R}^D$ , class mean  $\mu_y = \frac{1}{|X_y|} \sum_{x \in X_y} x$ , and  $M$  is a positive definite matrix. They learned a low-rank matrix  $M = W^T W$  where  $W \in \mathbb{R}^{m \times D}$ , with  $m \leq D$ .
However, with the shift towards deep feature representations, Guerriero et al. [17] assert that the learned representations with a deep convolutional network  $\phi : \mathcal{X} \to \mathbb{R}^D$ , eliminate the need of learning the Mahalanobis metric  $M$  and the isotropic Euclidean distance  $\mathcal{D}_e$  can be used as follows:
$$
y ^ {*} = \underset {y = 1, \dots , Y} {\operatorname {a r g m i n}} \mathcal {D} _ {e} (\phi (x), \mu_ {y}), \quad \mathcal {D} _ {e} (\phi (x), \mu_ {y}) = \left(\phi (x) - \mu_ {y}\right) ^ {T} \left(\phi (x) - \mu_ {y}\right) \tag {2}
$$
where  $\phi(x), \mu_y \in \mathbb{R}^D$ ,  $\mu_y = \frac{1}{|X_y|} \sum_{x \in X_y} \phi(x)$  is the class prototype or the average feature vector of all samples of class  $y$ . Here,  $\phi(x)$  is the feature vector corresponding to image  $x$  and could be the output of penultimate layer of the network. In Euclidean space,  $M = I$ , where  $I$  is an identity matrix.
The success of the NCM classifier for deep learned representation (as observed by [17]) has also been adopted by the incremental learning community. The NCM classifier with Euclidean distance is now commonly used in incremental learning [44, 68, 10, 79, 72, 42, 70, 76]. However, in incremental learning, we do not jointly learn the features of all data, but are learning on a nonstatic data stream. As a result, the underlying learning dynamics which result in representations on which Euclidean distances perform excellently might no longer be valid. Therefore, we perform a simple comparison with the Euclidean and Mahalanobis distance (see Fig. 4) where we use a network trained on  $50\%$  of
Figure 2: Illustration of distances (contour lines indicate points at equal distance from prototype).
(a) Singular values
(b) Old classes
Figure 3: (a) Singular values comparison for old and new classes, (b-c) Visualization of features for old classes and new classes by t-SNE, where the colors of points indicate the corresponding classes.
(c) New classes
classes of CIFAR100 (identified as old classes). Interestingly, indeed the old classes, on which the feature representation is learned, are very well classified with Euclidean distance, however, for the new classes this no longer holds, and the Mahalanobis distance obtains far superior results.
As a second experiment, we compare singular values of old and new classes (see Fig. 3a). We observe that the singular values of new class features vary more and are in general larger than those of old classes. This points out that new class distributions are more heterogeneous (and are more widely spread) compared to the old classes and hence the importance of a heterogeneous distance measure (like Mahalanobis). This is also confirmed by a t-SNE plot of both old and new classes (see Fig. 3b,c) showing that the new classes, which were not considered while training the backbone are badly modeled by a spherical distribution assumption, as is underlying the Euclidean distance.
Based on these results, two extreme cases of CIL are illustrated in Fig. 1, considering maximum stability where the backbone is frozen, and maximum plasticity where the training is done with fine-tuning without preventing forgetting. In this paper, we revisit the nearest class mean classifier based on a heterogeneous distance measure. We perform this for classifier incremental learning, where the backbone is frozen after the first task. However, we think that conclusions based on the heterogeneous nature of class distributions also have consequences for continual deep learning where the representations are continually updated.
Figure 4: Accuracy Comparison of NCM (Euclidean) and FeCAM (Mahalanobis) using common covariance matrix and a matrix per class on CIFAR100 50-50 (2 task) sequence, for Old, New, and All classes at the end of the learning sequence.
3.2 Bayesian FeCAM Classifier
When modeling the feature distribution of classes with a multivariate normal feature distribution  $\mathcal{N}(\mu_y,\Sigma_y)$ , the probability of a sample feature  $x$  belonging to class  $y$  can be expressed as,
$$
P (x \mid C = y) \approx \exp \frac {- 1}{2} \left(x - \mu_ {y}\right) ^ {T} \boldsymbol {\Sigma} _ {y} ^ {- 1} \left(x - \mu_ {y}\right), \tag {3}
$$
It is straightforward to see that this is the optimal Bayesian classifier, since:
$$
\operatorname {a r g m a x} _ {y} P (Y \mid X) = \operatorname {a r g m a x} _ {y} \frac {P (X \mid Y) P (Y)}{P (X)} = \operatorname {a r g m a x} _ {y} P (X \mid Y) P (Y) = \operatorname {a r g m a x} _ {y} P (X \mid Y) \tag {4}
$$
where optimal boundary occurs at those points where each class is equally probable  $P(y_{i}) = P(y_{j})$ .
Since logarithm is a concave function and thus  $\operatorname{argmax} P(X|Y) = \operatorname{argmax} \log P(X|Y)$
$$
\underset {y} {\operatorname {a r g m a x}} \log P (X | Y) = \underset {y} {\operatorname {a r g m a x}} \left\{- \left(x - \mu_ {y}\right) ^ {T} \boldsymbol {\Sigma} _ {y} ^ {- 1} \left(x - \mu_ {y}\right) \right\} = \underset {y} {\operatorname {a r g m i n}} \mathcal {D} _ {M} \left(x, \mu_ {y}\right) \tag {5}
$$
where the squared mahalanobis distance  $\mathcal{D}_M(x,\mu_y) = (x - \mu_y)^T\pmb{\Sigma}_y^{-1}(x - \mu_y)$ .
In the following, we elaborate on several techniques that can be applied to improve and stabilize Mahalanobis-based distance classification. We apply these, resulting in our FeCAM classifier (an ablation study in section 4.2.4 confirms the importance of these on overall performance).
Covariance Matrix Approximation. We obtain the covariance matrix from the feature vectors  $\phi(x)$  corresponding to the samples  $x$  at any task. The covariance matrix can be obtained in different ways. A common covariance matrix  $\pmb{\Sigma}^{1:t}$  can be incrementally updated as the mean covariance matrix for all seen classes till task  $t$  as follows:
$$
\boldsymbol {\Sigma} ^ {1: t} = \boldsymbol {\Sigma} ^ {1: t - 1} \cdot \frac {| Y ^ {1 : t - 1} |}{| Y ^ {1 : t} |} + \boldsymbol {\Sigma} ^ {t} \cdot \frac {| Y ^ {1 : t} | - | Y ^ {1 : t - 1} |}{| Y ^ {1 : t} |} \tag {6}
$$
where  $\pmb{\Sigma}^t$  refers to the common covariance matrix obtained from the features of samples  $x\in X^t$  from all classes seen in task  $t$  and  $|.|$  denotes the number of classes seen till that task. This common covariance matrix will represent the feature distribution for all seen classes and requires storing only the common covariance matrix from the last task.
The other alternative is to use a covariance matrix  $\Sigma_y$  for  $y\in 1,\dots Y$  to represent the feature distribution of each class separately. Here,  $\Sigma_y$  is the covariance matrix obtained from the feature vectors of all the samples  $x\in X_{y}$ . This will involve storing a separate matrix for all seen classes.
Normalization of Covariance Matrices. The covariance matrix  $\Sigma_y$  obtained for each class will have different levels of scaling and variances along different dimensions. Particularly, due to the notable shift in feature distributions between the old and new classes, the variances are much higher for the new classes. As a result, the Mahalanobis distance of features from different classes will have different scaling factors, and the distances will not be comparable. So, in order to be able to use a covariance matrix per class, we perform a correlation matrix normalization on all the covariance matrices. In order to make the multiple covariance matrices comparable, we make their diagonal elements equal to 1. A normalized covariance matrix can be obtained as:
$$
\hat {\boldsymbol {\Sigma}} _ {y} (i, j) = \frac {\boldsymbol {\Sigma} _ {y} (i , j)}{\sigma_ {y} (i) \sigma_ {y} (j)}, \quad \sigma_ {y} (i) = \sqrt {\boldsymbol {\Sigma} _ {y} (i , i)}, \quad \sigma_ {y} (j) = \sqrt {\boldsymbol {\Sigma} _ {y} (j , j)} \tag {7}
$$
where  $\sigma_y(i)$  and  $\sigma_y(j)$  refers to the standard deviations along the dimensions  $i$  and  $j$  respectively.
We identify the difficulties of obtaining an invertible covariance matrix in cases when the number of samples are less than the number of dimensions. So, we use a covariance shrinkage method to get a full-rank matrix. We also show that a simple gaussianization of the features using tukey's normalization [55] is also helpful.
Covariance Shrinkage. When the number of samples available for a class is less than the number of feature dimensions, the covariance matrix  $\Sigma$  is not invertible, and thus it is not possible to use  $\Sigma$  to get the Mahalanobis distance. This is a very serious problem since most of the deep learning networks have large number of feature dimensions [69]. Similar to [27, 57], we perform a covariance shrinkage to obtain a full-rank matrix as follows:
$$
\Sigma_ {s} = \Sigma + \gamma_ {1} V _ {1} I + \gamma_ {2} V _ {2} (1 - I), \tag {8}
$$
where  $V_{1}$  is the average diagonal variance,  $V_{2}$  is the average off-diagonal covariance of  $\pmb{\Sigma}$  and  $I$  is an identity matrix.
Tukey's Ladder of Powers Transformation. Tukey's Ladder of Powers transformation aims to reduce the skewness of distributions and make them more Gaussian-like. We transform the feature vectors  $\phi(x)$  using Tukey's Ladder of Powers transformation [55]. It can be formulated as:
$$
\tilde {\phi} (x) = \left\{ \begin{array}{l l} \phi (x) ^ {\lambda} & \text {i f} \lambda \neq 0 \\ \log (\phi (x)) & \text {i f} \lambda = 0 \end{array} \right. \tag {9}
$$
where  $\lambda$  is a hyperparameter to decide the degree of transformation of the distribution. In our experiments, we use  $\lambda = 0.5$  following [67].
We obtain the normalized features using Tukey's transformation and then use the transformed features to obtain the covariance matrices. When using multiple matrices, we perform correlation normalization to make them comparable. The final prediction and the squared Mahalanobis distance to the different class prototypes using one covariance matrix per class can be obtained as:
$$
y ^ {*} = \underset {y = 1, \dots , Y} {\operatorname {a r g m i n}} \mathcal {D} _ {M} (\phi (x), \mu_ {y}), \quad \mathcal {D} _ {M} (\phi (x), \mu_ {y}) = \left(\tilde {\phi} (x) - \tilde {\mu} _ {y}\right) ^ {T} \left(\hat {\boldsymbol {\Sigma}} _ {y}\right) _ {s} ^ {- 1} \left(\tilde {\phi} (x) - \tilde {\mu} _ {y}\right) \tag {10}
$$
where  $\tilde{\phi(x)}$  and  $\tilde{\mu_y}$  refers to the tukey transformed features and prototypes respectively and  $(\hat{\Sigma}_{\mathbf{y}})_s^{-1}$  denotes the inverse of the covariance matrices which first undergoes shrinkage followed by normalization. Note that the covariance matrices are computed using the tukey transformed features. Similarly, the common covariance matrix  $\pmb{\Sigma}^{1:t}$  can be used in Eq. (10).
3.3 On the Suboptimality of Learning Linear Classifier
Previous methods like [67, 27] in few-shot learning assumed Gaussian distributions of classes in feature space and proposed to transfer statistics of old classes to obtain calibrated distributions of new classes and then sample examples from the calibrated distribution to train a linear logistic regression classifier. In our setting, we consider a similar baseline for comparison which assumes Gaussian distributions for features of old classes (by storing the mean and the covariance matrix from the features of the old classes) and samples features from these distributions to learn a linear classifier. We state that this is not an ideal solution since the optimal decision boundaries need not be
Figure 5: Avg Acc comparison of bayesian and linear classifier on CI-FAR100  $(\mathrm{T} = 5)$  setting.
linear. The optimal decision boundaries are linear when the covariances of all classes are equal like in the Euclidean space. When the covariances of classes are not equal, the optimal decision boundaries are non-linear and forms a quadratic surface in high-dimensional feature space. We show in Fig. 5 that using the optimal Bayesian classifier obtains much better performance compared to sampling features and training a linear classifier, even when sampling many examples per class.
We evaluated FeCAM with strong baselines on multiple datasets and different scenarios.
MSCIL datasets and setup. We conduct experiments on three publicly available datasets: 1) CIFAR100 [26] - consisting of 100 classes,  $32 \times 32$  pixel images with 500 and 100 images per class for training and testing, respectively; 2) TinyImageNet [28] - a subset of ImageNet with 200 classes,  $64 \times 64$  pixel images, and 500 and 50 images per class for training and testing, respectively; 3) ImageNet-Subset [12] - a subset of the ImageNet LSVRC dataset [47] consisting of 100 classes with 1300 and 50 images per class for training and testing, respectively. We divide these datasets into incremental settings, where the number of initial classes in the first task is larger and the remaining classes are evenly distributed among the incremental tasks. We experiment with three different incremental settings for CIFAR100 and ImageNet-Subset: 1) 50 initial classes and 5 incremental learning (IL) tasks of 10 classes; 2) 50 initial classes and 10 IL tasks of 5 classes; 3) 40 initial classes and 20 IL tasks of 3 classes. For TinyImageNet, we use 100 initial classes and distribute the remaining classes into three incremental settings: 1) 5 IL tasks of 20 classes; 2) 10 IL tasks of 10 classes; 3) 20 IL tasks of 5 classes. At test time, task IDs are not available.