SlowGuess/NeurIPS_2023_OCR huggingface dataset:
FSCIL datasets and setup. We conduct experiments on three publicly available datasets: 1) CIFAR100 (described above); 2) miniImageNet [58] - consisting of 100 classes,  $84 \times 84$  pixel images with 500 and 100 images per class for training and testing, respectively; 3) Caltech-UCSD Birds-200-2011 (CUB200) [59] - consisting of 200 classes,  $224 \times 224$  pixel images with 5994 and 5794 images for training and testing, respectively. For CIFAR100 and miniImageNet, we divide the 100 classes into 60 base classes and 40 new classes. The new classes are formulated into 8-step 5-way 5-shot incremental tasks. For CUB200, we divide the 200 classes into 100 base classes and 100 new classes. The new classes are formulated into 10-step 10-way 5-shot incremental tasks.
Compared methods. We compare with several exemplar-free CIL methods in the many-shot setting [25, 44, 3, 21, 32, 68, 79, 78, 80, 43] and in the few-shot setting [58, 53, 70, 76, 8, 31, 72, 42]. Results of compared methods marked with  $*$  are reproduced. For the upper bound of CIL, a joint training on all data is presented as a reference.
Implementation details. We use PyCIL [73] framework for our experiments. For both MSCIL and FSCIL settings, the main network architecture is ResNet-18 [18] trained on the first task using SGD with an initial learning rate of 0.1 and a weight decay of 0.0001 for 200 epochs. For the shrinkage, we use  $\gamma_{1} = 1$  and  $\gamma_{2} = 1$  for many-shot CIL and higher values  $\gamma_{1} = 100$  and  $\gamma_{2} = 100$  for few-shot CIL in our experiments. Following most methods, we store all the class prototypes. Similar to [78], we also store the covariance matrices for all classes seen until the current task. In the experiments with visual transformers, we use ViT-B/16 [15] architecture pretrained on ImageNet-
Table 1: Average top-1 incremental accuracy in exemplar-free many-shot CIL with different numbers of incremental tasks. Best results - in bold, second best - underlined.
CIL Method	CIFAR-100	TinyImageNet	ImageNet-Subset
T=5	T=10	T=20	T=5	T=10	T=20	T=5	T=10	T=20
EWC [25]	24.5	21.2	15.9	18.8	15.8	12.4	-	20.4	-
LwF-MC [44]	45.9	27.4	20.1	29.1	23.1	17.4	-	31.2	-
DeeSIL [3]	60.0	50.6	38.1	49.8	43.9	34.1	67.9	60.1	50.5
MUC [32]	49.4	30.2	21.3	32.6	26.6	21.9	-	35.1	-
SDC [68]	56.8	57.0	58.9	-	-	-	-	61.2	-
PASS [79]	63.5	61.8	58.1	49.6	47.3	42.1	64.4	61.8	51.3
IL2A [78]	66.0	60.3	57.9	47.3	44.7	40.0	-	-	-
SSRE [80]	65.9	65.0	61.7	50.4	48.9	48.2	-	67.7	-
FeTrIL* [43]	67.6	66.6	63.5	55.4	54.3	53.0	73.1	71.9	69.1
Eucl-NCM	64.8	64.6	61.5	54.1	53.8	53.6	72.2	72.0	68.4
FeCAM (ours) - Σ1:t	68.8	68.6	67.4	56.0	55.7	55.5	75.8	75.6	73.5
FeCAM (ours) - Σy	70.9	70.8	69.4	59.6	59.4	59.3	78.3	78.2	75.1
Upper Bound	79.2	79.2	79.2	66.1	66.1	66.1	84.7	84.7	84.7
21k [52]. The extracted features are 512 dimensional when using Resnet-18 and 768 dimensional when using pretrained ViT. More implementation details for all hyperparameters are provided in the supplementary material.
4.2.1 Many-shot CIL Results
The results for an exemplar-free MSCIL setup are presented in Table 1. We present the results for a set of different MSCIL methods, joint training of a classifier, and a simple NCM classifier (Eucl-NCM) on a frozen backbone. FeCAM outperformed all others by a large margin in all settings. FeCAM version with  $\Sigma^{1:t}$ , storing a single covariance matrix representing all classes, already gave significantly better results than the current state-of-the-art method - FeTrIL. However, FeCAM with covariance matrix approximation  $\Sigma_y$  pushes the average incremental accuracy even higher and present excellent results. It is worth noticing that Eucl-NCM outperforms many existing CIL methods. Only FeTrIL performs better than Eucl-NCM on all datasets. In Fig. 6, we present accuracy curves after each task for ten task scenarios. SSRE has a lower starting point due to a different network architecture. The rest of the methods, despite having the same starting point, end up with very different accuracies at the last task. Eucl-NCM still presents more competitive results than SSRE. FeTrIL presents better performance but is still far from FeCAM with a common covariance matrix. The FeCAM with a covariance matrix per class outperforms all other methods starting from the first incremental task. Here, in comparison to common covariance matrix, we pay the price in memory and need to store covariance matrix per class. Despite storing a matrix per class, we have less memory overhead compared to exemplar-based methods and do not violate privacy concerns by storing images.
Additionally, we compare our method against popular exemplar-based CIL methods in Table 3, where the memory buffer is set to 2K exemplars. Our method outperforms all others that do not expand the model significantly (see #P column for the number of parameters after the last task). Only Dynamically Expandable Representation (DER), which grows the model almost six times, can outperform our method.
4.2.2 Experiments with pre-trained models
In Table 2 different settings for exemplar-free MSCIL are presented where we follow experimental settings of Learning-to-Prompt (L2P) method [63]. Here, all methods use a ViT encoder pre-trained on ImageNet-21K. L2P [63] is a strong baseline that does not train the encoder and learns an additional 46K prompt parameters. However, as Janson et al. [22] presented, a simple NCM classifier can perform better for some datasets in CIL, e.g., Split-ImageNet-R [19], Split-CIFAR100 [26] and for domain-incremental learning on CoRe50 [34].
We use the widely-used benchmark in continual learning, Split-CIFAR-100 which splits the original CIFAR-100 [26] into 10 tasks with 10 classes in each task unlike the other settings in Table 1, which have different task splits. Based on ImageNet-R [19], Split-ImageNet-R was recently proposed
(b) TinyImageNet
Figure 6: Accuracy of each incremental task for: (a) CIFAR100, (b) TinyImageNet, and (c) ImageNet-Subset and multiple MSCIL methods. We annotate the Avg  $\mathcal{A}$ cc. of all sessions between FeCAM and the runner-up method at the end of each curve. Here, prototype refers to NCM with euclidean distance.
(c) ImageNet-Subset
Table 2: Avg Acc or Test Acc at the end of the last task on class-incremental Split-Cifar100 [26], Split-ImageNet-R [19] and domain-incremental CoRe50 [34] benchmarks. All methods are initialized with pretrained weights from ViT-B/16 [15] for fair comparison.
CIL Method	Split-Cifar100	Split-ImageNet-R	CoRe50
Avg Acc	Avg Acc	Test Acc
FT-frozen	17.7	39.5	-
FT	33.6	28.9	-
EWC [25]	47.0	35.0	74.8
LwF [30]	60.7	38.5	75.5
L2P [63]	83.8	61.6	78.3
NCM [22]	83.7	55.7	85.4
FeCAM (ours)	85.7	63.7	89.9
Joint	90.9	79.1	-
Table 3: Comparison of our method with recent exemplar-based methods which store 2000 exemplars. For fair comparison, we show the number of parameters (in millions) by #P. Results on Imagenet-Subset excerpted from [74].
CIL Method	CIFAR-100 (T = 5)	ImageNet-Subset (T = 5)
#P	Ex.	Avg. Acc	Last Acc	Avg. Acc	Last Acc
iCaRL [44]	11.17	✓	65.4	56.3	62.6	53.7
PODNet [16]	11.17	✓	67.8	57.6	73.8	62.9
Coil [77]	11.17	✓	-	-	59.8	43.4
WA [71]	11.17	✓	69.9	61.5	65.8	56.6
BiC [64]	11.17	✓	66.1	55.3	66.4	49.9
FOSTER [60]	11.17	✓	67.9	60.2	69.9	63.1
DER [66]	67.02	✓	73.2	66.2	77.6	71.1
MEMO [75]	53.14	✓	-	-	76.7	70.2
FeCAM(ours)	11.17	✗	70.9	62.1	78.3	70.9
by [62] for continual learning which contains 200 classes randomly divided into 10 tasks of 20 classes each. It contains data with different styles like cartoon, graffiti and origami, as well as hard examples from ImageNet with a high intra-class diversity making it more challenging for CIL experiments. We use CoRe50 [34] for domain-incremental settings where the domain of the same class of objects is changing in new tasks. It consists of 50 different types of objects from 11 domains. The first 8 domains are used for learning and the other 3 domains are used for testing. Since it has a single test task, we report the test accuracy after learning on all 8 domains similar to [63, 22]. Results of compared methods excerpted from [22].
We use the proposed FeCAM method with the pre-trained ViT using a covariance matrix per class on CIL settings. In the domain-incremental setting, we maintain a single covariance matrix per class across domains and update the matrix in every new domain by averaging the matrix from the previous domain and from the current one. FeCAM outperformed both L2P and NCM, in all the settings. Notably, FeCAM outperforms L2P by  $11.55\%$  and NCM by  $4.45\%$  on the CoRe50 dataset.
4.2.3 Few-shot CIL Results
In many-shot settings, it is possible to obtain a very informative covariance matrix from the large number of available samples, while it can be difficult to get a good matrix in FSCIL from just 5 samples per class. To stabilize the covariance estimation, we use higher values of  $\gamma_{1}$  and  $\gamma_{2}$ . In Fig. 7 we present accuracy curves after each task for FSCIL on miniImageNet, CIFAR100, and CUB200 datasets. While TOPIC [53] does better than the finetuning methods, it does not perform well in comparison to the recent methods which has significantly improved the performance. ALICE [42] recently proposed to obtain compact and well-clustered features in the base task which helps in generalizing to new classes. We follow the experimental settings from ALICE [42]. We take the strong base model after the first task from ALICE and use the proposed FeCAM classifier (with a covariance matrix per class) instead of NCM classifier in the incremental tasks. We outperform ALICE significantly on all the FSCIL benchmarks.
Figure 7: FSCIL methods accuracy of each incremental task for (a) miniImageNet, (b) CIFAR100 and (c) CUB200. We annotate the performance gap after the last session and Avg Acc. of all sessions between OURs and the runner-up method at the end of each curve. Refer to supplementary material for detailed values.
Table 4: Ablation of the performance indicating the contribution from the different components of our proposed method FeCAM for MSCIL with five tasks on CIFAR-100 and ImageNet-Subset datasets. Note that here we use variance normalization (different from Eq. (7)) when using diagonal matrix.
Distance	Cov. Matrix	Tukey Eq. (9)	Shrinkage Eq. (8)	Norm. Eq. (7)	CIFAR-100 (T=5)	ImageNet-Subset (T=5)
Last Acc	Avg Acc	Last Acc	Avg Acc
Euclidean	-	X	-	-	51.6	64.8	60.0	72.2
Euclidean	-	✓	-	-	54.4	66.6	66.2	73.6
Mahalanobis	Full	X	X	X	14.6	29.7	33.5	45.1
Mahalanobis	Full	✓	X	X	20.6	36.2	54.0	65.6
Mahalanobis	Full	X	✓	X	44.6	59.3	39.9	56.9
Mahalanobis	Full	✓	✓	X	52.1	62.8	56.5	67.3
Mahalanobis	Diagonal	✓	✓	X	55.2	66.9	64.0	74.1
Mahalanobis	Full	X	✓	✓	55.4	65.9	58.1	68.5
Mahalanobis	Full	✓	✓	✓	62.1	70.9	70.9	78.3
4.2.4 Ablation Studies
FeCAM propose to use multiple different components to counteract the CIL effect on the classifier performance. In Table 4 the ablation study for MSCIL is presented, where contribution of each component is exposed in a meaning of average incremental and last task accuracy. Here, we consider the settings using a covariance matrix per class. We show that Tukeys transformation significantly reduces the skewness of the distributions and improves the accuracy using both Euclidean and Mahalanobis distances. The effect of the covariance shrinkage is more significant for CIFAR100 which has 500 images per class (less than 512 dimensions of feature space) while it also improves the performance on ImageNet-Subset. Here, we also show the usage of the diagonal matrix where we use only the diagonal (variance) values from the covariance matrices. We divide the diagonal matrices by the norm of the diagonal to normalize the variances. When using the diagonal matrix, the storage space is reduced from  $D^2$  to  $D$ . Finally, we show that using the correlation normalization from Eq. (7) gives the best accuracy by tackling the variance shift and better using the feature covariances.
Time complexity. While previous methods train the classifier [43] or the model [79, 80] for several epochs in the new tasks, we do not perform any such training in new tasks. Among the existing methods, FeTrIL [43] claims to be the fastest. We compare the time taken for the incremental tasks on ImageNet-Subset  $(\mathrm{T} = 5)$  for FeTrIL and the proposed FeCAM method. Using one Nvidia RTX 6000 GPU, FeTrIL takes 44 minutes to complete all the new tasks while FeCAM takes only 6 minutes.
Feature transformations. We analyze the t-SNE plot for the feature distributions of old and new classes from CIFAR100 50-50 (2 tasks) setting in different scenarios in Fig. 8. When the model is trained jointly on all classes, the features of all classes are well clustered and separated from each other. In CIL settings with frozen backbone when the model is trained on only the first 50 classes, the features are well-clustered for the old classes while the features for new classes are scattered and not well-separated. When we make the feature distributions more gaussian using Tukeys transformation, we observe that the new class features are comparatively better clustered.
(a) Joint-Training
Figure 8: The t-SNE plot for the features of new and old classes after Joint-Training (a) and after learning only the first 50 classes (b,c). In Joint-Training, the features are well clustered for all classes, however when the feature extractor is trained only on the first 50 classes, the new class representations are spread out. On applying Tukeys transformation, the new class embeddings are better clustered.
(b) CL-Frozen backbone
(c) CL-Frozen backbone with Tukeys transformation
Table 5: Analysis of performance when the first task has 20 classes only and 20 new classes are added in incremental tasks, on CIFAR-100 and ImageNet-Subset datasets.
Method	CIFAR-100	ImageNet-Subset
Last Acc	Avg Acc	Last Acc	Avg Acc
Euclidean-NCM	30.6	50.0	35.0	54.5
FeTrIL	46.2	61.3	48.4	63.1
FeCAM (ours)	48.1	62.3	52.3	66.4
CIL settings with a small first task. Usually one method sticks to one setting. Exemplar-free methods use  $50\%$  of data in the first task as equally splitting is a much more challenging setting which is usually tackled by storing exemplars or by expanding the network in new tasks.
When half of the total classes is present in the first task, the feature extractor is better. When we start with fewer classes (20 classes in the first step) and add 20 new classes at every task, we can observe the same behavior in Table 5. FeCAM still works and outperforms other methods. However, the average incremental accuracy is not very high in this challenging setting because the representation learned in the first task is not as good as in the big first task setting.
In this paper, we revisit the anisotropic Mahalanobis distance for exemplar-free CIL and propose using it to effectively model covariance relations for classification based on prototypes. We analyze the heterogeneity in the feature distributions of the new classes that are not learned by the feature extractor. To address the feature distribution shift, we propose our Bayes classifier method FeCAM, which uses Mahalanobis distance formulation and additionally uses some techniques like correlation normalization, covariance shrinkage, and Tukey's transformation to estimate better covariance matrices for continual classifier learning. We validate FeCAM on both many- and few-shot incremental settings and outperform the state-of-the-art methods by significant margins without the need for any training steps. Additionally, FeCAM evaluated in the class- and domain-incremental benchmarks with pretrained vision transformers yields state-of-the-art results. FeCAM does not store any exemplars and performs better than most exemplar-based methods on many-shot CIL settings. As a future work, FeCAM can be adapted to CIL settings where the feature representations are continually learned.
Limitations. The proposed approach needs a strong feature extractor or a large amount of data in the first task to learn good representations, as we do not learn new features but reuse the ones learned on the first task (or from pretrained network). Therefore, the method is not apt when training from scratch, starting with small tasks. We would then need to extend the theory to feature distributions which undergo feature drift during training; next to prototype drift [68] also covariance changes should be modeled.
Acknowledgement. We acknowledge projects TED2021-132513B-I00 and PID2022-143257NB-I00, financed by MCIN/AEI/10.13039/501100011033 and FSE+ and the Generalitat de Catalunya CERCA Program. Bartlomiej Twardowski acknowledges the grant RYC2021-032765-I.