Incremental few-shot learning via vector quantization in deep embedded space
Authors: Kuilin Chen
Chi-Guhn Lee
Venue: ICLR 2021 Poster (2021)
Keywords: incremental learning
few-shot
vector quantization

Abstract: The capability of incrementally learning new tasks without forgetting old ones is a challenging problem due to catastrophic forgetting. This challenge becomes greater when novel tasks contain very few labelled training samples. Currently, most methods are dedicated to class-incremental learning and rely on sufficient training data to learn additional weights for newly added classes. Those methods cannot be easily extended to incremental regression tasks and could suffer from severe overfitting when learning few-shot novel tasks. In this study, we propose a nonparametric method in deep embedded space to tackle incremental few-shot learning problems. The knowledge about the learned tasks are compressed into a small number of quantized reference vectors. The proposed method learns new tasks sequentially by adding more reference vectors to the model using few-shot samples in each novel task. For classification problems, we employ the nearest neighbor scheme to make classification on sparsely available data and incorporate intra-class variation, less forgetting regularization and calibration of reference vectors to mitigate catastrophic forgetting. In addition, the proposed learning vector quantization (LVQ) in deep embedded space can be customized as a kernel smoother to handle incremental few-shot regression tasks. Experimental results demonstrate that the proposed method outperforms other state-of-the-art methods in incremental learning.

Published as a conference paper at ICLR 2021
INCREMENTAL FEW-SHOT LEARNING VIA VECTOR QUANTIZATION IN DEEP EMBEDDED SPACE
Kuilin Chen
Chi-Guhn Lee
Department of Mechanical and Industrial Engineering
Department of Mechanical and Industrial Engincering University of Toronto Toronto, Ontario, Canada cglee@mie utoronto.ca
University of Toronto
Toronto, Ontario, Canada
muilin. chen@ma
beenmmii..utoooooooooo wtw.onto..oooooo
ABSTRACT
The capability of incrementally learning new tasks without forgetting old ones is a challenging problem due to catastrophic forgetting. This challenge becomes greater when novel tasks contain very few labelled training samples. Currently, most methods are dedicated to class-incremental learning and rely on sufficient training data learn additional weights for newly added classes. Those methods cannot be easily extended to incremental regression tasks and could suffer from severe overfitting when learning few-shot novel tasks. In this study, we propose nonparametric method in deep embedded space to tackle incremental few-shot learning problems. The knowledge about the learned tasks is compressed into a small number of quantized reference vectors. The proposed method Iearns new tasks sequentially by adding more reference vectors to the model using few-shot samples in each novel task. For classification problems, we employ the nearest neighbor scheme to make classification on sparsely available data and incorpo- intra-class variation, less forgetting regularization and calibration of reference vectors to mitigate catastrophic forgetting. In addition, the proposed learning vec- tor quantization (LVQ) deep embedded space can be customized as a kernel smoother to handle incremental few-shot regression tasks. Experimental results demonstrate that the proposed method outperforms other state-of-the-art methods in incremental learning.
INTRODUCTION
Incremental learning is a learning paradigm that allows the model to continually learn new tasks on novel data, without forgetting how to perform previously learned tasks (Cauwenberghs & Poggio 2001 Kuzborskij et al. 2013 Mensink et al. 20133). The capability of incremental learning becomes more important in world applications,
which the deployed models exposed possible out-of-sample data. Typically, hundreds of thousands of labelled samples in new tasks are required re-train or fine-tune the model (Rebuffi et 2017). Unfortunately, it is impractical to gather sufficient samples of new tasks in real applications. In contrast, humans can learn new concepts from just one few examples, without losing old knowledge. Therefore, it is desirable develop algorithms to support incremental learning from very few samples.
While a natural approach for incremental few-shot learning to fine-tune part of the base model using novel training data (Donahue et al., 2014] Girshick et al., [0144). the model could suffer from severe over-fitting on new tasks due a limited number of training samples. Moreover, simple fine-tuning also leads to significant performance drop on previously learned tasks, termed as catas- forgeicfo (Goodfellow et al. 2014). Recent attempts to mitigate the catastrophic forgetting generally categorized into two streams: memory relay of old training samples (Rebuffi et al. 2017 Shin et al.] 2017 Kemker & Kanan 2018)) and regularization on important model parameters Kirkpatrick al. 2017)] Zenke et 2017). However, those incremental learning approaches developed and tested on unrealistic scenarios where sufficient training samples are available in novel tasks. They may work well when the training samples in novel tasks are few (Tao et al. 2020b).

---

Published as a conference paper at ICLR 2021
To the best of our knowledge, the majority of incremental learning methodologies focus classi- fication problems and they cannot be extended to regression problems easily. In caaas-incrrmmntll learning, the model has expand output dimensions to learn N' novel classes while keeping the knowledge of existing N classes. Parametric models estimate additional classification weights for novel classes, while nonparametric methods compute the class centroids for novel classes. In com- parison, output dimensions in regression problems do not change in incremental learning as neither additional weights nor class centroids are applicable to regression problems.
Besides, we find that catastrophic forgetting in incremental few-shot classification can be attributed to three reasons. First, the model is biased towards new classes and forgets old classes because the model is fine-tuned on new data only (Hou et al. [019 ZZaa et 2020). Meanwhile, the prediction accuracy on novel classes is not good due to over-fitingg on few-shot training samples. Second, features of novel samples could overlap with those of old classes in the feature space, leading to ambiguity among classes in the feature space. Finally, features of old classes and classification weights are no longer compatible after the model is fine-tuned with new data.
In this paper, we investigate the problem of incremental few-shot learning, where only a few train- ing samples are available in new tasks. A unified model is learned sequentially to jointly recognize all classes or regression targets that have been encountered in previous tasks (Rebuffi et al.) 2017; Wu et al. 2019p. To tackle aforementioned problems, we propose nonparametric method han- dle incremental few-shot learning based on learning vector quantization (LVQ) (Sato & Yamada 1996) in deep embedded space. As such, the adverse effects of imbalanced weights in paramet- ric classifier can be completely avoided (Mensink et al. 2011 Snell et al., 2017]: Yu et al.) 2020). Our contributions are three fold. First, a unified framework is developed, termed as incremental deep learning vector quantization (IDLVQ), to handle both incremental classification (IDLVQ-C) and regression (IDLVQ-R) problems. Second, we develop intra-class variance regularization, less forgetting constraints and calibration factors to mitigate catastrophic forgetting in learning. Finally, the proposed methods achieve state-of-the-art performance on incremental few- shot classification and regression datasets.
2 RELATED WORK
Incremental learning: Some incremental learning approaches rely memory replay of old exem- plars to prevent forgetting previously learned knowledge. Old exemplars can be saved in memory (Rebuffi et al., 2017: Castro et al. 2018, Prabhu et al. 2020) or sampled from generative models (Shin et al. 2017] Kemker& Kanan 2018 van de Ven et al. 2020). However, explicit storage of training samples is not scalable if the number of classes is large. Furthermore, it is difficult to train a reliable generative model for all classes from very few training samples. In parallel, regularization network weights or outputs
approaches do not require old exemplars and impose regularization to minimize the change of parameters that are important to old tasks (Kirkpatrick et al., 2017] Zenke eet al., (017). To avoid quick performance deterioration after learning a sequence of novel tasks in regularization approaches, semantic drift compensation (SDC) is developed by learning an em- bedding network via triplet loss (Schroff et al. 2015) and compensates the drift of class centroids using novel data only YYu et al., 2020).. In comparison, IDLVQ-C saves only one exemplar per class and uses saved exemplars to regularize the change in feature extractor and calibrate the change in reference vectors.
Few-shot learning: Few-shot learning attempts obtain models for classification or regression
tasks with only few labelled samples. Few-shot models are trained on widely-varying episodes of fake few-shot tasks with labelled samples drawn from a large-scale meta-trainin dataset (Vinyals
et al. 2016 Finn et al.) 2017: Ravi & Larochelle, 2017; [Snell et al.] 2017, Sung et al. 2018). Meanwhile, recent works attempt to handle novel few-shot tasks while retraining the Enowledge of
the base task. These methods are referred to as dynamic few-shot learning (Gidaris & Komodakis 2018 Ren et al. 2019a] Gidaris & Komodakis, 2019). However, dynamic few-shot learning is different from incremental few-shot learning, because they rely on the entire base training dataset and an extra meta- meta-training dataset during meta-training In addition, dynamic few-shot learning does not accumulate knowledge for multiple novel tasks sequentially.
Incremental few-shot learning: Prior works on incremental few-shot learning focus on classifica- tion problems by computing the weights for novel classes in parametric classifiers, without iterative
2

---

Published as a conference paper at ICLR 2021
gradient descent. For instance, the weights of novel classes can be imprinted by normalized pro- totypes of novel classes, while keeping the feature extractor fixed (Qi et al. 2018). Since novel weights are computed only with the samples of novel classes, the fixed feature extractor may not be compatible with novel classification weights. More recently, neural gas network is employed to con- struct an undirected graph represent knowledge of old classes (Tao al. 2020b a). The vertices in the graph are constructed in an unsupervised manner using competitive Hebbian learning (Fritzke) 1995), while the feature embedding is fixed. In contrast, IDLVQ learns both feature extractor and reference vectors concurrently in supervised manner.
3 BACKGROUND
3.1 INCREMENTAL FEW-SHOT LEARNING
In this paper, incremental few-shot learning is studied for both classification and regression tasks. For classification tasks, consider the standard class-incremente setup in literature. After the model is trained on a base task (t = 1) with sufficient data, the model learns novel tasks sequentially. Each novel task contains number of novel classes with only few training samples per class. Learning a novel task (t > 1) is referred to as an incremental learning session. In task i, we have access only to training data D in the current task and previously saved exemplars (one exemplar per class in this study). Each task has a set of classes C = {ch...., c J. where n, is the number of classes in task
addition, it is assumed that there no overlap between classes in different tasks C" N = Ø for t * s. After an incremental learning session, the performance of the model evaluated on a test set that contains all previously seen classes = n C". Note that our focus is nof multi-task scenario, where a task ID is exposed to the model during test phase and the model is only required to perform given task one time (van de Ven & Tolias, 2019). Our model evaluated in agg-aggostic setting, where task ID is not exposed to the model at test time.
For regression tasks, we follow a similar setting with a notable difference that the target is real- valued y E R. In addition, the target values in different tasks do not have to be mutually exclusive, unlike the inseemnaremttall setup.
3.2 LEARNING VECTOR QUANTIZATION
Traditional nonparametric methods, such as nearest neighbors, represent knowledge and make pre- dictions by storing the entire training set. Despite the simplicity and effectiveness, they are not scalable to a large-scale base dataset. Typically, incremental learning methods are only allowed to store small number of exemplars to preserve the knowledge of previously learned tasks. However, randomly selected exemplars may not well present the knowledge in old tasks. LVQ is a classical data compression method that represents the knowledge through few learned reference vectors (Sato & Yamada 1996, Seo & Obermayer, 2003 Biehl et al. 2007). A new sample is classified to the same label as the nearest reference vector in the input space. LVQ has been combined with deep feature extractors as an alternative to standard neural networks for better interpretability (De Vries) al., 2018). The combinations of LVQ and deep
al., 2016 Villmann et al.) 2017 Saralajew Teature extractors have been applied to natural language processing (NLP), facial recognition and biometrics (Variani et al. 2015; Wang et al., [016 Ren et al., 2019b; Leng et al. 2015). We notice that LVQ is a nonparametric method which is well suited for incremental few-shot learning because the model capacity grows by incorporating more reference vectors to learn new knowledge. For example, incremental learning vector quantization (ILVQ) has been developed to learn classification models adaptively from raw features (Xu et al. 2012). In this study, we present the knowledge by learning reference vectors in the feature space through LVQ and adapt them in incremental few-shot learning. Compared with ILVQ by Xu et al. (2012), our method does not rely on predefined rules to update reference vectors and can be learned along with deep neural networks in an end-to-ena fash- ion. Besides, our method uses a single reference vector for each class, while ILVQ automatically assigns different numbers of prototypes for different classes.
3

---

Published as a conference paper at ICLR 2021
4 METHODOLOGY
4.1 INCREMENTAL DEEP LEARNING VECTOR QUANTIZATION
The general framework of IDLVQ for both classification and regression can be derived from Gaus- sian mixture perspective (Ghahramani & Jordan 1994). with a simplified covariance structure and supervised deep representation learning. In the base dataset (t 1), raw imput x is projected into feature space F! by deep neural network far, where 0 denotes the parameters in neural networks. In addition, N reference vectors M' {m,,... x } are placed in the feature space F!, which can be learned to capture the representation of the base dataset. More reference vectors will be added incrementally while learning novel tasks. The marginal distribution p(So (x)) of feature vector can
be described by Gaussian mixture model p(Je= Σi= (((i)p(Je (x) of N components. where the prior p(i) 1/N4 and the component distribution p(Je- (x)] is Gaussian. By assuming that each component distribution pf (x)li) is isotropic Gaussian centered at m with the same covariance, the posterior distribution of a component given the input is
k(fox (x), m!)
p (i[x)
(1)
Σ (for (x), (x), m!)
where x(fo m;) exp(-Ilfoixx) - is Gaussian kernel and is a scale factor. The conditional expectation of the output from a Gaussian mixture is و = Σi p² (ilx)z' where qf is the reference target associated with reference vector m.. In classification problems, gf is either 0 or 1 indicating whether m. and have the same label. Since each reference vector is assigned to class at initialization, 9: is fixed and does not require learning. Meanwhile, q: in regression problems is real-valued and has to be learned. The weights in neural networks 6¹ reference vectors M', reference targets of (in regression problems only) and the scale factor are learned concurrently by minimizing loss function between the true label y and the predicted label
The proposed IDLVQ is a nonparametric method as it makes prediction based on similarity to ref- erence vectors, instead of using any regression or classification weights. The capacity of the model grows naturally by adding more reference vectors to learn novel tasks, while the old knowledge is preserved in existing reference vectors.
4.2 INCREMENTAL DEEP LEARNING VECTOR QUANTIZATION FOR CLASSIFICATION
For classification problems, one reference vector is assigned to each class in our study. Thus, I rep- resents the predicted probability that an input belongs to a class. The model can be trained to classify data correctly by minimizing the cross-entropy loss LCE between the predicted probability ú and the true label y. Although the cross-entropy loss encourages separability of features in base classes, it does not guarantee compact intra-class variation in the feature space. Specifically, in an incremen- tal learning session, features of novel classes could overlap with those of previously learned classes. As result, the overall classification accuracy could deteriorate after incremental learning sessions. A desirable feature embedding leaves large margin between classes to mitigate overlap in features across old and new classes. Inspired by center loss (Wen et al., 2016) to enhance discriminative capability in facial recognition, a regularization term on intra-class distance to reference vectors is added to get compact intra-class variation.
Lintra
Σ (x) -m; v(x.8)_==i
(2)
As such, Jo (x) is forced to stay close to the reference vector with the same label and naturally moves away from other reference vectors. Consequently, features of new classes are more likely to lie in the margin between old classes to mitigate ambiguity in features across different classes. The total loss in training the base task is given by L = LcE + Cntrre where Aintra is a hyper-parameter to control the weight for intra-class variation loss. The total loss is differentiable w.r.t. neural network parameters g¹, reference vectors M = {m],... m. } and scaling factor y. All parameters in the model can be trained jointly in an end-to-end fashion.
In an incremental session (t > 1), a novel dataset D' contains n' classes and K' samples per class way K' K'-shot). n' new reference vectors are added and each reference vector is initialized
4

---

Published as a conference paper at ICLR 2021
as the centroid of features in a class m = Κ. Σk= So (xx). The new reference vectors along with the neural network parameters are fine-tuned on Dr to learn new knowledge in task t. To preserve the knowledge from the old tasks during incremental learning, the model should be updated only when necessary. Therefore, cross-entropy loss not used in incremental learning sessions because it always updates model parameters even if the sample is correctly classified. Let m², be the reference vector with the correct label and m' be the nearest reference vector with wrong
label. For a training sample (x, v) in D', the sample is classified correctly if ||. (x) - m 11 < for (x) m
fo (x) - m' ||a In this case, the loss should be O. When [[ - m'; the sample is misclassified. We adapt the margin based loss function LM from De Vries et al. (2016)
with a minor modification
LM ReLU [1160((3) m 112 | (x) - m ||
(3)
[() m²_ 112 [l m
where ReLU(-) stands for the rectified linear unit function. The margin based loss leads to slow training convergence because it only updates two reference vectors one time. However, the adapted margin based loss is well suited in learning from few-shot samples while avoids unnecessary param- eter updates.
Features for an old class could deviate away from the corresponding reference vector due to changes in 0 during incremental learning, leading catastrophic forgetting. A forgetting loss LF devel- oped to regularize the drift in the feature space
N CF CF= = Σ ( - fa-
(4)
where x; is the selected exemplar for class and N denotes the total number of classes in the base task and all previous novel tasks. Note that the exemplar x, for class i E [N NN) is picked from D' whose feature is nearest to m at the end of each learning session. The total loss in the incremental learning session t is L = LM + & Lintra where XP and Xintra are weights for forgetting loss and intra-class variation loss, respectively. The total loss is optimized w.r.t. neural network parameters 0 and new reference vectors [m (m'x--441- m }.
The reference vectors for previously learned tasks are not updated by novel data to prevent catas- trophic forgetting. However, they may not be well suited to represent knowledge and make classi- fication in the new feature space Fe as feature embedding is changed with updated 0¹. Although the true optimal location of those reference vectors are difficult to estimate without using the entire data from all tasks, they can be calculated approximately using the shift in features of exemplars. Considering that features of an exemplar x; are close to m; in the feature space, the shift of a refer- ence vector 8f in the new feature space can be approximated by the shift of the exemplar's features of for (x.) -foo (x²). Therefore, the reference vectors for previously learned tasks are calibrated mf = m + 6% where m! is the uncalibrated reference vector for class E [1, Nt-j.. A test sample, which could be from any seen classes, is classified according to the distance to reference vectors {m/, .... mfy:). }. The pseudo code for IDLVQ-C is presented in the appendix.
4.3 INCREMENTAL DEEP LEARNING VECTOR QUANTIZATION FOR REGRESSION
For regression problems, the model is trained to recognize regression targets by the minimizing mean squared error (MSE) loss LMSE (y 3)², where y is the real-valued target in training dataset. The MSE loss function differentiable w.r.t. neural network weights, reference vectors and targets, and scale factor. Therefore, all parameters can be trained jointly in end-to-end manner. The proposed IDLVQ-R can also be interpreted as a kernel smoother in deep embedded space. Compared with traditional kernel smoother, such as Nadaraya- Aadaraya-Watson Watson estimator (Nadarayal 1964), IDLVQ-R is sparse and hence more scalable as it only relies on a few reference vectors and targets.
In incremental learning session (t 1), we have access to data D that contains K" pairs of training samples (x(.8() nt new reference vectors (n' < Kt) along with corresponding targets are added to the model to learn new knowledge in the novel task f. We randomly select nz samples from
5

---

Published as a conference paper at ICLR 2021
D' to initialize reference vectors and targets follows
M = -x(()))
(5)
gi+N--1 y/.
(6)
where N is the total number of reference vectors in all previous tasks. The new reference vectors and targets are fine-tuned by minimizing MSE on 2' while keeping other parameters frozen. After new reference vectors and targets are fine-tuned with novel data D', the model makes prediction by smoothing targets of all reference vectors
DEE x(fo(x), m;)g:
ÿ N Σ x(fo(x), m,)
where No is the current total number of reference vectors.
EXPERIMENTS
We first describe the overall protocols, then we present the results on incremental few-shot classifi- cation and regression problems.
5.1 INCREMENTAL FEW-SHIOT CLASSIFICATION
We empirically evaluate the performance of IDLVQ-C on incremental few-shot classification on CUB200-2011 (Welinder et al., 2010) and minilmaeeeet datasets (Vinyals et al. (016)). The dataset is split into base classes and multiple groups of novel classes. We apply standard data augmentation, including random crop, horizontal flip and color jitter, on all training images. After each training session, the model performance is evaluated on test set, which contains all classes that the model has been trained on.
CUB dataset is composed of 200 fine-grained bird species with 11,788 images. We split the dataset into 5894 training images, 2947 validation images and 2947 test images. All images are resized to 224 x 224. In addition, the first 100 classes are chosen as base classes, where all training samples in base classes are used to train the base model. The remaining 100 classes are treated novel cat- egories and split into 10 incremental learning sessions. Each incremental learning session contains 10 novel classes and randomly selected training samples per class (10-way 5-shot).
miniImageNNet dataset is a 100-class subset of the original ImageNet dataset (Deng et al. 2009). Each class contains 500 training images, 50 validation images, and 50 test images. The images are in RGB format of the size 84 x 84. We choose 60 and 40 classes for base and novel classes, respectfully. The 40 novel classes are divided into 8 sessions and each session contains 5 novel classes with randomly selected training samples per class (5-way 5-shot).
ResNet18 (He et al. 2016) is used as the feature extractor for incremental classification problems. The learning process for each dataset is repeated 10 times and the average test accuracy is reported. The proposed method is compared with six methods for few-shot slass-incremental learning: fine- turning using D'. joint training using the entire training set from all encountered classes, iCaRL (Rebuffi et al., 2017), Rebalancing (Hou et al. 2019), ProtoNet (Snell et al., 2017). incremental learning vector quantization (ILVQ) (Xu et al. 2012). SDC (Yu et al. 2020), and Imprint (Qi al., 2018). Note that ILVQ is applied to the features extracted by neural networks in our experiment.
The incremental few-shot learning results on CUB and minilmageNet are shown in Table and respectively. Our method outperforms fine-tuning, iCaRL (Rebuffi et al. (017)), and ProtoNet (Snell al., 2017) by a large margin. Simply fine-tuning the weights in classifier with few-shot training samples for novel classes significantly deteriorates the prediction accuracy. Although iCaRL alle- viates catastrophic forgetting by tuning the model with mix of old exemplars and novel few-shot data, the prediction accuracy still drops quickly because iCaRL requires sufficient samples per class to achieve satisfactory performance. The ProtoNet relies on distance to prototypes (the mean of features within class) to make classification but the fixed feature extractor may not be able to well separate novel classes. ILVQ is slightly better than ProtoNet because prototypes can be learned adaptively when more classes available in incremental learning sessions. Some prototypes in

---

Published as a conference paper at ICLR 2021
ILVQ are close to the border of a class, which are more effective than class centroids in ProtoNet. However, ILVQ does not achieve the best performance because the feature extractor is fixed and cannot be learned along with the prototypes. IDLVQ-C has small gain in the first couple of incre- mental few-shot learning sessions compared with SDC (Yu et al. (020) and Imprint (Qi et al. 2018). Similar to ProtoNet, SDC also relies prototypes make classification. The performance of SDC is better than that of ProtoNet because SDC fine-tunes the feature extractor with novel dataset and compensates the drift in prototypes. However, the compensation for the drift of old-class prototypes can be less accurate in SDC because it is approximated by samples in novel classes. In parallel, the imprint method directly computes the normalized classification weights from the average of nor- malized features within a novel class. The imprint method avoids imbalanced classification weights and circumvents the overfitting in few-shot class-incremental learning through weight normaliza- tion. Nevertheless, the fixed feature extractor in the imprint method may not be well suited for novel classes. In contrast, IDLVQ-C updates the feature extractor only when necessary and compensates the shift of old reference vectors more accurately using exemplars from old classes. That is why the gain of IDLVQ-C increases with more incremental few-shot learning sessions. The performance of SDC, Imprint and IDLVQ-C is better than offline joint training in early sessions of incremen- tal few-shot learning. Offline joint training may not result in oracle performance due to extremely imbalanced samples between base classes and novel classes.
Table 1: Prediction accuracy on CUB all classes using the 10-way 5-shot incremental setting.
Method
aossions
10
Fine tune
77.30
46.23
34.71
23.16
20.65
16.21
13.32
11.98
11.17
10.76
Juint train
77.30
73.28
63.75
62.00
60.81
59.06
58.69
56.23
ICaRL
(Rebuth
2017
77.30
57.18
54.67
48.11
40.76
36:85
53.12
30.42
28.22
26.84
25.23
Rebalancing
77.30
64.53
38.92
34.39
31.01
27.12
24.66
23.61
ProtoNet
Snell
2017
77.30
69.76
66.01
62:29
59.58
57.10
55.13
54.09
51.65
50.36
LLV
Xu
2012
77.30
71.50
66.79
66.20
57.84
55.27
55.06
52,42
51.72
50.47
SDC
Yu al.
2020
77.34
74.45
69:45
65.27
61,81
58.26
56.14
55,71
53,31
52.79
51.52
Imprint
O0t
2018
77.02
73.39
69.50
65.61
62.81
60.74
59.39
58.61
56.85
55,93
$4.82
IDLVQ.C
77.37
74.72
70,28
67.13
65.34
63.52
62.10
61.54
59.04
58.68
57,81
Table 2: Prediction accuracy on minimmageeet all classes using the 5-way 5-shot incremental setting.
Method
sessions
5
9
Fine-tune
64.25
30:11
18.53
6.31
2.36
2.68
1.87
1.56
1.42
Join: train
64.25
58.80
55.26
52.38
49.71
48,37
45.91
44.68
43.38
iCaRL (Rebutii et
2017
64.25
48.04
43.13
38:28
30.01
21.16
21.85
19.81
17.76
Rebalancing (foo ml
2019
64.25
49.21
44.17
37.71
30.11
22.92
19.99
17.96
16.25
ProwoMet Ssell al
2017
64.25
55.12
51.67
48.91
46.52
44.25
41.91
40,07
38:42
ILVD Xu et
201
64.25
56.01
52.43
49.31
46.98
44.37
42.06
40.11
38.43
SDC at.
X00
61.62
59.63
55.39
50.92
48.30
45.22
42.97
42.51
41.24
Imprint Q al
2018
64.71
59.85
55.71
52.47
49.90
47.3
44.57
42,57
41.26
IDIVQ-C
64.77
39.87
55.93
$2.62
49.88
47.55
44.83
43.14
41.84
Ablation studies are conducted to analyze how individual components affect the performance of incremental few-shot learning. We study five variants of our methods: (a) new reference vectors are initialized as class centroids and no tuning is done for feature extractor or old reference vectors; (b) Lintra is not used in incremental learning sessions; (c) Lp is not used in the incremental learning sessions; (d) shift in old reference vectors are not compensated; (e) replace the margin based loss L with the cross entropy loss LCg. Table shows the results of our ablation studies on CUB dataset. Without any fine-tuning, the initial reference vectors for novel classes lead to descent accuracy in incremental few-shot classification. It demonstrates the robustness of nonparametric classifier. Cintra leads to 0.57% gain due to tight intra-class variation. The less forgetting regularization CF is proved to prevent forgetting old classes and achieves a performance boost by 2.35%. The shift compensation for reference vectors effectively adapts old reference vector in new embedding spaces with a gain round of 1%. The margin based loss is more effective in preventing forgetting in early sessions of incremental learning.
The effect of the number of training samples per class, The proposed method is evaluated under different shot settings CUB dataset to investigate the effect of different numbers of training
7

---

Published as a conference paper at ICLR 2021
Table 3: Ablation study CUB using the 10-way 5-shot incremental setting.
Method
spseions
3
8
No tuning
71.93
67.14
64.21
62.61
66.11
99.04
58.47
55.64
54.25
53.66
w.a. Cintra
74.75
70.26
66.89
65.05
63.18
61.84
61.36
58.61
58.14
57:24
W.o. LF
73.85
09.54
66.21
64.02
62:74
00.28
39.49
56.97
56.38
55,46
w.a 6,
74.67
70.01
66.74
64.81
61.42
60.73
58
57.62
Ext
LCE
73.22
66:03
63.93
60.98
58.67
58.11
70.28
67.13
samples in novel classes, including 5-shot, 10-shot and 20-shot settings. One reference vector is assigned to each class in all few-shot settings, As shown in Fig. 2 in the appendix, the performance of incremental learning improves as the number of samples per class increases. When the training samples are scarce, the training samples may not well present the generative distribution of training data. Therefore, the learned reference vectors could be biased and classification accuracy is low. With more training samples, the learned reference vectors could well present the center of the distri- bution and classification accuracy improved. The gap performance becomes more obvious as the number of incremental learning sessions grows. The detailed results reported in Table 7 and
5.2 INCREMENTAL FEW-SHOT REGRESSION
IDLVQ-R is tested that there no state-of-the-art method for incremental few-shot regression, we compare IDIVQ-R against three alternative methods: fine-tuning using novel task data only, fine-tuning using novel task data along with exemplars and offline training using the entire training dataset from all tasks.
regression datasets: sinusoidal and 3D spatial data. Considering
Sinusoidal wave is defined by function y sin (3TTr) + 0.3 cos(9 + 0.5 sin (7xx) + c, where e is white noise with a standard deviation of 0. 1000 training samples in the first task (bas task) are generated by sampling x E [-1.0, 1.0] uniformly. 5-shot training samples in two novel tasks are generated by sampling x E 1.5] and x E [1.5, 2.0]. respectively.
As shown in Fig. [(a] IDLVQ-R achieves comparable performance offline neural networks in Fig. [(d) which are trained using the entire training set from all tasks. In comparison, neural networks trained sequentially with few-shot training samples show catastrophic forgetting on old tasks in Fig. [[b] With the addition of exemplars during training, the networks perform better still suffer from catastrophic forgetting on the base task in Fig. [[c] conclusion, IDLVQ-R preserves old knowledge and adapts to new knowledge quickly using a few reference vectors and
achieves satisfactory performance on incremental few-shot regression tasks. The experiment details can be found in the appendix.
(a)
(b)
(c)
(d)
Figure 1: Comparison of performance for incremental few-shot regression. Red dots denote test samples for base task, purple dots denotes test sample for novel tasks, grey lines denote model predictions, and black crosses denotes few-shot training samples in novel tasks. (a) IDLVQ-R; (b) neural networks incrementally fine-tuned with novel data only in each session; (c) neural networks incrementally fine-tuned with exemplars and novel training samples; (d) offline neural networks trained with training samples from all tasks.
3D spatial data is collected in North Jutland, Denmark. The inputs are longitude o and latitude 222 and the output altitude y. 2482 training samples in the 1st task are collected the area where
https://archit usi edu/mdummmmm hts////mmaaaa wwwhhaaccicaaaaa ne- http:///aamoocciiieaaaaaaaaaaaaa a
8

---

Published as a conference paper at ICLR 2021
11 E [9.98, 9.995] and 22 [57.0, 57.05]. 20 training samples in the 2nd task collected in the area where #1 E [9.995, 10.0] and 22 E [57.02, 57 57.03]. training samples in the 3rd task are collected in the area where 21 E [9.995, 10.0] and 22 [57.03, Approximately 300 test
samples are collected for each task.
Table 4: Normalized RMSE of incremental few-shot regression 3D spatial data
sessions
Method
Joint train offline
0.02174(2e-))
0.02232(2e4)
0.02296(2e-)
Fine-tune novel data
0.021740o-4)
0.08462(4c-4)
0.11870(6e-4)
Fine-tune exemplars
0.02174(2e44
0.02988(2e-4)
0.03128((e-4)
IDLVQ-R
0.02181(2e-4)
0.02641(2e-4)
0.02817(2--4)
The normalized root mean squared errors (RMSE) between actual and predicted altitude in the test set are listed in Table 4 The prediction accuracy drops significantly when the model fine- tuned with novel data only. Catastrophic forgetting can be alleviated using exemplars from previous tasks. IDLVQ-R achieves better results than fine tuning with exemplars. The good performance of IDLVQ-R can be attributed two reasons. First, IDLVQ-R learns number of reference vectors and targets to preserve the knowledge in encountered tasks. Compared with a linear layer on top of neural networks, number of reference vectors represent richer information about the training data. Second, IDLVQ-R is nonparametric and can represent local and nonlinear relationship without learning any regression coefficient from few-shot data.
6 CONCLUSIONS
A new incremental few-shot learning approach is developed to harmonize old knowledge preserving and new knowledge adaptation through quantized vector in deep embedded space. Prediction made in nonparametric way using similarity to learned reference vectors, which circumvents biased weights parametric classification layer during incremental few-shot learning. For classification problems, additional mechanisms are developed to mitigate the forgetting in old classes and improve representation learning for few-shot novel classes. For regression problems, the proposed approach has been reinterpreted as kernel smoother to predict real-valued target over novel domain.
REFERENCES
Michael Biehl, Anarta Ghosh, and Barbara Hammer. Dynamics and generalization ability of Ivq algorithms. Journal Machine Learning Research, 8(Feb):323-360, 2007.
Francisco M Castro, Manuel Marin-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteck Alahari. End-to-end incremental learning. In Proceedings of the European conference on computer vision (ECCV), pp. 233-248, 2018.
Gert Cauwenberghs and Tomaso Poggio. Incremental and decremental support vector machine learning. In Advances in neural information processing systems, pp. 409-415, 2001.
Harm De Vries, Roland Mcmisevic, and Aaron Courville. Deep learning vector quantization. In ESANN, 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-248222 255..25..22 Icee, 2009.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In Inter- national conference on machine learning, pp. 647-655, 2014.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learming for fast adaptation of deep networks. Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1126-1135. JMLR. org, 2017.

---

Published as a conference paper at ICLR 2021
Bernd Fritzke. A growing neural gas network learns topologies. In Advances in neural information processing systems, pp. 625-632, 1995.
Zoubin Ghahramani and Michael Jordan. Supervised learning from incomplete data via an em approach. In Advances in neural information processing systems, pp. 120-127, 1994.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4367- 4375, 2018
Spyros Gidaris and Nikos Komodakis. Generating classification weights with gnn denoising autoen- coders for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 21-30, 2019.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for ac- curate object detection and semantic segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 580-587, 2014.
Ian Goodfellow, Mehdi Mirza, Aaron Courville Da Xiao, and Yoshua Bengio. An empirical investigation of catastrophic forgeting in gradient based neural networks. In In Proceedings of International Conference on Learning Representations ICLR, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 831-839, 2019.
Ronald Kemker and Christopher Kanan. Fearnet: Brain-inspired model for incremental learning. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):521-3352 2017.
et al. Overcom-
II Kuzborskij. Francesco Orabona, and Barbara Caputo. From n to nt 1: Multiclass transfer incremental learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3358-3365, 2013.
Mengjun Leng, Panagiotis Moutafis, and Ioannis A Kakadiaris. Joint prototype and metric learning for set-to-set matching: Application to biometrics. In 2015 IEEE 7th International Conference on Biometrics Theory, Applications and Systems (BTAS), pp. 1-8. IEFF, 2015.
Thomas Mensink, Jakob Verbeek, Florent Perronnin, and Gabriela Csurka. Distance-based image classification: Generalizing to new classes near-zero cost. IEEE transactions on pattern anal- ysis and machine intelligence, 35(1 (1111))222-22333 2013.
2013.
Elizbar A Nadaraya. On estimating regression. Theory of Probability & Its Applications, 9(1): 141-142, 1964.
Ameya Prabhu, Philip HS Torr, and Puneet Dokania. Gdumb: simple approach that questions our progress in continual learning. In European Conference on Computer Vision, pp. 524-540. Springer, 2020.
Hang Qi, Matthew Brown, and David Lowe. Low-shot learning with imprinted weights. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5822-5830, 2018.
Sachin Ravi and Hugo Larochelle, Optimization as a model for few-shot learning. 5th Interna- fional Conference Learning Representations, ICLR 2017, 2017.
10

---

Published as a conference paper at ICLR 2021
Sylvestre-Alvins Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampeer. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Mengye Ren, Renjie Liao, Ethan Fetaya, and Richard Zemel. Incremental few-shot learning with attention attractor networks. In Advances in Neural Information Processing Systems, pp. 5275- 5285, 2019a.
Zhenwen Ren, Bin Wu, Quansen Sun, and Mingna Wu. Simultaneous learning of reduced prototypes and local metric for image set classification. Expert Systems with Applications, 134:102-111, 2019b.
Sascha Saralajew, Lars Holdijk, Maike Rees, and Thomas Villmann. Prototype-based neural net- work layers: incorporating vector quantization. In Advances neural information processing systems workshop poster, 2018.
Atsushi Sato and Keiji Yamada. Generalized learning vector quantization. In Advances in neural information processing systems, pp. 423-429, 1996.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 815-822 2015.
Sambu Seo and Klaus Obermayer. Soft learning vector quantization. Neural computation, 15(7): 1589-1604, 2003.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems, pp. 2990-2999, 2017.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pp. 4077-4087 2017.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning compare: Relation network for few-shot learning. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 1199-1208, 2018
Xiaoyu Tao, Xinyuan Chang, Xiaopeng Hong, Xing Wei, and Yihong Gong. class-incremental learning. In European Conference Computer Vision. Springer, 2020a.
Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang, Songlin Dong, Xing Wei, and Yihong Gong. Few- shot incremmaaall learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12183-12192, 2020b.
Gido M van de Ven and Andreas S Tolias. Three scenarios for continual learning. In Advances in Neural Information Processing Systems Continual Learning workshop. 2019.
Gido M de Ven, Hava T Siegelmann, and Andreas S Tolias. Brain-inspired replay for continual learning with artificial neural networks. Nature Communications, 11:4069, 2020.
Ehsan Variani, Erik McDermott, and Georg Heigold. A gaussian mixture model layer jointly op- timized with discriminative features within a deep neural network architecture. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4270-4274. IEEE, 2015.
Thomas Villmann, Michael Biehl, Andrea Villmann, and Sascha Saralajew. Fusion of deep learning architectures, multilayer feedforward networks and learning vector quantizers for deep classifica- tion learning. In 2017 12th international workshop on self-organizing maps and learning vector quantization, clustering and data visualization ( (WSOM), pp. 1-8. IEEE, 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.
Wen Wang, Ruiping Wang, Shiguang Shan, and Xilin Chen. Prototype discriminative learning for face image set classification. In Asian Conference on Computer Vision, pp. 344-360. Springer, 2016.
11

---

Published as a conference paper at ICLR 2021
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSS UCCD Birds 200. Technical Report CNS-TR MUSS-TR-2010-1 2010000-, California Institute of Technology. 2010.
Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Qiao. A discriminative feature learning approach for deep face recognition. In European conference on computer vision, pp. 499-515. Springer, 2016.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 374-382, 2019.
Ye Xu, Furao Shen, and Jinxi Zhao. An incremental learning vector quantization algorithm for pattern classification. Neural Computing and Applications, 21(6): 1215, 2012.
Lu Yu, Bartlomiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang, Yongmei Cheng, Shangling Jui, and Joost van de Weijer. Semantic drift compensation for class-incremental learning. Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6982-6991, 2020.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning-Voll Volume 70, pp. 3987- 3995, 2017.
Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shu-Tao Xia. Maintaining discrimination and fairness in class incremental learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pultern Recognition, pp. 13208-13217, 2020.
A APPENDIX
A.1 PSEUDO CODE FOR IDLVQ-C
Algorithm IDLVQ-C
In the base task Initialize 0 {m]:... m and
Minimize L = LCE + Aintra w.r.t. 01, {m] m } and
Pick exemplars from D for classes in the base task: x; arg min [|fm=1(x) m for novel task f 2, 3... do Initialize (m - 1.: m
Minimize L = LM + >php + Ainntra w.r.t. 6" and {m +1 m
Calibrate old reference vector (m/-),.. ..., m/ using m m + 5
Pick exemplars from Dr for classes in the novel task t: x, arg minxeD [1 m end for
A.2 PSEUDO CODE FOR IDLVQ-R
Algorithm 2 IDLVQ-R
In the base task (t (t=10 1) Initialize 0, {m]
m y ). {g- ..., g and y
Minimize LMSE w.r.t. 0. [m m m tht ..., g and
for novel task f = 2,3, do
Initialize {m y and (ghe-141 d
Minimize LMSE w.r.t {m x 14--11 m and { 1411.... g're
end for
12

---

Published as a conference paper at ICLR 2021
A.3 EXPERIMENT DETAILS FOR INCREMENTAL FEW-SHOT CLASSIFICATION
The base model is trained by the SGD optimizer (momentum of 0.9 and weight decay of le-4) with mini-batch size of 64. For CUB dataset, the initial learning rate is 0.01 and decayed by 0.1 after 60 and 120 epochs (200 epochs in total). For minilmageNet, the learning rate also starts from 0.01 and is decayed by 0.1 every 200 epochs (600 epochs in total). In an incremental learning session > 1), the model is fine-tuned with with D' with learning rate of 0.01 for 100 epochs. Since novel data D' (t > 1) contains very few training samples, all training samples in D' are included in one mini-batch. In addition, larger Aintra leads to more compact intra-class variation. However, convergence could be slow Xintra is too large. In addition, larger XP results in less forgetting in old classes but makes learning novel classes more difficult.
use Aintra = 1.0 and Xy 0.5 for both datasets. Empirically,
A.4 ADDITIONAL RESULTS FOR INCREMENTAL FEW-SHOT CLASSIFICATION
The accuracies for base and novel classes are reported separately in Table 5 and for CUB amd minilmageNet, respectively. The prediction accuracy of novel classes ca calculated upon all novel classes the model has been trained on. Note that the accuracy in Table I and 2 is calculated upon all classes (including base and novel classes) that the model has been trained on. The proposed IDLVQ- C demonstrates strong capability of preserving old knowledge by achieving the best performance old classes across all learning sessions. In parallel, Imprint method performs slightly better on novel classes than IDLVQ-C in early incremental learning sessions, while IDLVQ-C outperforms Imprint method in longer incremental learning sessions. The advantage of IDLVQ-C can be attributed to the adaptive feature extractor, which is tuned in each learning session.
Table 5: Prediction accuracy on CUB base and novel classes using the 10-way 5-shot incremental setting.
Base classes
Sessions
10
Fine-tune
77.30
44.23
36.28
27.52
25.96
23.05
17.68
13.07
11.78
10.99
10.71
Joint train
77.30
75.83
74.58
73.74
23.11
73.25
73.18
ICaRL Rebulli et al. 2017
77.30
59.38
58.81
54,43
48:27
43.28
39.17
34.91
32.43
29.36
25.87
Rebolancing
77.30
66.43
60.32
55.36
41.76
37.12
31.26
27.03
24.25
ProtoNet Stodl
77.30
72.55
72.21
72.05
71.64
71.29
71.02
20.67
70.60
70.53
77.30
71.57
71.14
70.98
70.85
SDC
77.34
76.05
72.36
71.81
71.68
70.96
Imprint
74.76
70
70.00
69.27
ID-C
73.32
Novel elasses
sossions
10
11
15.85
11.37
10.81
Joint
47.78
36.68
38.52
38.91
43.28
[CaRL. (Robulii
2017
35.18
23.99
23.04
24.59
Rebalmncing
45.53
19.65
20.91
22.97
ProtoNet Snell
2017
41.86
35.01
29.72
29.43
28.72
28.65
30.02
29.56
30.59
30.19
ILVO
Xa
2012
40.74
32,89
29.54
29.30
30.38
28.82
29.17
30.32
30.09
SDC
Yu al
200
58.45
40.65
35.77
33.23
31.16
30.24
33.25
30.89
32.26
32.08
Inngeint
2018
59.69
44.15
38.68
38.11
40.46
41.14
42.17
40.33
40.47
40.37
IDLVQ-C
58.72
42.18
37.86
39.97
40.84
41.30
43.21
40.32
42.34
42.30
The test accuracy on CUB dataset using 10-way 10-shot and 10-way 20-shot incremental settings are reported in Table 7 and respectively. The prediction accuracy improves in all methods with more training samples per class. The iCaRL and Rebalancing methods show the most significant improvement when the number of training samples increases. The proposed IDLVQ-C is effective in different incremental few-shot scenarios as it achieves the best performance on 5-shot, 10-shot and 20-shot settings.
A.5 EXPERIMENT DETAILS FOR INCREMENTAL FEW-SHOT REGRESSION
Sinusoidal wave: A six-layer feedforward neural network with ReLU nonlinear activation is used the feature extractor. IDLVQ-R learns 10 reference vectors and targets from the base task. In
13

---

Published as a conference paper at ICLR 2021
Table 6: Prediction accuracy minilmageNet base and novel classes using the 5-way 5-shot incre- mental setting.
Base classes
64.25
Joint
64.25
62.18
62.68
61.89
iCaRI. (Relutti
2017
64.25
Rebalaning
2019
64.25
28.
ProsoNet Sacll
64.25
57.63
ILVO
Xarel
2012
64.25
60.24
59.62
59,02
58.61
57.71
57.16
56,83
56,69
SDX
Yu al
2037
64.62
63.58
62.78
61.12
60.29
99:37
59.05
59,97
59.87
Imprint (1) et
2018
64.71
63.32
62.96
02.13
61.17
61.27
00.63
59,85
39.64
IDINQ-C
64.77
63.77
63.22
62.44
61.32
61.47
60.97
60.66
60.44
classss
sessions
5
9
Fine-tine
4,07
4,49
3.75
1.93
1.53
1.77
1.61
1.36
Joins train
4,80
9.84
13:26
12.30
14.03
14.01
15.21
15,62
iCaRL (Rbuuli al
2017
4.60
8.09
8.92
7:57
9.10
8.19
7.86
7,70
Rebolancing (Hou et
2019
5.29
8.25
9.43
8.00
8.93
8.11
8.85
8.27
ProtoMet
Saell et
2017
5.32
8.41
9.79
11.42
12.14
11.67
11.39
11.35
IVQ
Xa el
201
5.25
9.20
10:47
12.09
12.35
11.86
11.45
11.34
SDC
Ya al
2000
12.23
11.05
10.12
12:33
11:46
10.81
12.58
13.30
Imppint
G
2018
15.81
12.21
13.3
16.09
13.81
12.45
12,93
13.69
IDIVQ-C
13.07
12.19
13.34
15.86
14.14
12.55
13.11
13.94
Table 7: Prediction accuracy CUB using the 10-way 10-shot incremental setting.
Method
10
Fine-tune
77.30
47.16
26.92
24.00
17.19
12.73
11.75
11:43
Joint
77.30
73.29
Rehurh
77.30
59.66
46.37
41.19
Reboltncing
2019
77.30
45
41,02
ProtoNet
77.30
69.82
61.17
58.85
55.82
2012
$6.01
SDC
74.67
62.14
61.33
56.62
Imprint (1t
2018
77.02
74.07
70,26
64.45
62.46
61.85
61.02
59,20
58.93
58.43
IDI.Q-CC
74.79
59.72
Table Prediction accuracy on CUB using the 10-way 20-shot incremental setting.
Method
sessions
6
10
11
Fine-tune
77.30
47.68
36.75
27.01
24.56
21.72
17. 88
15.24
12.84
11.89
11.52
Joint train
77.30
74.34
72.59
71.85
70.14
70.01
69.69
69.67
69.48
69.41
69.42
ICARI. Rehumli al
2017
77.30
68.85
63.56
60.34
57.71
56 75
55.97
55.02
$4.62
52.21
52.23
Rebalancing Hou
2019
77.30
69.36
65.49
61.32
59.30
58.68
58.77
57.42
56.25
35.33
$4.17
ProtoNet Smell
2017
77.30
70.84
67.82
64.81
62.95
61.71
60.98
60.73
59.51
59.30
$8.95
IVD
Xuet
77.30
71.50
68:77
66.12
64.31
62.89
62.3
62.00
61.52
59.70
59.23
SDC
or ml
200
77.34
74.62
71.63
68.65
66.75
65.34
64.21
63.62
61.97
61.54
61.11
Impeint
U1 el.ll.
2018
77.02
74.14
70.72
67.75
65.38
64.63
64.28
03.82
62.00
61.84
61.40
IDLVQ-C
77.37
74.84
72.07
69.05
67.28
65.51
65.19
64.84
62.77
62.55
61.96
each incremental learning sessions, pairs of reference vectors and targets are added. After new reference vectors and targets are fine-tuned, the model is capable of make prediction for all seen tasks. exemplars are selected uniformly from the training set the base task. In the incremental learning session of the 2nd task, the model is fine-tuned samples. After the training converges, novel training samples the current task are added to the exemplar set. In the incremental learning session of the 3rd task, the model is fine-tuned with 15 exemplars from old tasks and 5 novel training samples.
10 exemplars and 5 novel training
3D spatial data: We follow the same training and test protocols as the sinusoidal wave dataset. We choose 40, 15 and reference vectors and targets for 1st. 2nd and 3rd tasks, respectively. Adding more reference vectors does result in obvious improvement in accuracy in experiments.
14

---

Published as a conference paper at ICLR 2021
80.0%
75.0%
Overall accuracy
70.0%
65.0%
60.0%
5-shot
10-shot
20-shot
55.0%
100
120
140
160
180
200
Number classes
Figure 2: Comparison results of different few-shot settings, evaluated with ResNet18 on CUB dataset
A.6 VISUALIZATION OF IDLVQ-C
We show the visualization standard neural networks and IDLVQ-C with/without intra-class vari- ation loss in Fig. 3 MNIST dataset is used toy example for visualization. Classes 0-7 are old classes with sufficient training samples, and classes and 9 are novel classes with few-shot train- ing samples. It can be observed in Fig. 3(a) and 3(b standard neural networks and IDLVQ-C (without intra-class variation loss) trained by cross-entropy loss do not have compact intra-class variation. Consequently, features novel classes are more likely to overlap with old classes. In this case, the performance of inteemmaattmttal learning degrades very quickly because the classifier cannot distinguish between features from different classes. In comparison, the proposed IDLVQ-C makes intra-class variation compact and leaves large margin between classes in Fig 3()) As a result, the features of novel classes are less likely to overlap existing classes. The compact intra-class varia- tion and large margin between classes make features of novel classes distinguishable so that learning novel classes is easier. In addition, the margin based loss only updates the model parameters when necessary and avoids catastrophic forgetting of old classes.
15

---

Published as a conference paper at ICLR 2021
(a)
(b)
Figure 3: Visualization of feature spaces in different methods. Dots represent features of samples and crosses denote references vectors of classes. (a) standard neural networks; (b) I without Cintrai (c) IDLVQ-C.
16


Reviews:
[{"review_id": "k4Js-Hvop2J", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new method for incremental few-shot learning based on feature quantization. Each class is represented by a reference vector, which is initialized as a centroid of features within that class and fine-tuned using a margin loss, an intra-class loss, and a forgetting loss. \n\nPros:\n+ The paper is overall well-written.\n+ The proposed method compares favorably against various baselines, including strong ones such as SDC.\n+ Experiments are conducted on both classification and regression datasets.\n\nCons:\n- This paper is very related to the work of Xu et al. [1] but fails to cite it. Both this paper and Xu et al. propose an incremental learning method based on vector quantization and compare it with prototype-based classifiers. Compared with Xu et al., this paper performs VQ learning a deep embedding space and adopts a different learning rule. The relationship with Xu et al. needs to be thoroughly discussed. A baseline approach that naively combines Xu et al. with a deep network by replacing the nearest neighbor classifier in ProtoNet with the VQ classifier by Xu et al., should also be implemented and compared.  \n\n- In the classification experiments, the paper only reports the average accuracy for all classes the model has been trained on. It would be helpful to separately report the accuracy of previous classes and new classes so that readers can understand whether \n- The margin-based loss is not evaluated in the ablation study.\n\nMinor problems that do not affect my score:\nP1: are compressed -> is compressed\nP5: repeated “such as”\n\nOverall, I lean towards rejection, mainly because of the lack of discussion and comparison with Xu et al. [1]. I am willing to increase my score if this issue can be fixed in the rebuttal.\n\n[1] Xu, Ye, Furao Shen, and Jinxi Zhao. \"An incremental learning vector quantization algorithm for pattern classification.\" Neural Computing and Applications 21.6 (2012): 1205-1215.\n\n======================================================\n\nI've read the rebuttal and the authors have addressed most of my concerns in the revised paper so I raise my score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "cj3fRMP-7Lg", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper suggests to use a generative model to address the problem of 'few shot' incremental learning. The idea is to classify input data by maintaining a population of prototypes and measuring the distance of the examples to be classified from these prototypes. As, each prototype represents a class, an example to be classified is assigned to the class of the closest prototype.  The learning of the neural network transforming input data to the prototype space is learned using 2 loss functions: the first one maximizes the margin between the distance to the prototype of the correct class and the other prototypes, the second one makes the clusters as compact as possible.\n\nThe method is experimentally validated on 2 tasks: an incremental image classification task and an incremental regression task.\n\nThe use of generative methods seems a good thing to limit catastrophic forgetting, by anchoring classes on prototypes.  I am not aware of any other work that has proposed this before, which gives a certain originality to the paper. In addition, I found the paper clearly written.\n\nOn the negative side, I'm concerned by 3 points. My first concern is related to the size of the training sets. The paper addresses the very particular case where the number of training data per problem is very low, a case where classical incremental methods do not work well. On the other hand, the paper says nothing about the behavior of the method when the size increases.  Figure 2 shows that the performance increases, but the question is whether it still works better than traditional methods in such cases. If it does not, this would reduce the scope of the method considerably.\n\nMy second concern is that the authors seem to completely ignore the recent literature on GMM + neural networks. It is a literature that I myself know little about, but I know that it is substantial.  Below are some related references. One should confront the proposed method with this literature and not simply say, as said in section 3.2, that this topic has not attracted much attention in the community.\n- J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. In NeurIPS, 2017.\n- A Gaussian Mixture Model layer jointly optimized with discriminative features within a Deep Neural Network architecture, Ehsan Variani; Erik McDermott; Georg Heigold,  2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)\n- Prototype discriminative learning for face image set classification, W Wang, R Wang, S Shan, X Chen - Asian Conference on Computer Vision 2016\n- Simultaneous learning of reduced prototypes and local metric for image set classification, Z Ren, B Wu, Q Sun, M Wu, Expert Systems with Applications Volume 134, 15 November 2019\n- Joint prototype and metric learning for set-to-set matching: Application to biometrics\nM Leng, P Moutafis, IA Kakadiaris 2015 IEEE 7th International Conference on Biometrics Theory, Applications ...\n\nFinally, I did not find the experimental validation very convincing, as far as the part on incremental learning is concerned. The proposed method is only slightly better than methods that are not at the state of the art level. In this regard, I also noted that recent incremental learning methods were not cited in the state of the art. I invite the authors to look at the papers:\n- \"Prabhu et. al., Gdumb: A simple approach that questions our progress in continual learning.  ECCV2020\" \n- Zhao, Bowen, Xi Xiao, Guojun Gan, Bin Zhang, et Shu-Tao Xia. « Maintaining Discrimination and Fairness in Class Incremental Learning ». In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),\n- Tao, Xiaoyu, Xinyuan Chang, Xiaopeng Hong, Xing Wei, et Yihong Gong. « Topology-Preserving Class-Incremental Learning ». In ECCV, 16, 2020.\n- Hou, Saihui, Xinyu Pan, Chen Change Loy, Zilei Wang, et Dahua Lin. « Learning a Unified Classifier Incrementally via Rebalancing ». In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),\nwhich can provide both recent baselines and also entry points to additional recent literature.  \n\n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "QsTrfZh8P6A", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new non-parametric method for few-shot incremental learning. The aim is to perform few-shots classification and regression while being robust to catastrophic forgetting when trying to learn new classes.\nThe method learns exemplars (reference vectors) and feature representation in order to be able to generalize to unseen classes. \n\nQuality & Clarity:\nThe paper is well written and the proposed approach seems to work well for few-shot incremental learning. There are some minor issues with presentation and experimental results.\n\nOriginality & Significance:\nThe approach seems a good extension of incremental few-shot learning based on prototypes. Results show that the proposed extension leads to better performance on few-shot classification.\n\nPros:\n- The paper is well presented and motivated\n- Results show that the method is performing better than previous approaches on two datasets.\n- The ablation study shows the importance of every component in the method\n\nCons:\n- Some statements should be better justified \n- Some recent methods haven't been included in the comparison (e.g. Tao et al., 2020). Is there a clear reason?\n- The incremental few-shot regression is evaluted on artificial toy problems: Sinusoidal Wave and 3D spatial data\n\nAdditional Comments:\n- Bottom first page, it is not explained why incremental learning strategies cannot be extended to regression problems easily.\n- Top second page, it is not clear why over-fitting exacerbate the difficulty of learning new classes. \n- How does the method compare with previous approaches in terms of computational cost?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "El07E8RHVQw", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper Summary:\n\nThis paper proposes a nonparametric method in deep embedded space to address incremental few-shot learning problems. By compressing the learned tasks into a small number of reference vectors, the method could add more reference vectors to the model for each novel task, which could alleviate catastrophic forgetting and improve the performance of related tasks. Finally, this paper evaluates the proposed method on the classification and regression problem, respectively.\n\n\nStrengths:\n\nThe idea of employing reference vectors to address incremental few-shot learning is novel and interesting.\n\n\nWeakness:\n\n1. The writing of this paper is not clear. The authors do not introduce their motivation clearly. For example, in the Introduction Section, the authors said \"To the best of our knowledge, the majority of incremental learning methodologies focus on classification problems and they cannot be extended to regression problems easily.\" Here, the authors should further interpret the reason for this scenario. Meanwhile, compared with classification, the authors should analyze the challenges for the incremental few-shot regression problem. Besides, the authors should interpret the advantages of employing nonparametric method.\n\n2. The method section is hard to understand. I have no idea of the motivation of the proposed method. The authors should give more interpretations of the proposed method. In Section 4.2, the authors indicate the cross-entropy loss does not guarantee compact intra-class variation in the feature space. The authors should give some visualization analyses to demonstrate their opinion. Meanwhile, the authors should analyze the reason. Eq. (2) and Eq. (4) are two simple loss functions. The authors should give more interpretations of why Eq. (2) and Eq. (4) could guarantee intra-class variation and regularize the drift in the feature space.\n\n3. In Table 1 and 2, for the compared methods, the authors should give detailed citations. Besides, for CUB and miniImagNet dataset, the authors only evaluate their method under the 10-way 5-shot and 5-way 5-shot case. To demonstrate the effectiveness of the proposed method, the authors should evaluate their method on more cases. Based on Table 1 and 2, we are not sure whether the proposed method is effective. Besides, the authors should make a visualization analysis to further demonstrate the effectiveness of the proposed method. Finally, the proposed method contains some hyper-parameters. The authors should analyze the impact of these hyper-parameters.\n\nOverall, though the idea of employing reference vectors is interesting, the motivation of this paper is not clear. The method section is hard to understand. Meanwhile, the experimental results do not demonstrate the effectiveness of the proposed method sufficiently. The authors do not make a visualization analysis of the proposed method. Thus, I think this paper is below the acceptance.\n\nEDIT: The authors' rebuttals have solved my concerns partially. However, there still exist some concerns about this paper.\n\n1. In the introduction, the authors indicate that nonparametric methods compute the class centroids for novel classes. However, I am not clear whether computing class centroids is the only mechanism to solve class-incremental learning. Besides, I recommend citing more papers about this interpretation.\n\n2. The category centers are usually affected by the number of samples. When the samples are scarce, the constructed centers are not accurate, which affects the performance. The authors should give more discussions about category centers.\n\n3. For the regression problem, it is better to give more implementation details and ablation analyses. Besides, whether MSE is the only loss to train regression network.\n\nBased on these concerns, I update the score to 5.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There was quite some variance in opinion on this paper, with some reviewers commenting on problems with clarity and experimental evaluation. The authors rebuttals improved the reviewer opinions slightly. The rebuttal and accompanying revisions are convincing, and the new experimental results are convincing and also very much appreciated. This is one of the first papers taking a comprehensive look at incremental, few-shot classification AND regression. Despite some problems with clarity (which were well-addressed in rebuttal and revisions), the paper is original and presents novel ideas about incremental few-shot learning.

Pros: consideration of both few-shot classification and regression, ablation study well-executed and convincing.

(remaining) Cons: some minor problems with clarity - please take reviewer comments on board when preparing the camera ready version.
Decision: Accept (Poster)