Progressive Voronoi Diagram Subdivision Enables Accurate Data-free Class-Incremental Learning
Authors: Chunwei Ma
Zhanghexuan Ji
Ziyun Huang
Yan Shen
Mingchen Gao
Jinhui Xu
Venue: ICLR 2023 poster (2023)
Keywords: Voronoi Diagram
Computational Geometry

Abstract: Data-free Class-incremental Learning (CIL) is a challenging problem because rehearsing data from previous phases is strictly prohibited, causing catastrophic forgetting of Deep Neural Networks (DNNs). In this paper, we present \emph{iVoro}, a novel framework derived from computational geometry. We found Voronoi Diagram (VD), a classical model for space subdivision, is especially powerful for solving the CIL problem, because VD itself can be constructed favorably in an incremental manner -- the newly added sites (classes) will only affect the proximate classes, making the non-contiguous classes hardly forgettable. Furthermore, we bridge DNN and VD using Power Diagram Reduction, and show that the VD structure can be progressively refined along the phases using a divide-and-conquer algorithm. Moreover, our VD construction is not restricted to the deep feature space, but is also applicable to multiple intermediate feature spaces, promoting VD to be multilayer VD that efficiently captures multi-grained features from DNN. Importantly, \emph{iVoro} is also capable of handling uncertainty-aware test-time Voronoi cell assignment and has exhibited high correlations between geometric uncertainty and predictive accuracy (up to ${\sim}0.9$). Putting everything together, \emph{iVoro} achieves up to $25.26\%$, $37.09\%$, and $33.21\%$ improvements on CIFAR-100, TinyImageNet, and ImageNet-Subset, respectively, compared to the state-of-the-art non-exemplar CIL approaches. In conclusion, \emph{iVoro} enables highly accurate, privacy-preserving, and geometrically interpretable CIL that is particularly useful when cross-phase data sharing is forbidden, e.g. in medical applications.

Published as a conference paper at ICLR 2023
PROGRESSIVE VORONOI DIAGRAM SUBDIVISION ENABLES ACCURATE DATA-FREE CLASS- INCREMENTAL LEARNING
Chunwei Ma', Zhanghexuan Ji', Ziyun Huang", Yan Shen', Mingchen Gao', Jinhui G Xu
Science and Engineering, University at Buffalo *Computer Science and Software Engineering, Penn State Erie chunwe m, zhanghex, y2h mgao8, j jinhui
Department of Computer
{zxh201 @psu. edu
ABSTRACT
Data-free Class-incremental Learning (CIL) is a challenging problem because rehearsing data from previous phases is strictly prohibited, causing catastrophic forgetting Deep Neural Networks (DNNs). In this paper, we present iVoro, a novel framework derived from computational geometry. We found Voronoi Diagram (VD), a classical model for space subdivision, is especially powerful for solving the CIL problem, because VD itself can be constructed favorably in an incremental manner the newly added sites (classes) will only affect the proximate classes, making the non-contiguous classes hardly forgettable. Furthermore, we bridge DNN and VD using Power Diagram Reduction, and show that the VD structure can be progressively refined along the phases using divide-and-conquee algorithm. Moreover, our VD construction is not restricted to the deep feature space, but is also applicable multiple intermediate feature spaces, promoting VD to be multilayer VD that efficiently captures multi-grained features from DNN. Importantly, iVoro is also capable of handling uncertainty-awee test-time Voronoi cell assignment and has exhibited high correlations between geometric uncertainty and predictive accuracy (up to ~0.9). Putting everything together, iVoro achieves up to 25. 26.26% 37. 37.09%, and 33. 23.21% improvements on CIFAR-100, TinylmageNet, and ImageNet-Subset, respectively, compared the state-of-the-an non-exemplar CIL approaches. In conclusion, iVoro enables highly accurate, privacy-preserving and geometrically interpretable CIL. that is particularly useful when cross pross-phase data sharing is forbidden, e.g. in medical applications.
INTRODUCTION
In many real-world applications such medical imaging-bai based diagnosis, the learning system is usually required to be expandable new classes, for example, from common to rare inherited retinal diseases (IRDs) (Miere al., 2020), from coarse to fine chest radiographic find- ings (Syeda- Mahaood et al.) 2020). and importantly, without losing the knowledge already learned. This motivates the concept of incremental learning (IL) (Hou et al.
(A) Base Classes Voronol Diagram
(B) Incremental Voronal Diagram
(C) Progressive Voronol Diagram
Figure 1: Schematic illustrations of Voronoi Diagram (VD) for base sites (A), and when new site (B) or clique of new sites (C) is added to the system.
2019
2021b), also known as continual learning Parisi 2019). which has drawn growing interest recent years. Although Deep Neural Networks (DNNs) have become the de facto method of choice due
Wu et al. 2019; Zhu et al. 2021 Liu et al.
al.
2 Delange et al. 2021 Chaudhry et al.
to their extraordinary ability learn from complex data, they still suffer from severe catastrophic

---

Published as a conference paper at ICLR 2023
(A) Fine-tuning Voronoi Diagram
(B) PASS Voronoi Diagram
98
7-class 41.76%
ACC: 31.12%
ACC:
7-flact ACC,
10sritess 40.56%
phoce C 39,99
Biste O:: DO 00.
plaaee D: 00:00
phisse DI 99.99.3
Ahate O. 57 R5
phase 1: 99:722
phese 1: 00 DU
pfface 02.03
phase 1: 04, 75
phise 99 81/
Johase 2: 00]16
(C) IVore Vorenoi Diagram
(D) IVoro-AC Voronal Diagram
4-cless ACC: 96.70%
7-cless ACC: 73.42%
10-tless ACC: 64.84%
d-cles ACC: 99,47%
7-cless ACC. 72.95%
10 ACC: 61.77%
Visualization of of Voronoi Diagrams induced by (A) incremental fine-tuning, (B) PASS (Zhu et al. 2021), (C) iVoro, and (D) iVoro-AC on MNIST dataset in R2 (best viewed in color). The dataset was split to 4, 3, and 3 disjoint classes. (See Appendix BarennixBBooo for details.)
Figure 2:
forgetting (McCloskey & Cohen 1989 Goodfellow et al. 2014 Kemker et al. 2018) when adapting to new tasks that contain only unseen training samples from novel classes.
To mitigate this issue, Resue, Rebuffi al. (2017) proposed the paradigm of memory based class-incremental
learning (CIL) (Belouadah Popescu| 2019, Zhao al. 2020; Hou et al. 2019, Castro et al. 2018
Wu et al. 2019 Liu et al. 202 la 2020a; 2021b) in which small portion of samples (c.g.. 20 exemplars per class) will be stored to use in the subsequent phases. However, the storing and sharing of data, c.g. medical images, may not be feasible due to privacy considerations. Another line of methods memorize (part of) network and increase the model capacity for new classes (Rusu et al. 2016 Li et al. 2019 Wang al., (017; Yoon et al.) 2017). which may incur unbounded memory consumption for long task sequence. Hence, in this paper, we focus on the challenging data-free CIL problem under the strictest memory and privacy constraints no stored exemplars and fixed model capacity.
Despite extensive research in recent years (see Appendix A for literature review), three challenges still pose an obstacle to successful CIL. (I) During the course of isolated training on new data, the feature distributions of the old classes are usually dramatically changed (see Fig. (A) for an illustration). Knowledge Distillation (KD) (Hinton et al.] 2015) has become routine in many CIL
Schwarz et al. 2018] Castro et al. 2018 Hou al. 2019 Dhar et al., 2019; Douillard et al. 2020 Zhu et al. 2021 to partially maintain the spatial distribution of old classes. The KD loss, however, is typically applied onto the whole network, and a strong KD loss may potentially degenerate the network's ability to adapt to novel classes. (II) Without the full access to old data, the decision boundaries cannot be learned precisely, making it harder to discriminate between old and new classes. Taking inspiration from metric-based Few-shot Learning (FSL.) (Snell et al.] 2017). PASS (Zhu et al., 2021) memorizes a set of prototypes (feature centroids) and generates features augmented by Gaussian noise for a joint training new phases. However, feature centroids might be suboptimal to represent the whole class, which is not necessarily normally distributed (Fig. 2 (B)). (III) Since the old classes and the new classes are learned in disjoint manner, their distributions are likely to be overlapped, which becomes even severer in our exemplar-fre free setting as the old data totally absent. To circumvent this issue, Task-increnne incremental learning (TIL) (Shin
methods (Li & Hoiem 2017:
et al. al. 2017 Kirkpatrick et al.) 2017 Zenke et al. 2017 Wu et al., 2018 2018/LLpp77 Lopez-Paz & Ranzato [017]
Buzzega et al. 20200 Cha et al.] 2021) P02m Pham et al. 2021: Fernando al.) 2017 assumes the phase
within which class was learned is known, which is generally unrcalistic in practice. CIL is not
grounded on this assumption.
In this paper, we tackle the CIL problem from geometric point of view. Voronoi Diagram (VD) is a classical model for space subdivision and the underlying geometric structure of the 1-nearest neighbor classifier (Lee) 1982). We find that VD bears a close analogy incremental learning, because VD itself can be constructed favorably in an incremental manner the newly added sites (classes) will roughly change only the cells of the neighboring classes, making the non-contiguous
2

---

Published as a conference paper at ICLR 2023
classes untouched and thus hardly forgattable (see Figure 1). Based this intuition, in this paper, we present a holistic geometric framework based on VD that significantly surmounts all the listed obstacles. The contributions can be summarized follows:
We explore, for the first time, the idea of using prototypical networks (Snell et al. (017) for CIL,, which is equivalent to constructing a VD in the (fixed) feature space (denoted iVoro).
1.
iVoro
iVoro-D
2. We show that the within-phase boundaries of
phase phase
VD can be progressively refined using divide-and- conquer algorithm (iVoro-D).
prototyaical boundary
breciig protolypical
prokeypkal boundary
31056-phaa5
Figure 3: Schematic illustrations of iVoro
3. When comes to test-time Voronoi cell as- signment, we devise two protocols, augmentation consensus (iVoro-AC) and integration (oroo-A)),, for the postprocessing of Self-superrisee Learning (SSL)-based label augmentation, with quantitative uncertainty awareness.
(left) and iVoro-D (right). iVoro-D uses
probing-induced boundaries within phase.
4. Finally, we introduce multilayer features to build a multilayer VD, which consistently enhances the performance (i (iVoro-L..
Main Ideas and Results
iVoro. We begin with the simplest scenario in which the feature extractor is frozen after the first phase, and the prototypes ((c)) are used to construct VD.
iVoro-D. iVoro treats all prototypes equally regardless of at which phase they present, and determines Voronoi boundaries all by bisecting prototypes. However, without considering data distribution, the bisector of two prototypes is not optimal especially within a certain phase. We establish an explicit connection between DNN and VD using Voronoi Diagram Reduction (Ma et al. 2022a) and show that the within-phase disicion boundaries (induced by {c}) can be refined by DNN (i.e. linear probing) and be aggregated into the global VD by ddivdd-andd-onquee (D&C) algorithm (iVoro-D).
AVoro-AC/Al. Geometrically, SSL-based label augmentation (Lee et al. 2020 will duplicate one Voronoi cell to be multiple (possibly disjoint) Voronoi cells (sce Fig. 2 (D)), and this will cause ambiguity when assigning query example to cell, suggesting that uncertainty quantification cannot be neglected in test-time. Here propose two protocols to resolve this ambiguity, namely, augmentation consensus (iVoro-AC) and augmentation integration (iVoro-AD). We also show that the entropy-based geometric variance (Ding & Xu 2020) is a good indicator of the uncertainty of this assignment, with high Pearson correlation coefficients up to ~0.9.
iVoro-L. Until now, only deep features from the last layer are used for VD construction. However, the intermediate feature could also be informative to aid the VD construction. Cluster-induced Voronoi Diagram (CIVD) (Chen et al., 2013) 2017 Huang & Xu 2020 Huang et al., 202T), which allows for multiple centers per Voronoi cell, has recently achieved remarkable success in metric- -asee FSL by incorporating heterogeneous features VD (M et al., 2022a). As a matter of fact, for deep neural network, the feature induced by every layer can all be used to construct VD. Finally, we also explore the idea to build a multilayer VD by using features elicited from multiple blocks DNN.
Broader impact. The fully-fledged iVoro achieves up to 25.66%, 37.09%, and 33.21 % improvements on CIFAR- 100, Iinyem,,,Net, and ImageNet-Subsee Subset, respectively, compared with the tfte-off-the-an non-exemplar CIL approaches. Based on the frozen model trained at the first phase, iVoro and all its variants incur no additional training burden, and at the same time preserve the privacy of data from previous phases. It is worth noting that, although i'ooo focuses exemplar-free CIL, it outperforms even all the exemplar- based CIL methods. We believe iVoro could be further boosted when a small number of exemplars allowed, which leave for future work.
METHODOLOGY
2.1 PRELIMINARIES:
LEARNING
In CIL, the data comes as a stream and a single model is trained on current data locally without revisiting previous data, but should ideally be able to discriminate between all classes it has seen so
3

---

Published as a conference paper at ICLR 2023
far. Specifically, let D (De) be the data stream in which D; = { s, kr is the dataset at time step t, with data Zi,i E ID and label yt E C.. D is an arbitrary domain, e.g., natural image, and C2 is the set of classes at phase t. The dataset D; contains Ne,,, E (1,..., [,. samples for the K, classes (i.e. N,- = - Notice that C, C for two arbitrary phases i, j are disjoint, i.e. C, C, 0, Vi i + J. The unified model consists of feature extractor & and classification head
8. The feature extractor is a deep neural network x = Φ(x)) ) x E IR" that maps from image domain DD to feature domain RR" and is (traditionally) trained continuously each phase L. In this section,
T, I, and denote total phase, current phase, and historical phase, respectively (i.e. & E (1,-..7)
(1,.,))))
2.2 CONSTRUCTING VORONOT DIAGRAMS: A FEATURE EXTRACTOR ALL YOU NEED
In many CIL methods, the feature extractor & and classification head 0 are jointly and continuously optimized during every phase /. guided by carefully designed losses (Zhu et al. 202T). As a starting point, in this section, we freeze the feature extractor á after the first phase and use a Voronoi Diagram (i.e., a l-nearest-neigee mnearest-neighor classifiers) to be as an extremely simple baseline method (denoted as iVoro), upon which we will then gradually add component introduced in Sec. First, we introduce Power Diagram (PD), a generalized version of VD:
Definition 2.1 (Power Diagram (Aurenhammer 1987) and Voronoi Diagram). Let (w1, ... WK be partition of the space R", and C [1, ..... Cx be set of centers (also called sites) such ,w, 0. In addition, each center is associated with a weight Vr- E {VI,..., Dx) } S R Then, the set of pairs {(W), CI,Pi)..... (WK,Cr, [x)] is Power Diagram (PD), where each cell obtained via W, = {z R r(z) = r)r E {1,... K}, with r(z) = arg mill ke then a PD collapses to a Voronoi Diagram (VD).
that UK Wr R n -,-11:
d(z, --,, If the weights are equal for all k,,ie. U/ uy, Vk, k E (1, (1,..., K)
Prototypes. As baseline model, the class centers for i'oro are simply chosen to be the prototypes (feature mean of one class): Cr. = Nr. Lielh...Nraa)eee o(xr,i), (..,), Vr = 0, E {1,..., "......,,. f), k E
(1..., K.). We name those centers prototypical centers. Note that this set of centers } carries prototypes for all classes, old and new, up to time I. In test-time, a query sample x is assigned to the nearest class 9 C,
R s.t. d(z,c, c,( min,, d(z,, Cr,k) which d(zz, C,,s) || c.
Parameterized Feature Transformation. Although PASS Zhu et al. 2021) uses Gaussian noise to augment the data, the actual features are not necessarily normally distributed. To encourage the normality of feature distribution here we adopt compositional feature transformation commonly used in FSL (Ma et al.) 2022a): (1) L2 normalization projects the feature onto the unit sphere: f(z)= = T : (2) linear transformation performs the scaling and shifting: Iw.n (2) wz 17; and (3) Tukey's
if X + 0 ladder of powers transformation further improves the Gaussianity: hx(z) log(z) if = 0"
Finally, the feature transformation is the composition of three: (hx Iw., f2), parameterized by w, X. If all features (for both training and testing set) go through this normalization function, then iVoro becomes iVoro-N.
2.3 DIVIDE AND CONQUER: PROGRESSIVE VORONOT DIAGRAMS FOR CIL
As mentioned earlier, i'oro (and Voooo-N) treats all classes equally and separates them all by bisectors, regardless of at which phase they appear. However, for two classes C.,,, Cr, appear in the same phase т, we can in fact draw better boundary by training a linear probing model parametrized by W, b in the fixed feature space. After the training, the locating of new Voronoi center requires an explicit relationship between the probing model and VD. More formally, at phase linear classifier with cross- cntsrpyy loss is optimized on the local data Dí:
exp SSp Cy o(z) br.y)) W.p (()) + b.
b.) = Σ - -log -oogp(ll(a);: b.) = Σ (2.0)ED,
- log
(1)
(2.))ED
in which Wi,,. br,k are the linear weight and bias for class Cr,e. As parameterized model, this linear probing can idcally improve the discrimination within C, However, it is still non-trivial to merge all Wrr, &, br. === since the task identity is not assumed to be known like in TIL.. To solve this,
4

---

Published as a conference paper at ICLR 2023
get geometric insight from (Ma et al. 2022a) which directly connects linear probing model and VD
by the theorem shown as follows:
Theorem 2.1 (Voronoi Diagram Reduction et 2022a)). The linear classifier parameterized
by W. b partitions the imput space I" to a Voronoi Diagram with centers (cI,..., CK) given by C₁ W if - - 1....
For completeness, we also include the proof in Appendix During linear probing, Thm. 2.1 is satisfied, then it is guaranteed that the resulting centers (referred to probing-induced iinducedd centers) [Cr,))=11 will also induce VD (locally in phase t). Now given that we have two sets of centers (cr,a) and (Cr,a), with the latter being better locally but are not transferable across phases, we devise divide-anddconquer (D&C) algorithm that progressively construct the decision boundaries from the two sets of centers, boosting iVoro to iVooro-D
Divide. Fortunately, the total classes {C- )== have been split into already disjoint / cliques.
Conquer. Within each clique (i.e. phase) т, the boundary for any two classes Cr, Cr. is the bisector separating the prohing-induced centers 6r,K, Cr, denoted as T7,4i,,, {2'
E" "8'""==' - -- 0} where
6, Er. and q || 2| -Er,uylo [ Er,,,l
11-1 When merging cliques
Th, we instead resort to the protolypical centers for space partition: for any Cr in clique
Ti and any Cra.k' in clique 72, their bisector is T T..k,a,k = {=' E R"|="2"
0} where
In this way, the overall space partition would henefit from both locally probing-induce VD and globally prototype-hasel VD. See Fig. G.3 for an illustrative comparison of i'oro and i iooro-D.
G11 Gr2. [|e,,. Pr...--Sy.a Sry.erll and q 2|le,,
||c., -|c. -
Querying the VD. In test-time, one can find the assigned Voronoi cell for query example x by eliminating one class in each round according sign(v" 2 z' -q), starting from randomly selected boundary, so the time complexity is O(E,,,
2.4 AUGMENTATION INTEGRATION: UNCERTAINTY AWARE TEST-TIME VORONOI CELL ASSIGNMENT
Self-supervised Label Augmentation. To enhance the discriminative power of CIL method, SSL. based label augmentation (Lee et al. 2020) has been used to expand the original K2 classes to 4K, by rotating the original image Specifically, for image æ, the rotated image x (a) rotate( 2a), E {0, 2, will be assigned one of the expanded classes ý = k(()), k E [1, [1...., K......,,.... K},KK Σt= Kr. In training time, the model is trained on the expanded dataset; however, in testing time, each of the duplicated images (x( Jae could possibly be assigned to each of the expanded classes {K(a)) ( 1,,1,))) so this ambiguity has to be resolved, which has not been considered in previous CIL methods.
Augmentation Consensus. Let dia.a') RK be vector, each component of which denotes the distance from Φ<x) to a class that e has learned, i.c. de daaa)) d d()(x(a)), Cyter)) = [|)( Cgter 11, k {1,..., K}, a' {0, 1, Then we want to find consensus k, with the maximum occurrence among the x 4 predictions {arg min, ala,a')} Ja.rr((0.1233) Using augmentation consensus test-time, iVoro then retrofitted iVVoro-C.
Augmentation Integration. Using the consensus from the augmented samples should be more robust than the individual prediction arg min, a(000) itself, but has not considered the accumulated distance, so alternatively, we propose to integral over all predictions from augmented samples:
i = arg ming E. Eardh (a.a")
augmentation integration applied, then iVoro becomes AVoro-AI.
Uncertainty Quantification. Since in AC and iVoro-Al, the augmented samples collaboratively contribute to the final prediction, the quantitative uncertainty becomes non-negligible, this because for some intatio--ivvarian classes, e.g. balls, the rotation operation makes less Hence, when assigning a query sample æ to the augmented 4x Voronoi cells, an uncertainty quantification method needed.
5

---

Published as a conference paper at ICLR 2023
Truth Discovery Ensemble (TDE) (Ma et al. 2021 is the sfate-of-the-ar uncertainty calibration method for DNNs, which finds the consensus among ensemble members by the minimization of entropy-based geometric variance (HV). Here, we only borrow HV an indicator for the uncertainty of the x 4 predictions, and refer the readers to Ma et al. (2021) for more details about TDE. Given the mean vector of the augmented predictions d'
I6 2n a dlaaa E RK let V denote the total squared distance to d' (i.e., V = E. En, ||d* - diaaa)1' and gla,a)) denotes the contribution of each d (o.a") to V (i.e., gla,a)) = ||d 1/2/V. Then the entropy induced by {g(a.)") ? is:
H ==- (a.a') (ogg/aaa) 1/vV Ild d log(V /l/dd )
Based these, Definition 2.2 cintropy-based Geometric Variance (Dig & Xu 2020 Given the point set (d S R.K and a point d", the entropy based geometric variance (HV) is H x V where H. and V are defined shown above.
define the HV follows:
For every query example æ, we calculate (x: based on its (d Ja. J (.0.1. Later we will show how HV could favorably indicate the uncertainty of the augmented prediction, and tell when augmentation integration useful.
2.5 MULTILAYER VORONOL DIAGRAMS
iVoro-L
Until now, our VD construction is restricted the deep feature space. i.c., æ + 6(x) E R" However, the intermediate layers also contain information that sup- plementary to the final layer and can be useful to our VD construction. And this requires the integration of multiple VDs. Recently, Cluster-induced Voronoi Di- agram (CIVD) (Chen et al., 2017 Huang et al. 2021) and Cluster-to-cluster Voronoi Diagram (CCVD) (Ma iet al. 2022a), two advanced VD structures, have shown remarkable ability integrate multiple sets of centers for VD construction and achieve state-of-the-ar perfor- mance in metric- based FSL. In this paper, utilize the concept of CCVD for the integration of multiple VDs induced by multiple layers. We refer the readers to Ma et al. 2022a) for more details about CIVD/CCVD.
ll Mabee...
Block 4
VD,
Block 3
)
VD2
Block 2
VD,
Block
Cluster-induced
ResNet-18
Voronoi Diagram [44]
Figure 4: Schematic illustration of iVoro- L.
Definition 2.3 (Cluster-to-cluster Voronoi Diagram). Let Ω (Wiy wK3 be a partition of the space IR", and C = [Cl .... C be a set of totally ordered sets with the same cardinality L (i.e. |C,| = [c| = = CCxl = L). The set of pairs {(w1, ((wr,C1),... Ci),... (wK,CK) Ck)} is a Cluster-to-cluster Voronoi Diagram (CCVD) with respect to an influence function F F(x,, C(z)), and each cell is obtained via w., = (z R" r(2) r)." C(=) is the cluster (also a totally ordered set with cardinality L) that query point = belongs to, meaning that, all points in this cluster (query cluster) will be assigned to the same cell. The Influence Function is defined upon two totally ordered sets C₁ = (ck H and C(2) = (2))) F(C,, C(x)) sign(>) El=o d(c()) ())).
(1..., K), with r(z) arg max e (1,....k}
F(C,, C(x) where
As CCVD is flexible framework and can be applied D/Aro- here, as an example, we show how CCVD can be use to boost iVoro. In iVoro, the VD is induced ret....). that are feature means from the last layer Φ. Now, we arbitrarily extract L layers {o')) and generate the K totally ordered clusters generate the query cluster (x)) for the query example æ for Voronoi cell assignment. See Appendix for a summary of the notations and acronyms.
to construct CCVD and
3 EXPERIMENTS
In our geometric framework, starting from iVoro, the simplest prototype-induced VD model, we gradually add four components: (I) parameterized normalization (iVoro-N), (II) divide-and-conquer

---

Published as a conference paper at ICLR 2023
Table I: Comparison between the fully-fledged i'oro with state-of-the-art non-exemplar (marked by x) and exemplar-based (marked by v/ CIL methods in terms of the accuracy (in %) in the last phase and the average accuracy (Avg., in %) across all phases. imp. indicates the relative improvement upon the next best non-exemplar CIL method. &The best version of iVoro is shown here. See Tab. 2 for different versions of Voro. Note that RMM uses a 100-class subset of ImageNet that different from others (shown in blue).
CIFAR-100
Tiny ImugeNNet
ImageNett Subset
5 phases
10 phases
20 phases
5 ppases
10 phases
20 phases
10 phases
Methods
Avg.
Last
Avg.
Last
Axg.
Last
Axg.
Last
Axg.
Last
Avg.
Last
Avg.
Last
iCaRLew ctal
2017
$1.25
40.50
48.52
39.13
64.85
34.38
34.90
23.20
31.12
20.82
28.03
20.20
50.61
38,40
2017
59.13
45.00
53.91
45.38
50.79
40.88
46.08
34.43
43.42
53.333
38.08
27.65
611.9
50.06
FRI. (asthone 2011
60:15
50.13
55.91
47.63
52.70
42.63
47.5 56
35.46
45.76
34.77
40.61
20:60
63.40
52.91
UCI
100 2019
63.83
54.75
60.94
50.75
59.46
47.00
49:26
39.18
48.85
57.74
43.02
30.82
67.59
55.89
v RMM
68.86
59.00
67.61
59.03
78:47
71,40
x BWC
Kekparick al. 2017
24:23
9.00
21.15
5.50
16.26
7.75
18.83
5.98
15.90
3.39
12.57
5.00
20.26
9:03
x LwF
Lr Hoaem 2017
52.54
14:25
17.91
5.88
14:95
5.50
22:35
7.11
17:52
4.82
12:75
4.39
25.57
11.54
x LWF-NCC
Thller
2017
46.06
33.38
27.31
15.75
19.99
11.88
29.09
15:46
23:22
13.23
17.46
8.16
31.22
20.69
56
32:35
78
35.03
PASS
2021
63.88
55.75
60.07
49.13
56.21
48.75
09.55
41.86
47.30
39.38
42.04
32.86
02.26
x IVoro (Best))
83.57
74.40
83.52
74.39
81.24
71.45
K1.74
73.34
80.32
71.13
79.06
69.95
90.04
83.84
Mmp.1
+19.69
+18.65
+23.45
+25.26
+23.03
70
+31
56
+30.48
+32.92
+31.75
+37
+37
+27.78
-33.21
for progressive VD construction (iVoro-D), (III) augmentation consensus/integration (iVoro-AC/AD, and (IV) multilayer VD (iVoro-L.). In this section, our main goals are to: (1) validate the strength of every single component; (2) exhaust as many combinations of components as possible to see how different combinations collaboratively contribute to the overall result; and (3) investigate at which circumstances method does or does not work, by analyzing data size, number of layers, and quantitative uncertainty.
3.1 Datasets, Benchmarks, and Implementation Details. Three standard datasets, CIFAR-
100 (Krizhessky et al. 2009). TinyImageNet (Le & Yang 2013) and ImageNet-Subbee (Deng et al. 2009a) for CIL are used for method evaluation. We follow the popular benchmarking protocol in exemplar-free CIL. used by (Liu et al.) 20216] Zhu et al., 2021 Douillard et al.) [020] Hou et al.,
2019) in which the inital phase contains a half of the classes while the subsequent phases each has
5. To or 2g of the remaining classes. We mainly compare our method to non-exemplar methods
including EWC (Kirkpatrick et al. 2017). LwF (Li & Hoiem 2017). LwF-MC (Li & Hoiem 2017).
LwM (Dhar et 2019).. and MUC (Liu et al. 2020b), but we also compare with several recent exemplar-base methods iCaRL RRebutfi et al.) 2017). EEIL (Castro et al. 2018). UCIR (Hou et al., 2019). and RMM (Liu et al. 2021b) for reference. A ResNet-18 18 (He et al. 2016) model is used for all experiments. We follow PASS (Zhu et al. (02T) to train the feature extractor on the first phase data but freeze it afterwards for all subsequent phases. All classes are expanded via rotating the original image by 90', 180", and 270", See Appendix for more details about the implementations of all the 12 ablation methods in Tab. 2
CIFAR-100 (5 phases)
10F (10 phases)
CIFAR-100 (20 phases)
Ava
Ava.
Avg.
Accuracy
Accuracy
Accurscy
20
Number classes
Number classes
Number classes
EWC
LWF ICaRL-NOM
LWF-MC
LwM UCIR
PASS
|CaRL-ONN
EEIL
iVero
Figure 5: Top-1 classification accuracy on CIFAR- during 5/10/20 phases of CIL.
3.2 iVoro: Simple VD is A Strong Baseline. Surprisingly, by only using prototypes for VD construction, our baseline method i'oro can achieve competitive performance for short phases and much better results for long phases, compared to the state-of-th of-the--rtt non-exemplar CIL method. For example, the difference in accuracy in comparison to PASS is 0.29% w1t...991//...3... for 5/10/20-phass CIFAR- 100, -3.58% 1096/1..9%//44433 433% for 5/10/20-phase TinylmageNet, and 4.76% for ImageNet- Subset. We suspect that this is because the features generated by the frozen feature extractor can be
7

---

Published as a conference paper at ICLR 2023
Table 2: Ablation experiments by testing with different combinations of parameterized feature nor- ) and multilayer
malization (*), progressive VD ), augmentation consensus or integration VD (7). See Table G.T for the complete table.
CIFAR-100
TinyImageNet
ImageNet-Subset
phases
10 phases
20 phases
5 phascs
10 phases
20 phases
10 phases
Methods
Avg. (Las)
Avg. (Las)
Avg. (las)
Avg. (tas)
Avg. (lar)
Avg. (Las))
Avg. (Las)
iVoro 66.39 (50.5)
66.09 156.03)
62.33 152.38)
45.12 (38.27)
45,09 (38.29)
45,04 (38,29)
66.50 (53.40)
iVoro-ND
67.55 (57.25)
66.89 (9075)
64.65 (54.72)
51.83 (43.43)
50.71 (42,40)
50.17 (42.10)
69.07 (58.52)
IVoro-AC
81.38 (706a))
81,25 (70.63)
78, (66.14)
64.01 (55.26)
64.01 (55.28)
641.0 (55.29)
83.41 (71.00)
iVoro-Al
62.33 (50.18) 60.37 (50.18)
65.39 (99.113
55, 60.60 (48.11)
55.63 (48.12)
55..00 (48.11)
72.47 (00.66)
iVoro-NDAl
69.00 (56.35)
63.76 (52.57)
63.94 (57.52)
81.00 (7L.29)
79.64 (70.10)
78.17 (68.70)
86.92 (78.64)
*
iVoro-NDAC
82,31 (72.04)
82.29 (72.19)
80.53 (70.013
59.75 (49.78)
59.75 (49.90)
59.774 (49.780
84,31 (71.72)
*
Noro-NDACL.
83.57 (74.40)
83.52 (74.99
72.64 (61.14)
59.49 (50,00)
599.51 (50.10)
59.55 (50.18)
84.83 (76.24)
iVoro-NDAIL 77.57 (66.54)
72.50 (62.28)
81.24 (71.45)
81..4 (72.3))
80.22 (71.13)
79.08 (69.95)
90.04 (83.84)
satisfactorily separable by linear bisectors (Fig. 2 As we can see, the features for other methods all dramatically changing during the phases, but those for iVoro are all fixed, making incremental VD construction possible. Moreover, the accuracy of the last phase usually drops significantly with longer task sequence (e.g. 20 phases vs. 5 phases), but i'ooo is highly robust at the last phase, because the final VDs are the same no matter how many phases it goes through. These results show that iVoro works favorably with long phases. When parameterized normalization is applied, i iVoro-N further consistently improves upon i'Voro by up to 2.40% (10-phase ImageNet-Subset) (see Tab. (2) by encouraging the compactness of feature distribution. See Appendix H about the detailed analysis of Voro-N.
CIFAR-100 (20 phases)
TinylmageNet (20 phases)
ImageNet- (10 phases)
70-
IVoro
NOAL
70-
iVera-
M Mare-
NAC
66-
N
O
N
Accuracy
AC
Accuracy
AC
Accuracy
20
AC
62
NACL
50
NOW
Rooe
PASS
30-
P400
Melhads
Melhods
Methods
Figure 6: Illustration of the performance for the ablation methods shown in Table
ImsgeNNt-Subsel phases)
3.3 Normalization (iVoro-N) and D&C (iVoro-
EWC
D): Synergistic Effects. Our very baseline
LWF
LWF-NC
80
method, iVoro, ignores at which phase class
MUC
Accursoy
was learned, and computes prototypes indiffer-
PASS
80-
ently to construct the VD (i.e. 1-nearest neigh- bor model). To determine the decision bound- aries more subtly, iVoro-D focuses on the re- finement of the within-phase boundaries. These two components, iVVrr-N/DD can individually improve Voro, but also have collective impacts. For example, as shown in Tab. 2 and Fig. 6 iVoro-ND>> > Woro-N/iVoree necessary. More specifically, in the three datasets, iVoro-D makes the largest contribution to Tinylm- ageNet (1.75%-3,4 3.44%) than to CIFAR-100 (0.48%-0.966 can be explained by the fact that there are 100 classes in the first phase in the TinyImageNe dataset, while only 50 in the other two, making TinyImageNet harder dataset if only vanilla prototypes used to construct the VD.
ICAR--NCM
UCIR
Moro
Number of classes
Figure 7: Top-1 classification accuracy on ImageNet-Subset (10 phases).
> iVoro, corroborating that every single contribution is useful and
to ImageNet-Subset (0.72%). This
3.4 Why and When Will Augmentation Integration (iVoro-CC/AD Help? When augmentation consensus (iVoro-AC) or integration (iVoro-Al) is applied, the improvement is significant. For example, AVoro-AC obtains 13.76%, 17.00%, and 16.50% improvements upon iVoro CIFAR-100.
8

---

Published as a conference paper at ICLR 2023
TinylmageNet, and ImageNeet-uuee Subsett, respectively. iVoro-Al itself is worse than iVoro-AC, but if combined with normalization and D&C, it further elevates the accuracy by a large margin, c.g. as high as 68.70% (iVoro-NDAL) on 20- 20-phase TinyImageNet and 78.64% on 10-phase ImageNet-Subset To investigate the reason of this prominent improvement, we calculate the entropy-based geometric variance in class level and plot them as function of the Aaccuracy (i.e. the improvement in accuracy after augmentation integration is used), as shown in Fig. Sand Fig. G.2] Interestingly, there is a clear correlation between HV and Aaccuracy, and this is more notable on ImageNNet-ubbset (Pearson's R ~0..), probably because of its high resolution (224 x 224). This tendency suggests that the higher the variance within the assignments from augmented images to expanded classes, the better the improvement after using augmentation integration. See Appendix [ for uncertainty analysis, and Appendix N Appendix for class-leve/saamlle-bee analysis.
3.5 How Good Should the Feature Extractor Be?
NmageNettSuuset (10 phases)
As Vooo is heavily dependent on the feature extractor, which cannot be evolved in any way along the learn- ing process, one may wonder our method still work with a poorly trained feature extractor. To verify this,
Entropy based Geometric Variance t g 5
Pearson's -0.927 0.906
we gradually decrease the number of classes used to
train the feature extractor 6. As shown in Appendix ! compared with PASS, the best version of iooro still has 17.75%, 13.59%, 10.89%, and 1.60% improvements with 40, 30, 20, and 10 initial classes, respectively. This means that, even if there is no strong feature extractor, our method can still reach acceptable performance higher than the state-of-the-an the-art method.
020
436 6:33 615
020
02
6.0
AAccuacy
Figure 8: Entropy- geometric vari- ance (HV) in class level function of Aaccuracy (ImageNet-Subsee),
3.6 iVoro-L: VD Can Also Go Deeper. In this sec- tion, we extract the feature from the 3rd block and es- tablish a Cluster-to-cluseer Voronoi Diagram (CCVD) us- ing features from both lay- ers. As expected, shown in Appendix К] the overall performance degenerates sub-
Table 3: Comparison of iooo to joint training in which all classes are learned in the same phase. The accuracy from ResNet- on the original label set are marked bold, while @ denotes result upon the augmented label set (4x) ). The results of ioro on 5 5/10/20 phases of CIL are also shown. The best results are underlined.
Methods
CIFAR-100
TinyImageNet
ImageNNtt
ResNet-18
75.34
$7.37
80.44
ResNet-18@
66.94
$2.78
69.56
Joint
iVoro
64.21
47.56
78.54
iVoro-AC
81.65
67.90
92.72
stantially, e.g. iVoro-NAT drops from 35.84% 42.03%. However, and interestingly. if integrated with CCVD and D&C. iVoro-NDAIL obtains even higher accuracy of 72.34%. The
iVoro-Al
80.66
74.91
93.82
(Best)
74.40
74.39
71.45
72.34
71.13
69.95
83.84
final results are presented in Tab. [ Fig. K.3 and Appendix G Our final model, multilayer VD, surpasses all previous methods by large margin of 25.26%, 37. 37.09%, and 33.21 % on CIFAR-100, TinylmageNet, and ImageNet-Subseed respectively, even higher than exemplar-based CIL methods.
3.7 Comparison with Joint Training. In CIL, the classes are sequentially learned at each phase, whereas joint training simultaneously learns all the classes in the same phase, providing upper bound for our CIL experiments. In Tab. 3 iVoro (and its variants) is applied to joint training, and is compared with ResNet on both the original and expanded label sets. Although there is still substantial gap between i'oro (best) and the upper bound, the catastrophic forgetting is considerably overcome (see Appendix MM. In addition, and surprisingly, iVVro-AC/AI can also promote the performance of joint training, e.g. +6.31%, +17.54%, and +13.38% for CIFAR-100, Tiny Net,,,,,,et, and ImageNet-Subset, respectively, suggesting that our augmentation integration method is also beneficial to general training where relf-supervised label augmentation is involved.
CONCLUSION
In this paper, we use progressive Voronoi Diagram to model the class-incremental learning problem, and propose a number of new techniques that handle various aspects of this VD construction process that gradually and greatly improve the CIL performance. Thus, iVoro is shown to be flexible, scalable, and robust framework that strictly maintains the privacy of previous data. Our code is available at https//machunweii.gtee

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This research was supported in part by NSF through grant IIS- 1910492.
REFERENCES
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference Computer Vision (ECCV), pp. 139-154, 2018.
Franz Aurenhammer. Power diagrams: properties, algorithms and applications. SIAM Journal on Computing, 16(1):78-96, 1987.
Randall Balestriero, Romain Cosentino, Behnaam Aazhang, and Richard Baraniuk. The geometry of deep networks: Power diagram subdivision. Advances in Neural Information Processing Systems, 32:15832-15841, 2019.
Randall Balestriero, Sebastien Paris, and Richard G Baraniuk. Max-affine spline insights into deep generative networks. In International Conference on Learning Representations 2020, 2020.
Eden Belouadah and Adrian Popescu. 112m: Class incremental learning with dual memory. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 583-592, 2019.
Eden Belouadah and Adrian Popescu. Scail: Classifier weights scaling for class incremental learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp.
2266-1275 2020.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems, 33: 13:15920-15930 2020.
Francisco M Castro, Manuel Marén-Jiménzz, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In Proceedings of the European conference on computer vision (ECCV), pp. 233-248, 2018.
Hyuntak Cha, Jaeho Lee, and Jinwoo Shin. Co21: Contrastive continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9516-9525, 2021.
Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 532-547, 2018a.
Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. arXiv preprint arXiv:1812..04400, 2018b.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Mare' Aurelio Ranzato. On tiny episodic memories in continual learning. International Conference on Machine Learning (ICMI.) Workshop. 2019.
Danny Z. Chen, Ziyun Huang, Yangwei Liu, and Jinhui Xu. On clustering induced voronoi diagrams. In 54th Annual IEEE Symposium on Foundations of Computer Science, FOCS 2013, 26-29 October, 2013, Berkeley, CA, USA, pp. 399. IEEE Computer Society, 2013. doi: 10.1109/FOCS.200 49.
Danny Z. Chen, Ziyun Huang, Yangwei Liu, and Jinhui Xu. On clustering induced voronoi diagrams. SIAM J. Comput., 46(6): 2017. doi: 10.1137/15 1447444
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
10

---

Published as a conference paper at ICLR 2023
Matthias Delange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2021. doi: 10.1 109/TPAAM..2011.355447.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp.
lece, 2009a.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. leee, 2009b.
Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. Learning without memorizing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5138-5146, 2019.
Hu Ding and Jinhui Xu. Learning the truth vector in high dimensions. Journal of Computer and System Sciences, pp. 78-94, 2020.
Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-taske incremental learning. European Conference on Computer Vision, pp. 86-102. Springer, 2020.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint 1 2017.
Robert M French. Catastrophic forgetting in connectionist networks. Trendy in cognitive sciences, (4):128-135, 1999.
Siavash Golkar, Michael Kagan, and Kyunghyun Cho. Continual learning via neural pruning. Advances in Neural Information Processing Systems (NeurIPS) Workshop, 2019.
lan Goodfellow, Mchdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. Proceedings of the International Conference on Learning Representations (ICLR), 2014.
Tyler L Hayes, Kushal Kafle, Robik Shrestha, Manoj Acharya, and Christopher Kanan. Remind your neural network to prevent catastrophic forgetting. In European Conference Computer Vision, pp. 466 Springer, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377, 2021.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network. arXiv preprint arXiv: 153.05553..0531, 2(7), 2015.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 831-839, 2019.
Ziyun Huang and Jinhui Xu. An efficient sum query algorithm for distance-based locally dominating functions. Algorithmica, 82(9):2415-244) 2020. doi: 0011://000005300-0000000
Ziyun Huang, Danny Z. Chen, and Jinhui Xu. Influence-hased voronoi diagrams of clusters. Comput. Geom., 96: 101711116. 2021. doi: (0.101//.comgee.2221.11m
Ching Yi Hung, Cheng- Hao Tu, Cheng- En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu Chu-Song Chen. Compacting, picking and growing for unforgetting continual learning. Advances Neural Information Processing Systems, 32, 2019.
11

---

Published as a conference paper at ICLR 2023
Ahmet Iscen, Jeffrey Zhang, Svetlana Lazebnik, and Cordelia Schmid. Memory-efficiet incremental learning through feature adaptation. In European Conference on Computer Vision, pp. 699-715. Springer, 2020.
Ronald Kemker and Christopher Kanan. Fearnet: Brain-inspired model for incremental learning. Proceedings of the International Conference on Learning Representations (ICLR), 2017.
Ronald Kemker, Mare McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan. Measuring catastrophic forgetting in neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Trabska-Barwinska et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Master's thesis, 2009.
Abhishek Kumar, Sunabha Chatterjee, and Piyush Rai. Bayesian structural adaptation for continual learning. In International Conference Machine Learning, pp. 1850-5860. PMLR, 2021.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mmist/, 1998.
Der-Tsai Lee. On k-nearest neighbor voronoi diagrams in the plane. IEEE transactions on computers, 100(6):478-o) 487, 1982.
Hankook Lee, Sung Ju Hwang, and Jinwoo Shin. Self-supervised label augmentation via input transformations. In International Conference on Machine Learning, pp. 5714-5724. PMLR, 2020.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In International Conference on Machine Learning, pp. 3925-3934. PMLR, 2019.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pallern analysis and machine intelligence, 40(12):2935-29) 2017.
Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun. Mnemonics training: Multi-class incremental learning without forgetting. Proceedings of the IEEE/CVF conference on Computer Vision and Puttern Recognition, pp. 12245-12254, 2020a.
Yaoyao Liu, Bernt Schiele, and Qianru Sun. Adaptive aggregation networks for class-incremental learning. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2544 2553, 2021a.
Yaoyao Liu, Bernt Schiele, and Qianru Sun. Rmm: Reinforced memory management for class- incremental learning. Advances in Neural Information Processing Systems, 34, 2021b.
Yu Liu, Sarah Parisot, Gregory Slabaugh, Xu Jia, Ales Leonardis, and Tinne Tuytelaars. More classifiers, less forgetting: A generic multi-classifier paradigm for incremental learning. In European Conference on Computer Vision, pp. 699-716. Springer, 2020b.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Chunwei Ma, Yan Ren, Jiarui Yang, Zhe Ren, Huamming Yang, and Sigi Liu. Improved peptide retention time prediction in liquid chromatography through deep learning. Analytical Chemistry, 10818188081111888, 2018.
Chunwei Ma, Zhanghexuan Ji, and Mingchen Gao. Neural style transfer improves 3D cardiovascular MR image segmentation on inconsistent data. International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 128-136. Springer, 2019.
12

---

Published as a conference paper at ICLR 2023
Chunwei Ma, Ziyun Huang. Jiayi Xian, Mingchen Gao, and Jinhui Xu. Improving uncertainty calibration of deep neural networks via truth discovery and geometric optimization. In Uncertainty in Artificial Intelligence, pp. 15-85. PMLR, 2021.
Chunwei Ma, Ziyun Huang, Mingehen Gao, and Jinhui Xu. Few-shot learning via dirichlet tessellation ensemble. In International Conference on Learning Representations, 2022a. URL https: //openreviee ew. net feeet/oormmmiidddkk Sct.eeedodccciickko
Chunwei Ma, Ziyun Huang, Mingchen Gao, and Jinhui Xu. Few-shot learning via dirichlet tessellation ensemble. In International Conference Learning Representations, 2022b. URL https:
//openreview. net. wtt.:e/ooooimiiioiiiiiaaaaa
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp.
7765-7773, 2018.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Alexandra Miere, Thomas Le Meur, Karen Bitton, Carlotta Pallone, Oudy Semoun, Vittorio Capuano, Donato Colantuono, Kawther Taibouni, Yasmina Chenoune, Polina Astroz, al. Deep learning- based classification of inherited retinal diseases using fundus autofluorescence. Journal of Clinical Medicine, 911))33333 2020.
Oleksiy Ostapenko, Mihai Puscas, Tassilo Klein, Patrick Jahnichen, and Moin Nabi. Learning to remember: A synaptic plasticity driven framework for continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 11321-11329, 2019.
German Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
Quang Pham, Chenghao Liu, and Steven Hoi. Dualnet: Continual learning, fast and slow. Advances in Neural Information Processing Systems, 34, 2021.
Petra Poklukar, Vladislav Polianskii, Anastasia Varava, Florian Pokorny, and Danica Kragic. Delaunay component analysis for evaluation of data representations. arXiv preprint arXiv:2202.06866. 2022.
Vladislav Polianskii and Florian T Pokorny. Voronoi boundary classification: high nigh-ddmeeniionl geometric approach via weighted monte carlo integration. In International Conference on Machine Learning, pp. 5162-5170. PMLR, 2019.
Vladislav Polianskii and Florian T Pokorny. Voronoi graph traversal in high dimensions with applications to topological data analysis and piecewise linear interpolation. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp- 2154-2164, 2020.
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On the expressive power of deep neural networks. In international conference on machine learning (ICML), pp. 2847-2854, PMLR, 2017.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pallern Recognition, pp. 2001-2010, 2017.
Mohammad Rostami. Lifelong domain adaptation via consolidated internal distribution. Advances in Neural Information Processing Systems, 34, 2021.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick. Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 01.66666666777 2016.
13

---

Published as a conference paper at ICLR 2023
Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Baabska-Barwinsk,, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In International Conference Machine Learning, pp. 4528 -4337. PMLR, 2018.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, pp. 4548 4557. PMLR, 2018
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
Chawin Sitawarin, Evgenios Kornaropoulos, Dawn Song, and David Wagner. Adversarial examples for k-nearest neighbor classifiers based on higher-order voronoi diagrams. Advances in Neural Information Processing Systems, 34, 2021.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning, Advances in Neural Information Processing Systems (NIPS), 30:4077-4087, 2017.
Tanveer Syeda- Mahmood, Ken CL. Wong, Yaniv Gur, Joy T Wu, Ashutosh Jadhav, Satyananda Kashyap, Alexandros Karargyris, Anup Pillai, Arjun Sharma, Ali Bin Syed, et al. Chest x-ray report generation through fine-grained label learning. In International Conference on Medical Image Computing and Computer-Assiste Intervention, pp. 561-571. Springer, 2020.
Shixiang Tang, Peng Su, Dapeng Chen, and Wanli Ouyang. Gradient regularized contrastive learning for continual domain adaptation. In Proc. 35th AAAI Conf. Artif. Intell,(AAAI), pp. 2-13, 2021.
Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint 1rX1v::904.07734 2019.
Riccardo Volpi, Diane Larlus, and Grégory Rogez. Continual adaptation of visual representations via domain randomization and meta-learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4443 4453, 2021.
Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Growing brain: Fine-tuning by increas- ing model capacity. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2471-2480, 2017.
Zichao Wang, Randall Balestriero, and Richard Baraniuk. max-affine spline perspective of recurrent neural networks. In International Conference on Learning Representations, 2018
Yue Wu, Yinpeng Chen, Lijuan Wang. Yuancheng Yc, Zicheng Liu, Yandong Guo, Zhengyou Zhang, and Yun Fu. Incremental classifier learning with generative adversarial networks arXiv preprint arXiv: 1802. 00853, 2018
Yue Wu, Yimpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu, Large scale incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 374-382, 2019.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. Proceedings of the International Conference on Learning Representations (ICLR), 2017.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pp. 3987-3995. PMLR, 2017.
Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shu-Tao Xia. Maintaining discrimination and faimess in class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13208-13217, 2020.
Fei Zhu, Yao Zhang, Chuang Wang, Fei Yin, and Cheng-Lin Liu. Prototype augmentation and self-supervision for incremental learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5871-5880 2021.
14

---

Published as a conference paper at ICLR 2023
A EXTENDED RELATED WORK
A.1 RECENT PROGRESS IN INCREMENTAL LEARNING
Incremental Learning (Rebuffi et al. 2017 Hou et al. 2019 Wu et al. 2019 Zhu et al. 2021 Liu et al.
2021b) requires continuously updating model using sequence of new tasks without forgetting the old knowledge, which is also referred to as continual learning (Parssi et al., 2019 Delange
et al., 2021 Chaudhry al.] 2019). The main challenge of incremental learning catastrophic
forgetting (McCloskey& & Cohen 1989 French, 1999, Goodfellow et al.) 2014 Kemker et al. [2018).
where deep neural network is prone performance deterioration on the previously learned tasks as the model parameters overfit to the current data to optimize the stability-plastica trade-off.
A.1.1 CAUSATION OF CATASTROPHIC FORGETTING
Generally speaking, in deep neural networks, catastrophic forgetting McCloskey & Cohen 1989 Goodfellow et al. 20144 Kemker et al.] 2018) comes from two sources: the feature distribution shifting of the old classes in the feature embedding space as well as the confusion and imbalance of the decision boundary of the classifier when learning new task. The former is caused by the excessive plasticity and parameter changing of the feature extractor of the deep model during finetuning on unseen data/classes, thus deteriorates the feature extraction and prediction on previous classes; while the latter is due to the highly overfitting and bias of the classifier on current task as well as the overlapping between the representation of new and old classes in the feature space.
A.1.2 INCREMENTAL. LEARNING SCENARIOS
Three common incremental learning scenarios are widely explored in recent papers (Van de Ven & Tolas 2019).
Task-incremental learning (TIL) Ostapenko et al. 2019 Shin et al. 2017 Kirkpatrick et al. 2017;
Zenke et al.] [017]] Wu et al. 2018 Lopez-Paz & Ranzato 2017 7017:c1 Chaudhry et 2019 Buzzega et al.
2020 Cha et al. [2221] Pham et al. Ferando et al. (2017) incrementally learns a sequence of tasks in multiple phases, where each task contains unseen data of a new set of classes. To mitigate catastrophic forgetting, TIL assumes a simple setting where the task identity is known at inference time. The methods under this scenario keep leaning new lask-independent classifiers or growing the model capacity by attaching additional modules (e.g. kernels, layers or branches), each corresponding to specific task or subset of classes. Since the task ID) is available during inference, the model can directly select proper classifier or module without inferring task identity, which effectively solves the confusion boundary and classifier bias between old and new tasks, and often achieves satisfying performance. However, knowing task identity at test time is normally unrealistic in real-world situation hence restricts practical usage. Moreover, may incur unbounded memory consumption for super long task sequence if increasing the model capacity for new tasks.
Unlike TIL constrained by the availability of task identity, dlass-incremente learning (CIL) Liu
ct al., 2020b
Belouadah & Popescu, 2019; Chaudhry et al.) 2018a Zhu et al. 2021 Douillard
et al.
2020: Rebuffi et al. 2017 Hou et al.
2019 Liu et al. 2020a 202 la b) updates a unified
classifier for all classes learned so far while task identity is no longer required during inference. To
compensate the missing task identity and alleviate forgetting issue, branch of works (Rebuffi et al.
2017: Hou et al. 2019 Liu et al. bb2 Douillard et al. 2020 Castro et al.) 2018 Wu al.) 2019) alternatively follow a memory-
setting, in which a limited number of samples from old classes (c.g.. 20 exemplars per class) is stored and maintained in a memory buffer, which are later replayed to jointly train the model with current data (normally combined with knowledge distillation) in order to constrain the feature distribution shifting of the old classes and the decision boundary bias of the classifier. However, their performance deteriorates with smaller buffer size, and eventually, the storing and sharing of previous data, e.g. medical images, may not be feasible when memory limits and privacy issue are taken into consideration. Given the potential memory issue, another direction of works (Kirkpatrick et al., 2017: Zenke et al. 2017 Li & Hoiem, 2017 7 Dhar et al. 2019 Zhu et al. 2021) intend to explore CIL in much challenging setting without memory rehearsal, mainly based on regularization and knowledge distillation techniques, which is known as exemplar-fres free CIL. In this paper, are following this CIL. setting.
15

---

Published as a conference paper at ICLR 2023
innaiinniccrmmmntall learning (DIL) (Rostami 2021 Tang et al. 2021 Volpi al. 2021). different from the aforementioned two scenarios, incrementally learning new domains of the same classes in each phase. Some domain adaptation techniques, e.g. meta learning, data shifting, domain randomization, are implemented in DII. to increase the model robustness and generalizability to handle various domain distributions. Since this scenario is not quite related to this paper, no detailed discussion will be included.
A.1.3 CATEGORIES OF INCREMENTAL LEARNING METHODS
There are three categories of existing IL methods to overcome catastrophic forgetting (Delange et al. 2021].
Regularization-batt based methods constrain the plasticity of the model to preserve old knowledge. This can be addressed by directly penalizing the changes of important parameters for previous tasks (Aljundi et al. 2018 Chaudhry et al., 2018a Kirkpatrick et al.] 2017 Zenke et al. 2017
Kumar et al. 2021p or regularizing the gradients when training unseen data (Lopez-Paz & Ranzato 2017,Chaudhry et al. 2018b). Knowledge distillation is another regularization solution, which is widely used in various IL methods to implicitly consolidate previous knowledge by introducing regularization loss term on model representations, including output logits or probabilities (Li &
Hoiem] 2017; [Schwarz et al. 2018 Rebufh et al.) [217]] Castro al., (018) and intermediate
features (Hou et al. 2019; Dhar et al. 2019; Douillard et 2020) Zhu et al. (021). Some other
works focus on correcting the classifier bias on new classes (Belouadah & Popescu 2019 Wu et al.,
2019 Belouadah & Popescu| 2020 Zhao et al.) 2020).
Rehearsal-based methods either store and replay a limited amount of exemplars from old classes as
Chaudhry et al. 2019 Buzzega (et al. 2020) or embedded features (Hayes et al. 2020 Iscen et al. 2020) to jointly train the model in the incremental phases, alternatively generate exemplars of previous classes (Ostapenko et al.,
raw images (Rebuffi et al. [017; Hou al. 2019;; Liu et al. 202 lb
2019; (Shin et al. 2017: Wu et al.] [018 Kemker & Kanan 2017). The former relies on memory buffer for all learned classes, where the performance is constrained by the buffer size limits and it is impracticable when data privacy is required and storing data is prohibited. The latter requires continuously learning deep generative model, which is also prone catastrophic forgetting thus the quality of generated exemplars is not reliable.
Architecture-basee methods aims at dynamically adapting task-specific sub-network architectures, which requires task identity to select proper sub-network. Some works directly expand the network by adding new layers or branches (Rusu al., [016 Li al., [019 Wang et al., 2017 Yoon et al. 2017), which is limited in practice due to unbounded model parameter growth. Others freeze partial network with masks for old tasks (Golkar et al. 2019 Hung et al. 2019 Mallya & Lazebnik] 20188) Serra et al. 2018), but suffering from running out of model parameters for new knowledge. The architecture- based methods are usually combined with memory buffer and distillation, and can achieve good results.
Our work is focusing on the most challenging but also the most practical non-exemplar class- incremental learning problem, which is general real-world scenario when no old data can be stored due to memory limits or data privacy and task identity is unavailable during inference, with the constraint of fixed model capacity in the same time.
A.2 COMPUTATIONAL GEOMETRY FOR DEEP LEARNING
Computational geometry is an emerging perspective for studying various aspects of deep learning. The geometric structure of deep neural networks is first hinted at by (Raghu et al. 2017) which reveals that piecewise linear activations subdivide input space into convex polytopes. Afterward, (Balestriero al. 2019) points out that the exact structure a Power Diagram (PD, a generalized form of Voronoi Diagram) (Aurenhammer) 1987) which is subsequently used to explain the recurrent neural networks (Wang et [ 8) and generative models (Balestriero et al. 2020). The Power Diagram (or Voronoi Diagram) subdivision, however, is not necessarily the optimal model for describing the partitioning of deep/intermediate feature spaces. More recently, several works in computational geometry (Chen et al. 2013 2017 Huang et al.) (021) use influence function F(C, 2) to measure the joint influence of all objects in C on a query z to build Cluster-induces Voronoi Diagram (CIVD), providing an advanced reform of the classical Voronoi Diagram. Observing that Prototypical
16

---

Published as a conference paper at ICLR 2023
Network (Snell et al., 2017). widely adopted metric-based few-shot learning (FSL) method, is essentially Voronoi Diagram in the feature space, Deep Voro (Ma et al. (022b) first unifies various kinds of FSL methods, and then constructs CIVD by incorporating heterogeneous features, achieving the state-of-the-arr performance FSL.
Besides FSL. Voronoi Diagram subdivision has also been used for deep learning uncertainty calibra- tion eM et al. 2021 adversarial robustness (Sitawarin et al. 2021). topological data analysis (Polian skii & Pokorny 2019 2020 Poklukar et al. 2022), and medical applications (Ma et al. 2018 2019). In this paper, distinct from the three aforementioned lines of research (i.e. regularization-bnn
rehearsal-based and architecture-bai methods), we propose the geometry- based CIL method iVoro (and its variants), inspired by Voronoi Diagram subdivision.
B DEMONSTRATIVE ILLUSTRATION ON MNIST DATASET IN 2D SPACE
In 2 MNIST [eCun 1998), a small and simple dataset, was used for the illustration of four methods, fine-tuning, PASS (huu et al. 202T), , iVoro, and iVooro-C, because of the convenience of embedding the examples into R The total 10 classes are split into sequence of 4, 3, and 3 classes. R8 model is used as the feature extractor for all four methods. (A) In fine-tuning, the model is firstly trained on the classes in the first phase, and then fine-tuned only on the subsequent and classes in phase and phase 3. (B) In PASS, SSL-based label augmentation is applied on all three phases and expands the classes to be 16, 9, and classes. The default hyper-parameternn are used to train PASS (i.c. the weight for knowledge distillation is 10 and the weight for prototype augmentation is 10). To ensure the final subdivision of space is Voronoi diagram, Thm. 2.1 (i.e. Voronoi diagram reduction in AlgorithmiD T is applied during the training of fine-tuning and PASS. (C)
In iVoro, the feature extractor from the first phase of (B) is frozen and used without tine-tuning for all the subsequent phases. The feature means are calculated as prototypes and no feature transformation used. Note that only the features from the original images without rotation are used in i'Vro.. (D) The only difference with (C) is that all the expanded classes are also considered as independent cells, allowing for further integration.
Result Analysis. For (A) fine-tuning and (B) PASS, the model's 's accuracy for data at individual phases are also shown in shadow. Fine-tuning are able achieve near-perfect prediction for classes the current phase locally, but fails to maintain satisfactory performance on any historical class (accuracy ~0%). PASS, on the other hand, basically deteriorates slightly on the classes from the first phase, due to the high weights on the KD loss and the prototype loss, but it also becomes almost incapable of learning on new classes (accuracy ~0%). iVoro, i.e. the simplest t-nearest-neighbor model, surprisingly obtains superior accuracy (64.84%, 24.28% higher than PASS) by only using fixed feature extractor trained from only classes (16 expanded classes). iVoro-AC achieves comparable result (61. (7%) with iVoro, but the 2D embedding makes it harder demonstrate the efficacy of our proposed method. From this 2D) illustration, it is obvious that the much higher performance of iVoro/iVoro-AC: is achieved through much better space partitioning.
C NOTATIONS AND ACRONYMS
In this section, we list all the notations used in the Methodology in Tab. C.I the notations and acronyms for various geometric structures used in the paper in Tab. C.2 and all ablation methods in Tab.
17

---

Published as a conference paper at ICLR 2023
Table C.I: Complete list of all notations used in Methodology
Notations
Descriptions
T
total phase,
current phase, E {1. [1.... 73}
historical phase, E {1....,) 1}
Dr
dataset phase
(t..)
data (image) and label in phase 1, i E {l, Neh. fl,,., 73 the set of classes in phase
Ce
C..
the h class in phase
Ni.k
number examples in class at phase
N:
number of all examples in phase i, i.e. N; Ek= Nt.6
feature extractor (typically deep neural network)
(
feature extractor, but only outputs the features from the 1 layer fcature for æ. i.c. 2 df x z E R"
classification bead, can be either Voronoi Diagram, or logistic regression prototypical Voronoi center for phase and class Cr,6 linear probing-induced induced Voronoi center for phase and class Cr.k La normalization
Cr,k
Gr.k
f
linear transformation 9, (=)
7)
Tukey's ' ladder of powers transformation, parameterized by linear weight and bias for class C, parameters for the linear bisector T z' = index of four rotations, E {0, 1, 2, 3)
We,,, be,
v.g
((a.a') R
the collection of the distances from oi x ), to K classes with rotation index a' Entropy- based geometric variance
HV
Table C.2: Notations and acronyms for VD, PD, and CCVD, three geometric structures used in the paper.
Geometric Structures
Acronyms
Notations
Description
center for Voronoi cell we. {1,., dominating region for center ct, & {1,... K}
Voronoi Diagram
VD
Ca
center Power cell wy, E (1..,, K}
Power Diagram Aurenhammerr
1987
PD
weight center cs, & [L....] dominating region for center & {1,.., K}
C₁
cluster as the "center" for CCVD cell we, k (1...,
Cluster-oocluster
CCVD
Wh
dominating region cluster the cluster that query point a belongs
C(s)
Vornnii Diagram
2112
influence function C(=)) from C2 to query cluster C(=)
magnitude influence
18

---

Published as a conference paper at ICLR 2023
Table C.3: Complete list of all variants iVoro.
Augmentation Consensus
Augmentation Integration
Methods
Prototype
Normalization
D&C
Multilayer VD
iVoro
x
iVoro-N
x
x
x
X
iVoro-D
X
X
X
iVoro-ND
X
X
iVoroAAC
x
X
X
*
iVro-AAI
x
X
V
x
Voro-NAC
V
X
x
iVor--NA
X
V
ioroo-DDC
X
iVoro-DDAl
iVoro-NACL
V
X
iVoro-NAIL
X
NVor--DAAL
X
iVoro-NDAIL
19

---

Published as a conference paper at ICLR 2023
D DATASET DETAILS
Here we give the detailed statistics the three datasets used the paper. Augmentation consensus (iVoro-AC) and augmentation integration (iVoro-AI) work more favorably with images with higher resolution, e.g. ImageNet-Subset (see Fig. 8). as the rotation operation makes less sense if the image too blur.
Table D.I: Summarization of the datasets used in the paper.
Datasets
Image size
Training Images
Total Classes
Number of Phases
CIFAR-100 (Krizhevsky al. 2009
32 32 x
60000
100
10, 20
TinylmageNer (Le & Yang 2015
64 64 x
100000
200
10,
ImageNet-Sutee (Deng 2009b
224 x 224
130000
100
10
E POWER DIAGRAM SUBDIVISION AND VORONOI REDUCTION
In this section we provide the proof of Theorem 2.1
Lemma E.1. The vertical projection from the lower envelope of the hyperplanes (Ix(2)) WTx bk) De onto the input space IR defines the cells of PD.
Theorem 2.1 (Voronoi Diagram Reduction (Ma et al. 2022a)). The linear classifier parameterized by W, b partitions the input space R" to a Voronoi Diagram with centers {či,..., Cx given by Ck = W. b = 1 W 1, 1,..,, K.
Proof. We first articulate Lemma E. and find the exact relationship between the hyperplane (z) and the center of its associated cell in R". By Definition 2.1 the cell for point x E I2 is found by comparing d(z, cx)2 V4 for different so we define the power function p(z, S) expressing this value
S)= = (z -
in which S C IR." is a sphere with center and radius r. In fact, the weight associated with center Definition 2.1 can be interpreted as the square the radius r². Next, let denote a paraboloid 2 let ( be the transform that maps sphere S with center u and radius into hyperplane
II(S): y = 2z u u +
can be proved that II is a bijective mapping between arbitrary spheres in I" and nonvertical hyperplanes in B=+1 that intersect U (Aurenhammer 1987). Further, let z' denote the vertical projection of z onto U and z" denote its vertical projection onto I[S),, then the power function can be written
(4) which implies the following relationships between sphere in R and an associated hyperplane in ga++ (Lemma 4 in (Aurenhammer 1987)): let S₁ and S₂ be nonco-centeric spheres in IR", then the bisector of their Power cells is the vertical projection of II(S;) II(22) onto R Now. we have direct relationship between sphere S, and hyperplane IT(S). and comparing equation with the hyperplanes used in logistic regression (T, W. z by gives us
S) d(z,=') d(z,="),
1
u 2
Wv
1
2 by + 4
W 11.
Although there is guarantee that by + model, we can impose constraint on - keep it be zero during the optimization, which implies
1 is always positive for arbitrary logistic regression
b
W 1
(6)
By this way, the radii for all spheres become identical (all zero). After the optimization of logistic regression model, the centers { W. will be used probing-induced Voronoi centers.
20

---

Published as a conference paper at ICLR 2023
F IMPLEMENTATION DETAILS AND RESULT ANALYSIS OF COMPREHENSIVE ABLATION STUDIES
iVoro. We generally follow the protocol of PASS (Zhu et al. 2021) to train the feature extractor but only on the data from the first phase, i.e. 50 (for 5/10 phases) or 40 (for 20 phases) classes of CIFAR- 100. 100 classes of TinyImageNet and 50 classes of ImageNet-Subbe the results of PASS, using the same nyper-parameters, e.g. the weight for the knowledge distillation loss set at 10, and the weight for the prototype augmentation loss set at 10. For i'ooro (and all its subsequent variants), the trained model is frozen after the first phase and throughout all the remaining phases. At phase I, the prototypical centers {c} are computed for both the current and the historical phases 7 E (1,-... (), and are used to construct the Voronoi Diagram.
We also reproduce
The simplest Voro method (i.e. the vanilla Voronoi Diagram, or 1-nearest-neighhor) can already achieve comparable even better results than the state-of-the-an non-exemplar CIL methods. For example, the difference in accuracy in comparison to PASS is 0.29% %/3.6%3.. 63% for 5/10/20- phase CIFAR-100, -3.58% 1.0%%/5443% for 5/10/20-phase Imagen, and 4.76% for 10-phase ImageNet-Subset. Notably, there is always significant elevation of accuracy on long-phase data, suggesting the continuous fine-tuning of model, even with improved loss functions, tends forget seriously on earlier data. With a fixed feature extractor, iVoro has shown an improved ability to overcome catastrophic forgetting. On the other hand, for short-phase data, iVoro similar or worse than the tateeoffthhe-at art method, probably because of the prototypical centers are computed without considering data distribution (i.e. simply the mean of features).
iVoro-N. To inspect the effectiveness of the parameterized feature transformation, we apply L normalization with//ithout Tukey's 's ladder of powers transformation (A varying from 0.3 to 0.9), and compare with iVoro. Generally, the improvement acquired from the feature normalization is more prominent on more complex datasets (e.g. Imngmmeeeeet and ImageNet-Subseee or simpler datasets with longer phases (e.g. CIFAR-100 with 20 phases), with improvements ranging from 1.65% to 2.40% higher than Vooro. The detailed analysis is presented in Sec. [
Woro-DDVVoro-ND. The detailed algorithm of iVoro-D is presented in Alg. Specifically, for each phase 7 E [1, ..., ft. the local dataset D, is used to train a logistic regression model (restricted by Thm. 2.Ip with weight decay ( 0.0001 and initial learning rate at 0.001. The result is also shown in Tab. 2 Aided by the D&C algorithm and local logistic regression, iVoro- is consistently better than iVoro, e.g. 1.48%-0.96% higher on the CIFAR-100 dataset, 1.75%~~ on the TinylmageNet dataset, and 0.72% higher on the ImageNet-Subset dataset. When further combined with feature normalization, iVoro-ND achieves even higher accuracy, 54.72% on 20-phase CIFAR-100, 42. 10% on 20-phase IinylmageNet, and 58.52% on ImageNet-Subset. For comparison, PASS reaches 45% on 20-phase CIFAR-100 32.86% on 20-phase TinylmageNet, and 50.63% on ImageNet-Subset. Therefore without incorporating more sophisticate techniques like iVoro- Rao-R/iiorrr AC/iVoro-L, the Voro-ND method can already surpass previous state-of of-the-arr art method by a large margin e.g. 249.7////994447788999
higher
While the previous variants of iVoro only consider the prediction the original image/class, here we show that the prediction can be substantially improved by augmentation consensus (iVoro-AC) and augmentation integration (iVoro-Al) proposed in this paper. Specifically, iVoro-AC improves upon iVoro by 13. 76.6~1.45.8555 on CIFAR-I00, 100, by 16. 66.99%~17.000% on TinylmageNet, and by 16.50% on ImageNet-Subset. i'Voro-Al itself generally works worse than iVoro-AC, e.g. improves up 6,73% on CIFAR- up to 9.82% on TinylmageNet, and 5.26% on SrageNNet-Subet,, but if combined with feature normalization, i iVoro-NAl performs much better than iVoro-NAC on TinyImageNet (up 65.83%) and ImageNet- When augmentation consensus/interraion is used, the previous variants iVVoroN and iVoro-D can all be promoted. Generally, adding an additional component will bring in more performance gain, as shown in Tab. 2lin detail.
(77.12%)
We further validate the multilayer VD for multiple feature spaces. As proof of concept, we only extract the feature from the third block to build an additional VD and conduct the integration using CCVD (Def. 2.3 with y set at 1. Compared with iVoro- further improves both average accuracy and last accuracy on
NAC, multilayer VD CIFAR- 100 under 5/10 phases settings (2.36% better on last phase, respectively), which also realizes the highest performance on the given settings among all ablation settings. The final
21

---

Published as a conference paper at ICLR 2023
performance ImageNettSubsset is also improved by 2.52%. In the meanwhile, multilayer VD does not make obvious difference on TinylmageNet. When adding multilayer VD to iVVro-NAI, labelled as Voro-NNAL. significant performance gain is observed on all experiments of CIFAR-100 (9.1 9.1%/9.11 1%/15.76% average accuracy increments on 5/10/20 phases) and large improvement is also achieved ImageNet-Subset (6.06% final accuracy growth). On the contrary, only limited gain is presented on IinylmageNNt. The above ablation results may demonstrate that multilayer VD with augmentation integration works extraordinarily well for small class (100 classes), but not very effective when class number increases.
Robustness Analysis. We run PASS 5 times on CIFAR-100 with the 10-phase setting, and the last accuracies (%) are 48. 48.25%, 49.03%, 53.03%, 53.95%, and 54.75%. respectively, with mean and standard deviation (std) being 51.80% 2.6%%; meanwhile, we also test PASS for runs on ImageNet-Subset with 10 phases, and the last accuracies (%) are 49.85%, 50.63%, 51.03% 51.88%, and 52.52%, respectively, with meanttstd being 1..18%%-..94% As shown above, even though PASS achieves relatively good accuracy average, the training is not very stable on CIFAR- 100, where the difference between the highest and lowest accuracy on CIFAR- while the performance ImageNet-Subsee is slightly better regarding the robustness, PASS still ranges from 49.85% to 52.52%. On the contrary, compared to PASS, our iVoro method is naturally robust various datasets with fluctuation in performance, due the frozen feature extractor and unbiased classifier based on VD.
as large as 6.5%;
ImageNat-Subee (10 phases)
PASS
IVer
Ném
Ners-A
Moroht
Accuracy
Accuracy
NDX
Accurazy
Nore NooNNN
MAC Vershil MDALL
NN
Number of Gleses
Num ber of dlasses
Number ber cissses
Figure F.1: Top-1 classification accuracy on SuaseNet-Subbet with all 12 ablation methods during 10 phases of CIL.
22

---

Published as a conference paper at ICLR 2023
phases)
PASS
Voro
Noro NACL Miro-Ni
Nom
Mers-c
MereAAC
Noro N
(yjro-A)
VeroWW
Noro NDNL
Aoro
Hore-DDW
Ditro- NDV
Accurazy
Accurssy .
Accurasy
Number of cheses
Number of dasssee
Number of classes
CIFAR-100 (10 phases)
Wore
Maro
NACL Mar--Al
Aem
Noro
Nobo A)
Vorc Avl
worr NDWL
(ibrai)
Marc-MWW
Ietros NM
Accuracy
Accuracy sp_
TVtro NDW
Accuracy
Number of cissess
Number of dassses
Number bet of classes
CIFAR-100 (20 phases)
100
Nor
*-
NM
Acciracy
Aocuracy
(Voor NDW
229200366000
3-0------- 20000 20000000
Number
of
Number
Figure F.2: Top-1 I classification accuracy CIFAR-100 with all 12 ablation methods during 5/10/20 phases of CIL.
23

---

Published as a conference paper at ICLR 2023
TinylmageNel (5 phases)
PASB
Nom Noro Aóro MiroN
Mess-c
1Mtr NAC
Accurasy
Accurasy
Accurany
Moro (Vor NAC
VonshiA Vorr Mill VorssDDAL
(Vor NN
(Vors-DA
Number of Cheses
Number of dassee
Number of classes
TinylmageNet (10 phases)
Wore
N
Noro Al
(irra)) NoroND
Accurscy
Accuresy
NOW
Number of ciseees
Number of dasses
Number classes
TinylmageNet (20 phases)
Nob
Acciracy
Accirey
NDW
Accurasy
VersNAC
2
Number
Number
Figure F3: Top-1 classification accuracy TinylmageNet with all 12 ablation methods during 5/10/20 phases of CIL.
24

---

Published as a conference paper at ICLR 2023
G ADDITIONAL FIGURES AND TABLES
Table G.I: Comprehensive ablation experiments by testing with different combinations of the five components: parameterized feature normalization (*), progressive VD via D&C (+), augmentation consensus or integration (+) ) and multilayer VD for multiple feature spaces (V). The solid symbol (*+++T) indicates corresponding component applied while the grayed symbol (* & T) means the component is ablated. To clearly show the improvement contributed to by certain compo- nent, the colored numbers represent the relative improvements compared to certain row indicated by the triangle (D>*о*>>) with the same color. The red, blue, or green background indicates when Voronoi Diagram subdivision, augmentation consensux/ineegration or multilayer Voronoi Diagram introduced, respectively.
CIFAR-100
TnyIImageNe
ImageNet-Suur
phases
10 plases
20 plases
5 phases
10 phases
20 phases
10 plases
Methods
Avg.
Last
Avg.
Last
Avg.
Last
Avg.
Last
Avg.
Last
Avg.
Last
Avg.
Last
VVoro
56.05
62.33
52.38
45.12
38.29
45.04
38.29
66.50
55.40
-VoroNN
66.80
56.39
66.51
56.40
63.50
54.03
47.40
40.48
47.35
40.48
47.29
40.48
68.19
57.80
+0.42
+0.34
+0.43
+0.35
+1.17
+1.65
+2.29
+2.21
+2.26
+2.19
+2.25
+2.19
+1.69
+2.40
Vooro.D
56.53
63.84
53.34
50.46
48.33
40.04
67.24
56.12
+0.85
+0.89
+0.47
+0.48
+1.51
+0.96
+5.34
43.44
+3.96
+2.24
+3.28
+1.75
+0.74
+0.72
Noro-ND
67.55
57.25
66.89
56.75
64.65
54.72
51.83
43.43
50.71
42.48
50.17
42,10
69:07
58.52
+0.74
+0.86
+0.38
+0.35
+1.15
+0.69
+4.42
+2.95
+3.36
+2.00
+2.88
+1.62
+0.88
+0.72
AC
81,38
70.63
81,25
70.63
78.16
66.14
64.01
55.26
64.01
55,28
64.00
55,29
83.41
71.90
+5.00
+14.58
|+15.16
+14.58
+15.84
13.77
+18. 90
+16.99
+11.99
+16.99
+18.96
+17.00
16.90
* 16.50
iVoro-AT
62.33
50,18
60.37
50.18
65.39
59.11
55.60
48.11
55.63
48,
55.50
48.11
72.47
60.66
-4.05
-5.87
-5.72
-5.87
+3.07
+6.73
+10.48
+9.9
+10.54
+9.83
+10.46
+9.82
+5.97
+5.26
IVoro-NAC
82.31
72.04
82,29
12.19
80.53
70.01
39.75
49.78
59.75
49.80
59.74
49.
84.31
73.72
*
+1.41
+1.04
+1.56
+2.36
+3.87
-4.26
.4.26
-5.51
+0.90
+1.82
iVoro-NAl
62.53
50.42
60.77
50.62
62.43
56.54
73.21
65.84
73.14
65.90
73.13
65.83
86.04
77.12
+0.24
-0.41
+0.44
-2.97
-2.57
+17.61
+17.73
+17.51
+17.78
+17.63
+17.72
+13.57
+16.46
IVoRo-NDAI
69.00
56.35
63.76
52.87
63.94
57.52
81.00
71.29
79.64
70.10
78.17
68.70
86.92
78.64
+6.47
+5.93
+2.99
+2.25
+1.51
40.98
+7.79
+5:45
+6.50
+4.20
+5.04
+2.87
+0.889
+1.52
VVoo-NNCL
74.39
72.64
61.14
59,49
50.10
59.52
50.13
84.83
76.24
+1.25
+2.36
+1.23
+2,20
-7.89
-8.87
-0.26
+0.31
-0.24
+0.30
-0.22
+0.35
+0.51
+2.52
VVro-NNILL
71.63
61.14
69.87
60.37
78.19
70.35
74.42
67.34
74.35
67.37
74.39
67.37
89.38
83.18
+9.10
+10.72
+9.10
+9.75
+15.76
+13.81
-1.21
+1.50
+1.22
+1.47
+1.26
+1.54
+3.34
+6.06
Voro-NDAIL
77.57
66.54
72.50
62.28
81.24
71.45
81.74
72.34
80.22
71.13
79.08
69.95
90.04
83,84
+5.95
+5.40
+2.63
+1.91
+3.05
+1.10
+7.31
+5.00
+5.86
43.76
+4.69
+2.58
+0.66
+0.66
TinylmageNel (5 phases)
TinylmageNee (10 phases)
TinylmageNel (20 phases)
Aug
Avg.
Accuracy
Accuraoy
Accuraac
8
3
8
8
Number of classes
Number
classes
Number of classes
EWC
LWF
LWF-MC
MUC UCIR
PASS
iCaRL-ONN
iCaRL-NOM
EEIL
Mero
Figure G.1: Top-1 classification accuracy on Tinylmagenet during 5/10/20 phases of CIL.
25

---

Published as a conference paper at ICLR 2023
CIFAR-100 (5 phases)
TinylmageNet (5 phases)
ImageNet-Subset (10 phases)
Geomattii Varianoe
Pearson's -0.378 0.702
Entropy Geom vaariance
0.042 0.617
Geomatric variance
Pasmonn's R:
Entropy based
Entropy
5 5
24
8.00 a/s
000
AAccuracy
AAccuracy
&Accuracy
Figure G.2: Entropy-based geometric variance (HV) in class level function of Aaccuracy.
phase
prototypical boundary
Dj (larss ) set of classes C. Cr
probing-inducee boundary
prootypiccl centers Crysk Cry k
probing- induced centers Cr Or
Figure G.3: Illustration of iooo (left) and iVoro-D (right). iVoro treats all prototypes indifferently regardless of the phase, whereas iVoro-D - refines the within-phase boundaries through divide-and- conquer algorithm.
= teddy
a/a (r)
teddy(1)
(1)
Augmentation Integration:
teddy(2)
(2)
argmin, Ea Ea fa,a'
teddy(3)
(3)
teddy(4)
(
Uncertainty Quantification via Geometric Variance
Figure G.4: Illustration of augmentation integration using sample image from the "teddy" class. Each of the rotated images can possibly be assigned to each of the expanded labels (i.e. teddy()) (eddy(2), teddy(3), and (t).y(4)). The final score of "teddy" will be the aggregation of all the 16 predictions. The variance of the 16 predictions reflects the confidence of this prediction.
26

---

Published as a conference paper at ICLR 2023
H DETAILED ANALYSIS ON PARAMETERIZED FEATURE NORMALIZATION
In this section we give a detailed comparison between i'Voro (no normalization) and i'Vrr-N (param- eterized feature normalization). On both CIFAR-100 and TinyImageNer with different phases, we show the distribution of accuracy (across all phases) for iVoro, iVoro-- with only L2 normalization, and iVoro-N with both Lg normalization and Tukey's s ladder of powers transformation (A varying from 0.3 to 0.9). As shown in Fig. H..I] feature normalization benefits both datasets, but the efficacy more prominent in more complex dataset e.g. TinylmageNer. Overall, iVoro-N improves the accuracy in the last phase by up to 1.65% on CIFAR-100, 2.21 % on TinylmageNet, and 2.40 on ImageNet-Subbet, respectively, compared to Vooo..
CIFAR- 1100 (5 phases)
TinylmageNet (5 phases)
No
(V2
(Mrr A+
Moro-M
Moro-NL2, A-C.8)
Moro-NLL2, 8-0.8
(oroN0) A-C.77 7)
Moro-kL12, I-0.7)
MoroNi2,, A-C 6)
Moro-MLL2, A=0.))
[MroHi0. A+0.5)
Moro-M]1, 5:
(Mrr A=0.4)
Moro-MLL2, 4:0
(MroNi12, A-0.3 5)
Moro-MLL2, k-0.3)
Voro
Noro
70
52.5
883
Accuracy (All Phases)
Accuracy (All Phases)
CIFAR-100 (10 phases)
TinylmageNet (10 phases)
Noro NIL2)
IVor ML22
(VbroNiL.. A-6.4 8)
Noro-MLL2 X-0.8)
NVbr Am0.8)
MLL2, A-0.8]
(VoroN0... AMO./))
-ero-MiLz, 1e0.7)
Maro-M 6:
(VbroNi),2, AHG.8) 8)
Noro-N[L2 A-0.5
(Nrrs40).2 J-C 4)-
Moro-MLL2, 1-044
(VoroN() AHO.3)
Moro-MLL 1=0.3
Noto
Maro
55
70
37.
40.0
42.5
45.0
50.0
52.5
150
Accuracy (All Phases)
Accuracy (All Phases)
CIFAR-100 (20 phases)
TinylmageNet (20 phases)
NVrro NIL2)
MMro NIL2, (m0.9)-
Noro MJL2, A-0.9)
|VorsNL22 A+0.8)-
Mero-ML2,, 2:0.8)
(Mro-/022, (-0.7)-
Moro M,2, An0.7)
(Vbr>N(L2 1=0.6)
Moro-MLL2, x=0.6)
[VroNNL2,, (=0.5)
Moro MLL2, Am0.5)
NVrr 4R0
Mero-M]L 2=0.4]
(Vbro NIL2 J=0.3)-
Moro -ML2, An0.3
Moro
Noro
37.5
<0.0
43.5
Accuracy (All Phases)
Accuracy (All Phases)
Figure H.I: Effect of parameterized feature transformation on CIFAR-100 and TinyImageNet.
27

---

Published as a conference paper at ICLR 2023
I UNCERTAINTY ANALYSIS
In this section, we demonstrate that the Entropy-based geometric variance (HV) is a good indicator of predictive uncertainty induced by SSL-based label augmentation, and also exhibits high correlation with predictive accuracy. Fig. [.I. Fig. 1.2 and Fig. I.3 present the distribution of HV of each class during each phase (0 to in 5- 5-phase CIFAR- 10AR--00 data. The color of the box reflects the accuracy of iVooo for the specific class. It can be observed in these figures that, the class with higher predictive accuracy will typically exhibit lower geometric uncertainty.
CIFAR-100 (5 phases): phase
Entropy based Geometric Variance
2.0-
1.5
0.5
13
1a
16-
17
18-
20
26
27
30-
31
33-
37-
40
61
44
85
46
67-
48
18
50
Class
19-4
CIFAR-100 (5 phases): phase 1
Entropy based Geometric Variance
2.5
12
I7
18
53
56-
96
37
39
40
46
60
50-
51
55
26
57
58
33
00
Class
Figure I.1: The distributions of Entropy-based geometric variance for each class in each phase on
5-phase CIFAR-100 (phase 0 to phase 1).
28

---

Published as a conference paper at ICLR 2023
CIFAR-100 (5 phases): phase 2
10-4
Entropy based Geometric Variance
10
Class
CIFAR-100 (5 phases): phase 3
Entropy based Geometric Variance
2.0
1.5
Class
Figure L.2; The distributions of Entropy- geometric variance for each class in each phase on 5-phase CIFAR-100 (phase 2 to phase 3).
29

---

Published as a conference paper at ICLR 2023
CIFAR-100 phases): phase
2.00
Entropy based Geometric Variance
1.50
1.25
1.00
0.75
0.50
0.25
Class
10-4
CIFAR-100 phases): phase
2.00
Entropy based Geometric Variance
1.50
1.25
0.75
0.50
0.00
Class
Figure L.3: The distributions of Entropy-buac based geometric variance for each class in each phase on 5-phase CIFAR-100 (phase 4 to phase 5).
30

---

Published as a conference paper at ICLR 2023
J ANALYSIS ON THE FEATURE EXTRACTOR
In order to examine the effect of the feature extractor on the final result, we gradually decrease the number of classes in the first phase from 50 to 40, 30, 20, and 10 and still include 5 classes in each subsequent phase. When compared with PASS, the best version of ivoro still has 17.75%, 13.59%, 10.89% and 1.60% improvements with 40, 30, 20, and 10 initial classes, respectively.
CIFAR-100 (12 phases)
PASS
iVoro
IVora
IVoro-AC
Noro-N
90-
iVoro-Al
Voro-D
IVoro-NAC
Moro-ND
iVoro NAI
70-
Avg.
Accuracy
iVoro-NDAi
70
80-
A/g
12-
50-
Number of classes
CIFAR-100 (14 phases)
PASS
iVoro
Vooo
iVoro-AC
Moro-N
IVoro-Al
Noro-D
iVaro NAC
Moro-ND
iVoro NAI
Accuracy
IVoro-NDAl
Avg.
80-
Number of classes
Figure J.1: Top-1 classification accuracy on CIFAR-100 during 12/14 phases of CIL,, which the number of classes in the first phase are decreased to 40/30, respectively.
31

---

Published as a conference paper at ICLR 2023
CIFAR-100 (16 phases)
100
PASS
iVoro
Nora
iVoro-AC
Moro-N
(Voro-AI
IVoro-NAC
Vors-ND
-NAI
Accuracy
Voro-NDAI
70-
Avg.
80
@
Number of classes
CIFAR-100 (18 phases)
100
IVoro
Nord
iVoro-AC
Noro-N
90
IVoro-Al
Noró-ND
Norr-D
80-
IVoro-NAC
NAI
Accuracy
70-
IVoro-NDAI
Avg
Ø Φ + + &
Number of classes
Figure J.2: Top-1 classification accuracy on CIFAR-100 during 16/18 phases of CIL, in which the number classes the first phase are decreased to 20/10, respectively.
32

---

Published as a conference paper at ICLR 2023
K ANALYSIS ON THE MULTILAYER VORONOI DIAGRAMS
In this section, we extract the feature from the 3rd block and rebuild iVoro, to be Vooo33, etc. The results for iVoro3, iVoro3-N, iVoro3-NAC, and iVoro3-NAI are shown in Fig. F2 Fig. E3 and Fig. F. As expected, there is always substantial decrease in accuracy if only the features from the 3dd block are used. For example, i'oro drops from 38.27% 18.71%, from 56.05% to 43.22%, and from 55.40% to 35.10% on TinyImage CIFAR-100, and ImageNet-Subsse However, when augmentation integration used, the accuracy of iVoro3-NAT becomes 42.03% (TinylmageNet), 63.13% (CIFAR-100) and 64.32% (ImageNet-Subset), even much higher than iVoro and PASS. Moreover, when integrating the features from & and ((3) using CCVD, iVoro-NDAIL achieves the best performance across all variants of iooro, 72.34% on TinylmageNet and 83.84% on ImageNet-Subset.
respectively.
ImageNet-Subset (10 phases)
80
PASS
IVoro
iVoro-N
Accuracy
IVoro-NAC
70
iVoro-NAl
|Voro3
iVoro3-N
IVor3-NAAC
|Voro3-NAI
50
40
Avg.
Number of classes
Figure K.1: Comparison between iVoros constructed from the final block (iVoro) and the 3rd block (iVoro3) w.r.t. the top-1 classification accuracy on ImageNet-Subbet during 10 phases of CIL.
33

---

Published as a conference paper at ICLR 2023
CIFAR-100 (5 phases)
90
PASS
IVrro
80
iVoro-N
IVoro-NAC
Accuracy
iVoro-NAI
70
IVoro3
IVoro3-N
IVoro3-NAC
[Voro3-NA]
80
Number of classes
CIFAR-100 (10 phases)
PASS
iVoro
80
iVoro-N
IVoro-NAC
Accuracy
iVoro-NAI
70
iVoro3
IVoro3-N
80
IVoro3-NAC
IVoro3-NAI
Avg.
Number of classes
CIFAR-100 (20 phases)
90
PASS
80
iVoro
IVoro-N
iVoro-NAC
Accuracy
70
IVoro-NAI
|Voro3
80
IVoro3-N
IVoro3-NAC
50
iVoro3-NAI
40
8 600 e 2000
Number of classes
Figure K.2: Comparison between ioVoos constructed from the final block (iVoro) and the 3rd block (iVoro3) w.r.t. the top-1 classification accuracy on CIFAR100 100 during 5/1 5/10/20 phases of CIL.
34

---

Published as a conference paper at ICLR 2023
TinylmageNet (5 phases)
PASS
70
IVrro
iVoro-N
Accuracy
IVoro-NAC
iVoro-NAI
80
IVoro3
IVoro3-N
IVoro3-NAC
[Voro3-NA]
20
100
120
200
Number of classes
TinylmageNet (10 phases)
80
PASS
iVoro
iVoro-N
80
Accuracy
IVoro-NAC
iVoro-NAI
iVoro3
IVoro3-N
40
IVoro3-NAC
IVoro3-NAI
30
Ava..
n10
120
130
Number of classes
TinylmageNet (20 phases)
PASS
70
iVoro
IVoro-N
Accuracy
iVoro-NAC
IVoro-NAI
50
|Voro3
IVoro3-N
IVor33-NAC
iVoro3-NAI
20
Number of classes
Figure K.3: Comparison between ioooos constructed from the final block (iVoro) and the 3rd block (iVoro3) w.r.t. the top-1 classification accuracy on Tinylmagenet during 5/10/20 phases of CIL.
35

---

Published as a conference paper at ICLR 2023
L COMPLETE LIST OF ALGORITHMS
Here we provide the algorithm for Voronoi Diagram reduction (Ma et al. 2022a in Alg. the i'oro algorithm in Alg. 2 the iVoro-D algorithm in Alg.
Algorithm 1: Voronoi Diagram-based Logistic Regression.
Data: Local data D, in phase
Result: W
Initialize W < :
for epoch 1, #epoch do
+ dr,k < - 1 7, 12, Vk 1,.., Kr
Apply Thcorem 2.1 forward propagation backward propagation
Compute loss C(W,, b-);
Update W
end
return W
Algorithm 2: iVoro Algorithm.
Data: Training datasets until phase t:
D. = {(Dr.i Mr,i) (r. 2r. E Dr E C,, E (1.... U. query example a Result: prediction !
for E {l....,t) t do
for k E {1, [L,... K2 do
Cr N,, Lieg Lie Nr. ((x+,));
prototypical centers
Vr +-
end end +-
(6x: +-
(hx o gw.n 0 f))
VoVor-,, optional
+
C s.t. d(z, Cr' (x) = min,, d(z, c7,k)
10 return (cr,))
36

---

Published as a conference paper at ICLR 2023
Algorithm 3: iVoro-D Algorithm. The time complexity for the establishment VD is
K,) the time complexity for querying the VD is O(Er
Data: Training datasets until phase t: D. i Vr,.) )= Fr E Dr E C,, fi...., t). query example
Result: Prediction y
for E (1,...,() do
W,. b, <- Algorithm D+)
C. f 2 Wr,k:
probing- induccd centers prototypical centers
Cr,k + Algorithm D+)
for Cy C2 E
Cr.hi -Gr
[e+,i -c.
1 11 11,,
9
within clique boundaries
end
end
10 for ,r E Cr. Cra,k' E Cr2 do
Crgil 1e Crg
11
||o, 110g 2 x C
12
at
cross-clique boundaries
end
14
for I' E {I') do
15
Delete candidate and its boundaries w.r.. sign(v"'s x 9) if only candidate Crr,k' v remains then
17
ÿ + Cr'
end
19 end
20
return '
37

---

Published as a conference paper at ICLR 2023
M FORGETTING ANALYSIS
Following PASS (Zhu et al. 2021). we also quantitatively measure the degree of the catastrophic forgetting. Specifically, at phase the accuracy drop from the maximum accuracy current accuracy for datasets from each phase E {1,.... is denoted as the average forgetting, as shown in Fig. M.] and Fig. M.2 for CIFAR- 1 and TinylmageNet, respectively. For PASS, the average forgetting keeps growing during phases. However, for VVoro/VVoro-Ni N/Voro-NAC/iooro forgetting significantly overcome. For example, on CIFAR-I00 (5 phase), forgetting in the last phase is decreased from 20.28 to 8. 8.17/8.95/5
the catastrophic
For local dataset D, {(xr, vr,))) - 1,, E ID; Mr, E C, and set of fixed prototypes (Cr,x)re(1.....)) (....).................. the prediction 9 = arg min
for iVoro/i Voro-N/i
C= cr,1112 is less likely to change compared to continuously updated model, in PASS, implying that, aligning with intuition (Sec. []]. the VD structure can naturally and successfully combat catastrophic forgetting.
(1.... (-). llØ(z)
CIFAR-100 (5 phases)
PAS3
0.00
13.52
15.03
17.33
19.23
20.20
Method
Nom
0.00
4.95
5.87
7.40
7.50
8.17
Mero-N
0.00
4.94
5.70
7.55
7.91
8.96
Noro NAC
0.00
3.02
3.67
5.14
5.38
5.02
Nors NA
0.00
3.02
3.67
5,38
5.92
Phase
CIFAR-100 (10 phases)
PASS
9.76
19.45
Method
3:38
5.88
7.11
7.34
7.58
Norro-N
0.00
334
4.17
5.62
7.24
7.37
7.72
7.79
8.22
8.73
Nom-NAC
0.00
1.84
3.62
5.03
5.28
5.56
5.71
Norro-AA
0.00
1.84
2.86
3.03
3.62
4.92
5.03
5.19
5.28
5.58
5.71
Phase
CIFAR-100 (20 phases)
PA:9
0.00
4.87
7:09
8.53
10.91
12:39
13.91
15:90
17.28
17.50
19.34
21:30
21.89
22
23.39
24.94
22.89
24.78
26.41
25:47
26.15
Method
3.73
5:40
5.62
2.15
4.35
4.88
7.44
7.72
6.08
5.11
2
7
10
12
13
14
15
17
18
18
20
Phase
Figure M.I: Results of average forgetting on CIFAR-100.
Table M.1: Average Forgetting ({): Comparison between i'Vro with -tate-of-the-art CIL methods
CIFARI00
TinyImageNet
Methods
phases
10 phases
20 phases
5 phases
10 phases
20 phases
iCaRLeNN Rebuffi et al.
2017
42.13
45.69
43.54
36.89
36.7
45.12
iCaRLeNN (Rebulli ei al.
2017
24.90
28.32
35.53
27.15
28.89
37.40
EEIL (Castro et al 2018
23.36
26.65
32.40
25.56
25.91
35.04
UCIR Hou et al, 2019
21.00
25.12
28.65
20.61
22.25
33.74
LwF-MC LL
Hoiem.
2017
44.23
50.47
$5.46
$4.26
54.37
63.54
X MUC [L.u ct al. 2020h
40.28
47.56
52.65
51.46
50.21
58.00
X
PASS
Zhu el al. 2021
20.28
29.93
26.15
8.91
23.11
30.55
x iVoro (ours)
8.17
7.97
6.95
5.68
5.55
5.34
38

---

Published as a conference paper at ICLR 2023
TinylmageNet (5 phases)
PA33
0.00
8.70
6.11
66
7.84
8.81
Method
Nom
0.00
3.44
4.02
4.48
5.35
6.68
Noro-N N
0.00
3.14
3.71
4.35
5.22
5.72
Nom-NAC
0.00
2.57
3.14
3.86
4.34
4.72
Nors NV
0.00
2.57
3.66
4,34
4.72
Phase TinylmageNet (10 phases)
Method
Marc-N
4.88
0.00
1.51
2.72
2.85
3.48
4.10
4.41
Norro-AAl
0.00
1.51
2.72
2.88
4.10
4.41
Phase
TinylmageNet (20 phases)
PA.99
0.00
5.23
5:87
7.20
8:23
11.10
13.01
15.21
15.43
14.59
16.31 18.
18.81
20,
22.88
25 95
26.00
26, 02
27 56
02-22
30.55
Method
Moro
0.00
1.02
1.17
2:39
3.36
3.06
3:24
3:41
3.59
3.60
3.80
4.02
4.33
4.30
4.68
6.00
1.14
5.19
5:20
6.34
Noroo
1.30
.848
2.16
2.31
3.02
3.11
3.67
0.02
4.05
6.65
4.05
1.36
5.00
Noro-NAC
3.49
3.40
2
7
10 Phase
12
13
14
16
17
16
20
Figure M.2: Results of average forgetting on TinylmageNe
39

---

Published as a conference paper at ICLR 2023
N CLASS-LEVEL ANALYSIS
In order to examine the reason for our strong two-digit improvement (25. 26%, 37.09%, and 33.21% for CIFAR-II --000, TinylmageNet, and ImageNet-Subset, respectively). in this section (Fig N.1 Fig. N.2 Fig. N.3 Fig. N4 Fig. N.5 Fig. N.6 Fig. N.7 and Fig. N.8), we illustrate the class-level accuracies along the 5 phases for all the 200 classes ImaylmageNet. Generally, iVoro competitive PASS, and Voro-NAC/NAL performance significantly better than the former two. This is more evident for novel classes (i.e. class 199), showing that augmentation consensus/ineerration particularly useful for unseen classes
(0) umbrella
111 potter's wheel
(a) highern
(3) tara ntula
RASS
6.8
10
Mora 0.8
Moro
Moro
Nor
Mora HAC
Noro-NAC
Moro MMC
Ntro MAC
Wore
Mare-Nil
Noro
Noro 0.0
0.5
0.4
0.2
0.5
Couliflower
(5) flagpole
(6) volleyball
drumstick
ASSS
RASS Were
PASS
PASS
Nera NAC
Nits
0.0
Nom
Moro MAC
Noro MAC
Nom
Nora NA
Voru NAI
Noro NAI
Néro MA
0.7
0.8
0.6-
0.4
0.5
(8) mantis
syringe
(10) pizza
(11) beach wagon
NASS
o.n
HA5S
More
0.5
0.6
0.7
0.0
0.4
0.4
9.3
0.4
(12) slug
(1.3) banana
[14] Labrador retriever
(15) centipede
PASS
B655
Woro
3.
NAC
Nore
MAC 0.8
3.5
0.5
0.34
0.4
0.3
Figure N.1: Class-level comparison of PASS, iVoro, VVro--AC, and iVoro-NAI on the TinyImageNet dataset (5 phases). The axis and y axis denote the phase index and the average accuracy in this class, respectively. arbitrarily selected sample image class is also shown for every class.
40

---

Published as a conference paper at ICLR 2023
Spoan
(18) baskotball
(19)
PASS
7AS5
PASS
PA55
Were NAC
Woro-AC
Moro
More
Noro NAC
A.B
Noro MAC
Hore
WWre
Vorc-NAl RAI 3.5
Noro NA
D.7-
Néro MA
0.6
0.4
(20) dining table
(2.1) trilobite
cardigan
(23) brown bear
1.00
Vers-Na
0.6-
0.5
0.6
0.5
0.5
(24) kimone
(25) dragonfly
(26) dumbbell
(27) geldfish
AASS
0.8
MSSS
R65S Moro
RASS
1.000
Mers
AWrr-BAC
More
0.7
Nor
Nov
2.50
8-
2.6-
Notr--Mil
50
6.5
.6
0.1
0.4
77
(28) freight car
(29) pretze!
(30) chain
(31) abacus
PASS
RASS
PASS
PASS
Mer
Woro
Moro
Nero
Nera NAc
RAC
Noio MaC
Nero MAC
Nort NM
Noro-BA RAI
Moro NM
Ntr MAJ
0.7
(32) American alligator
MSS
0.7
RASS
More
0.5-
Moro
NAC0.8
0.7
6.2
0.4
0.25
0.1
0.4
(36) convertible
(37) burkett
(38) gasmask
(39) Eurapean fire salamander
1.00
9.10
Moro 0.4
ficre NAI
577
0.3
0.4
0.70
23
(40) scwing machine
(43) sea slug
(42) remate control
(43) plunger
R455
MSs
PASS Miro
Nor NAc 0.8
Nor
Woro 3.
Moro
Moro MAC
Nor
Nor
Noro NAI 07
Woro Nai
0.5
Moro HAI
0.7
Mor
055
8.5
0.4
0.0
Figure N.2: Fig. continued.
41

---

Published as a conference paper at ICLR 2023
(44) beaker
(45) dugong
(46) water Tower
(47) oranguuaa
PASS
MASS
1.00
0.6
RASS
Mere
0.8
Mora
Morr MAC
Noro MMC
Ntr M6C
Noro-t 9-7
0.25
D.6
0.6
0.5
0.6
0.73
0.4
0.70
(48) laleside
(49) maypole
(50)
(51) Egyptian
Nire
ASSS
0.8
RASS
Were
Nom
P65S
0.0-
Nem
PASS
Nera NAC
Nora NA
WAr Voro NAI
Noro NA
Moro MAC
0.7
Noro MAC.
Néro MAi
D.6
0.5
0.2
0.6
(53) frying_pan
(54) dwelling
(55) spiny
NSSS
8855
NMC
0.6
Moro-Ac"
NAC
0.0
More
Miro MMC
0.6
0.7
0.5-
0.6-
0.5
0.5
(56) pap. bottle
[57) go kart
(58) guacamole
(59) ladybug
PASS
1.0
B655
1.00
Woro
0.55
0.6
No.
0.1
0.2
5.6
9.0
0.5
(60)
(611 a.om
(62) racking chair
(631 black widow
PASS
PASS
PASS
PASS
1.00
Mero
Nore
0.6
Fibro
Wirr NAC
Vero-NAC
More-MAC
Nibro MAC
WWrr NAI
Voro-NAj
Moro-AU
DJ7
Nnbro 140
0.7
0.70
0.4
talled
shephard
RA55
PASS
PABS
PASS
Mero
Noto
Vore
8.9
Wers
Niro Mac
Nor Na
Moro NAC
Noro NAI
0.6
Vore-nal
Voru-Nac Nac
WWru-NA NAI
WWrl-AG
0.6
0.7
0.5
0.4
8.5
0.5
0.4
0.4
0.3
52
(68) sports
jug
(70)
RASS
PASS
PASS
PASS Mor
WWre No.
Mers-AAC
Mero
Nero
5.5
noro MAC
Mers Na.
Nomotu
0.6
0.55
0.50
67
5.6
Figure N.3: Fig. N. continued.
42

---

Published as a conference paper at ICLR 2023
(72) ox
(75) monarch
(74) desk
[75] apron
0.50
PASS
1.00
0.8
RASS
Moro MMC
0:43
Mero
Ntro
Mora
0.10
NoretN
Niro
0.35
0.5
0.4
0.25
0.3
0.75
[76] heater
vestment
(78) alp
(79) chimpanzee
PASS
RASS
R65S
RASS litr
Nire
Were
Nom
Wites mM1
Vere-MAC
Nore-tCC Noro NA
Mbro-Mic00
Nera NAI
0.8
0.7
Woro
0.5
0.6
Néro MA
0.2
0.6
0.4
potpie
pomegranate
NSSS
8855
More
NACO.
Moro-Mc 9.5
NACA
Moro MMC
Mor--All
NoroRM
Miro MAD 50
0.85
0.5
0.6
(84) born
(85) ice Inlly
(86) snall
(87) projectile
PASS
0.50
B655
Woro
Nore
NAC
2.0
0.35
0.0
3030
(88) hourglass
(80) pancho
(90) computer keyboard
(91] military, uniform
AASS
PASS
PASS
PAss
0.8
MWro
Noro
Notoo
Noro
NWro NoC
Noro MACO
New
Noro
+
Nom--W
Nom
0.5
0.4
5.50
0.4
0.3
(92) organ
(93) gazelle
(94) viaduct
(95) walking stick
IASS
MS5
ISSS Nom
PASS
Nite
Wen
8.40
Ntm
NereAACD
Aite-WCCO
Vere-ta)
More-U
fibr--M AW
0.80
0.7
0.0
(96)
RASS
5ASS
RSSS
PASS
Mero
Moro
NiC
Noro
Miro
AWrs-Mii
0.7
NorotM
Miro-M4
0.5
0.5
0.5
0.4
0.3
0.4
Figure N.4: Fig. N.1 continued.
43

---

Published as a conference paper at ICLR 2023
(100) punching bag
(101) sulphur butterfly
(102) bow tie
(103) chest
MS5
RASS
Mere
Mora
Wore-AAlp 9
More No.
55
0. 0.50
0.2
0.75
0.2
0.0
(104) steel arch bridge
(105) Christmas stocking
(106) beacon
(107)
MS5
RASS Were
M655
0.0
PASS fem
Mire
0.
Nom
Wires
Ners 1M
0.50
Vere-AC
Vere-All
Nore-N
fitro MMC
Fibro Mp
10
0.45
6.5
0.40
0.75
(1.08]
tabby
lamoshade
ruoby
0.56
NASS
8855
0.9
More
NMC
0.8
Mor--All
6.8
Moro MAD 85
Moro
0.7
0.6
0.70
0.5
0.3
(1122 darm
(113) cannen
(114) barrel
(115) bison
0.8
Woro
More
NACo.6
0.6
0.5
0.5
0.4
0.3
0.2
0.0
(116) orange
(117) sunglasses
(118) bathtub
(119) king penguin
Asss
0.8
Pess
Rss
5.6
PASS
Mor
More 9.
Noro
Moc
0.0
Moro NAC
Moro
Moro
Moro
0.5
0.4
0.3
8.4
0.2
(120) American lobster
(121) jinrikdsha
(122) gondola
(123) lesser panda
AASS
Nite
3.05
RA5S
Nen
0.75
ISSS
Nhm
MSSS Nom
Nom MVC
fom-wccl
Nora N4
Noro N4
0.65
Miro Nu
More NM
0.50
3.45
3.00
0.4
9.35
0.50
8.3
(124)
(125) suspension bridge
sombrere
(127)
0.8
MS5
0.9
B855
0.0
Mor
MAC
0.6
Nomotu 0.7
Miro McC0
Miro
0.5
0.4
0.5
54
Figure N.5: Fig. N. continued.
44

---

Published as a conference paper at ICLR 2023
mashed
(1350)
(131)
PASS
PASS
BA55
Mere -
Wer--M
Wore
temoiic ferotU
0.5
Mero
Mero-MCC MeroAW
Ver--AA
0.6
0.4
0.6
0.-
0.3
0.2
0.5
0.0
(1321 bullet train
33) acadertic gown
1134) jellyfish
(135) triumphal arch
0.59
1.00
9.50
3.50
0.55
3.45
V6G, 15
6.45 A5
X4 10
0.70
0.85
2.75
0.80
0.30
0.70
(136) brain coral
(137) corric book
(139) backpack
(139) magnetic compass
1a
1.0
0.9
0.0
0.6
0.7
0.4
corstrictor
(141) albatross
0.9
MSS
BA5S
Mere
Mer
Mer MiC
Mer--AC
Nemoicc 0.6
Mem MMC
(Arr--i 0.7
Mer--WA
Neroau
More-M
0.6
0.5
0.5
0.4
D.6-
0.3
(144) bee
(145) cockreach
(146) butcher shop
(347) parking meter
0.9
0.8
Maru
9.7
0.7
0.5
0.8
6.4
0.4
03
(1.48) lown, mower
(144)) black stork
[150] Persian caf
(151) Arabian camel
PASS
0.
PASS
1.0
PASS
1.3
PASS
Mers
Woro
Ivers
NAC
Ivor:
0.30
0.4
0.5
0.25
0.20
(152) fly
(153) torch
(154) coral reef
(155) pill bottle
Alrs
RASS
Ware
Mom
0:35
Nor
PASS
0.8
MereAMn Moro Hil
Nor
WoreAAl
Noro
0.6
0.85
3.3
Figure N.6: Fig. N.1 continued.
45

---

Published as a conference paper at ICLR 2023
(1566 spider web
(15.7) thatch
(158) cash machine
(159) birdhouse
PASS
1.0
MSSS
Moro
0.45
R655
Moro
Noo
8655
Mor NACO3
Moro
Ntr M6G
0.55
Nrr
077
0.5
0.7
D.6
0.5
0:33
(160) Chihuahun
(161) lion
(162) bell
(163) sock
Nora NAC
Mero
PASS
0.7
WA Yoro NA/
NASS
Moro
0.60
Noro MAC
MASS
More
0.
Noro
Néro MAC
P825
Nora NA
Noro NM
Néro Mei
0.6
0.3
0.40
0.5
0.5
(164) cougar
(1033 pay-phone
(1.6) seashore
(167) stopwatch
0.75
MSS
0.8
0.70
Vers Naf
0.6
0.60
0.4
0.4
0.5
50
0.2
0.2
(168) scorcboard
(369) turnstlle
(170) racl
(171) miniskitt
0.9
PASS
Woro
0.7
0.6
Nore
B655
Mero-NAl
0.7
More-W
0.5
0.7
0.3
0.-
6.1
0.5
(172) refrigerator
(173) fountain
(174) guinea
(175) Yarkshire terrier
PASS
MSSS
PASS Nero
MASS
MWro
Nor
Wore
0.6
NWro MM.
Noro
Noto-MM
WoreHAC
NWmm
RM
Wer-NAl
0.7
0.6
6.5
0.4
6.6
9.4
0.1
6.3
0.4
2
(176) swimming trunks
(177) nack brace
(178) sea cucumber
(179) tractor
PASS
MSS
PASS
5.9
PAss
Were
More
Nore
WWrr NA-
Moro NAC
Niro MAC
WWr NAI
-
Were Na)
Miro HAI
Noro
D.7
375
0.15
350
0.5
scorpion
lifeboat
RASS
RASS
PASS
PASS
0.25
No.
Mors-ACC
Mero
0.5
Nero
Noro M..
Moro
30
WWr
Horo-RA
Noro
Noro NAP 25
0.65
0.6
0.20
0.5
0.3
0.15
0.7
0.10
0.1
Figure N.7: Fig. continued.
46

---

Published as a conference paper at ICLR 2023
pelice van
(185) teapet
(186) fur cost
(187) pole
0.55
R65S
0.45
0.50
MMC 0.5
0.3
0.35
0.3
0.10
0.2
0.0
(188) ice, cream
(189) obelisk
(190) hog
[191) bear bettle
PASS
MSSS
0.80
PASS
Ress
Mora MAC
Moro NAI
Mora
0.5
+
Woro
Woro
Noro-Nag
Moro Hil
Moro 30
Moro
Noro Noc 0.5 Moro
Nir
0.70
0.20
0.3
0.45
25
(192) confectionery
(193) grasehopper
(195) clif
0.75
RASS
TASS
RASS
BA55
Mero
Nere
0.5
Noro
Wer NA
NAC
Moro NAC
Noro MAc
WWr NA
Woru RA)
Moro NAI
Néro MAi
0.7
0.55
0.50
0.4
0.5
(196) standard poodle
[197)
(198) mushroom
(199) teddy
0.25
0.0
0.250
228
0.200
0.42
0.6
0.5
0.6
125
0.3
100
0.2
Figure N.8: Fig. N. continued.
47

---

Published as a conference paper at ICLR 2023
O SAMPLE-LEVEL ANALYSIS
In Sec. N we have analyzed the class-level improvement, and demonstrated that most the time iVVrr-NAa can surpass others by virtue of the postprocessing of the predictions from SSL-based label augmentation. In this section, we go into more detail at the sample level aiming at revealing why misclassification can be corrected once our method is applied.
To do so, we arbitrarily inspect two examples from the first class (i.e. "umbrella"); and show them in Fig. 0.1 and Fig. 0.2 respectively.
In Fig. O.I "umbrella" is within the top-3 labels predicted by PASS, which is, however, overwhelmed by "ice lolllllll.. From row 2 to row label set I/II/II/VV represent the expended labels for the original, and II/III/I for the 90/1 90/180/270-derre rotated, see Fig. G.4), and the distances from the query image (original or rotated) to the 100 classes (original expanded) are shown. Interest- ingly, the original image is predicted as the same label ("ice_lolly") by i'Voro. However, for the rotated images, the mis-predicted labels are various, "abacus", "bannister", "limousine", "vestment", "sewing machine", "plunger", and "dumbbell". But importantly. the distance to the ground-truth label "umbrella" is always relatively small. Hence, when integrated altogether, the ground tauntdttuthhh label "umbrella" achieves the nearest distance (see upper right figure).
Similarly, in Fig. 0.2 the sample is misclassified as "beach_wagon by PASS, and still misclassi- fied by iooo as "volleyball". However, when augmentation integration (iVoro-AI) is applied, the accumulated distances clearly direct to the correct label "umbrella".
In conclusion, these sample-level analyses show that Voro-AC/Al elevates the performance by substantially enhancing the robustness of the (augmented) prediction - the rotated image set and the expanded label collectively contribute the strong two-digit improvement.
48

---

Published as a conference paper at ICLR 2023
PASS, predicted label: Ice. Jollyy
[Voro-Al, predicted label: umbre
0.630
8.625
WM Mrmm
label set predicted Bibol: Ice Jolly
100
label Sat prodicted Lbal: Ice lce)aali Jally
205
1.c
www.
3.5
1.1
10:
label Set III, predicted Tabel: ice lolly
100
label Set N. predicted Tabel: ice Jolly
1.5
1.0
1.5
label set predicted Tabel abacus
label set il, prodicted label: abacus
1.5 M
3.7
1.4
3.00
Set predicted label: bannisser
label Set predicted label: ab'acus
An (wo /
2.0
M. owehewhww MMMM
Betel predicted label: limousine
100
label 'et predicted Gabel: vestment
Ww MMMMM
3.0
iBbel fli, prodi Fed labal: sewing machine
25 300
label 20
WV, prodicted fhat: to vasimant
MMMMM
1.00
3.50
label set prodicted label: plinger
label predicted libell: dumbb
3.80
WW Mornn
Woll M
3.7
3.5
3.5
1.a
3.4
labef'st predicted Sbel: plünger
100
label Bet prodicted (hee duffbelll
M Mww MM
M M
1.-
Figure O.I: Sample-level analysis of one example from the "umbrella" class. The y axis denotes the logit for 100 classes for PASS, and denotes the distance to 100 prototypes for iVoro. The green line indicates the ground-trufe tround-truth label while the red line denotes the predicted label.
49

---

Published as a conference paper at ICLR 2023
PAS5, predicted label: beach, wagon
IVoro-Al, predicted label: umbrella
4.14
3.05
3.00
3.63
label Set prodicted label: volleyball
100
label Sat proflcted labol: barbörshop.
1.6
3.7
1.0
label set IlI, predicted label: barbershap 160
labe set IV. predicted label: labe:: kimona
1.90
1 www MWw
3.5
label Set 1. predicted label: umbiella
label Set II, prodicted lab el: flagpale
M-MMM MAM
M MALA
1.5
1.0
3.4
label'Ser Set predicted label: umborella
label Set IV, predicted Yabel: flappole
Mill MWn
3.5
3.5
labēf set predicted Fabel: abacus
100
label Set I, predicted Yabel: abacus
1.6
VA My Aww MMM
3.0
labol sot IlI, predicted labal: abacus
label Set IV. predicted Tabel: 20
W My M
Lynch M, Mww M
3.00
1.4
1.00
label set predicted cted label: chim panzee
babel
predicted Labradoo retriever
1.6
3.75
1.3
3.50
1.2
1.
label 28 predlcted lafet: chimpanzee 100
lahet set N. prodi etfed label Labraddf retriente
Man M M
3.20
Figure O.2: Sample-level analysis of one example from the "umbrella" class. The y axis denotes the logit for 100 classes for PASS, and denotes the distance to 100 prototypes for iVoro. The green line indicates the ground-trui -ounh label while the red line denotes the predicted label.
50


Reviews:
[{"review_id": "xPdHONz4nU", "reviewer": "Reviewer_K7j8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper constructs a Voronoi Diagram in an incremental manner to perform Data-free Class-Incremental Learning (CIL). The proposed method is shown to be a flexible, scalable and robust with theoretical insights and experiments, and it promotes the performance of CIL. ", "summary_of_the_review": "I am not an expert in this field so I may be wrong, but based on the papers of my expertise that I have read before, I think this is a good paper in general. ", "strength_and_weaknesses": "Strength:\n- The paper is clearly written and easy to understand. \n- The paper deals with class incremental learning in a new perspective. \n- The paper investigates fundamental problem in machine learning.\n\nWeakness:\n- I am not quite sure how the method will perform when the distribution of class labels is (highly) skewed. \n- The proposed method and theory will work when separation between classes in the feature space is sufficiently clear. The authors mention this as a challenge in the introduction, but I am not quite sure how they alleviated this problem.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "c1BQmrwv4Ci", "reviewer": "Reviewer_uHig", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies class incremental learning and proposes several approaches based on Voronoi Diagrams called iVoro. In particular, the paper contributes four approaches: iVoro, iVoro-D, iVoro-AC/AI, and iVoro-L. The introduced approaches are validated on three datasets (CIFAR-100, Tiny ImageNet and ImageNet-Subset) reporting large improvements over previous non exemplar class incremental methods.", "summary_of_the_review": "The introduced ideas are interesting and show good performance in the task of class incremental learning. The paper is well validated and contains significant number of ablations. Overall, the reviewer would lean towards acceptance, assuming that the authors could improve the presentation clarity with a focus on the introduction and methodology sections.", "strength_and_weaknesses": "Strengths:\nNice idea of using Voronoi Diagrams in class incremental learning\nRather extensive validation\n\nWeaknesses (details see section below):\nClarity of presentation could be improved\nMight be hard to reproduce the models from the current writing of the methodology section\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YaLlc5qiK7", "reviewer": "Reviewer_zQrg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method for class-incremental learning, i.e. learning a classification task while during training the model has no access to already used data and new samples can contain new classes. Dividing the classification network in feature extractor and classification head, the authors start with the substitution of the classification head by a 1-nearest-neighbour decision landscape (Voronoi diagram) and derive refined methods from there.  \nThe model is tested on the three established datasets CIFAR-100, TinyImageNet and ImageNet-Subset. \nSpecifically, the authors develop four components that enhance their baseline model: parametrised normalisation, a divide-and-conquer strategy for iterative construction of decision landscapes, augmentation consensus and integration and multilayer compatibility. \nThe authors test different combinations of these components to test their respective effects as well as conducting comparison to several benchmark models. The method outperforms all benchmark models in terms of common performance measures on the three datasets by far. ", "summary_of_the_review": "The work at hand is well structured and written. The presented methods are to the best of my knowledge novel in the context of CIL. While the experiments and results stress superiority to other methods, the paper does not discuss its findings sufficiently. An attempt to explain why the method outperforms benchmark models to such a great extent would strengthen the work. ", "strength_and_weaknesses": "**Positive:**\n- the paper is structured well and the ideas are presented in a intuitive way\n- Illustrations support the main text well \n- Extensive citation of the literature\n- Code will be made available \n- The method shows improvement compared to benchmark methods for CIL\n\n**Negative:**\n\n- (Major) The experimental results show strong two-digit improvement compared to the best baseline (cf. Tab. 1). I missed a section where the authors provide an intuition and reasoning why the method works so much better than everything that has been done before. There is neither a discussion nor a conclusion in “4 Discussion and Conclusion”, more a summary. \n\n- (Minor) It is easy to loose track of the different versions and reading flow suffers from looking up in the text what the different iVoro-x models actually are. \n\n**Questions:**\n- It seems the baseline method already shows major improvements. You write *“We suspect that this is because the features generated by the frozen feature extractor can be satisfactorily separable by linear bisectors (Fig. 2). As we can see, the features for other methods are all dramatically changing during the phases, but those for iVoro are all fixed, making incremental VD construction possible. “*  Can you provide a line of argumentation why this could be beyond the above sentence? \n- 2.1 - C_t once is the dataset at time step t and once is the set of classes at phase t. If those are different letters I could not distinguish them. The sentence, that C_i are pairwise disjoint irritates because this is not clear. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster