Persistence Homology Distillation for Semi-supervised Continual Learning
Authors: YanFan
Yu Wang
Pengfei Zhu
Dongyue Chen
Qinghua Hu
Venue: NeurIPS 2024 poster (2024)
Keywords: Continual learning; Knowledge Distillation; Semi-supervised learning; Topological data anaylsis; Persistence Homology;
Primary area: machine_vision

Abstract: Semi-supervised continual learning (SSCL) has attracted significant attention for addressing catastrophic forgetting in semi-supervised data. Knowledge distillation, which leverages data representation and pair-wise similarity, has shown significant potential in preserving information in SSCL. However, traditional distillation strategies often fail in unlabeled data with inaccurate or noisy information, limiting their efficiency in feature spaces undergoing substantial changes during continual learning. To address these limitations, we propose Persistence Homology Distillation (PsHD) to preserve intrinsic structural information that is insensitive to noise in semi-supervised continual learning. First, we capture the structural features using persistence homology by homological evolution across different scales in vision data, where the multi-scale characteristic established its stability under noise interference. Next, we propose a persistence homology distillation loss in SSCL and design an acceleration algorithm to reduce the computational cost of persistence homology in our module. Furthermore, we demonstrate the superior stability of PsHD compared to sample representation and pair-wise similarity distillation methods theoretically and experimentally. Finally, experimental results on three widely used datasets validate that the new PsHD outperforms state-of-the-art with 3.9% improvements on average, and also achieves 1.5%  improvements while reducing 60% memory buffer size, highlighting the potential of utilizing unlabeled data in SSCL. Our code is available: https://github.com/fanyan0411/PsHD.

Persistence Homology Distillation for Semi-supervised Continual Learning
Yan Fan, Yu Wang', Pengfei Zhu, Dongyue Chen, Qinghua Hu College of Intelligence and Computing. Tianjin University, China
Haihe Laboratory of Information Technology Application Innovation, China fyan_0411 1ean_041100 04t_00111ijuiu (4.11@tju..ede edu.. cn, wang yu@tju. edu. edu en, zhupengfe @tju. edu. dyuechen@tju. edu. cn, huniigggghuuttiju edu. cn
Abstract
Semi-supervised continual learning (SSCL) has attracted significant attention for addressing catastrophic forgetting in semi-supervised data. Knowledge distillation, which leverages data representation and pair- similarity, has shown significant potential in preserving information in SSCL. However, traditional distillation strate- gies often fail in unlabeled data with inaccurate or noisy information, limiting their efficiency in feature spaces undergoing substantial changes during continual learn- ing. To address these limitations, we propose Persistence Homology Distillation (PsHD) to preserve intrinsic structural information that is insensitive to noise in semi-supervised continual learning. First, we capture the structural features using persistence homology by homological evolution across different scales in vision data, where the multi-scale characteristic established its stability under noise inter- ference. Next, we propose a persistence homology distillation loss in SSCL and design an acceleration algorithm reduce the computational cost of persistence homology in our module. Furthermore, we demonstrate the superior stability of PsHD compared to sample representation and pair-wise similarity distillation meth- ods theoretically and experimentally. Finally, experimental results three widely used datasets validate that the new PsHD outperforms state-of-the -frthe-rrt with 3.9% improvements average, and also achieves 1.5% improvements while reducing 60% memory buffer size, highlighting the potential of utilizing unlabeled data SSCL. Our code is available: https://github.. https//github..oo//aaanaae
www.//aooo/a///4//m/ 1/PsHD.
Introduction
Continual learning involves developing practical approaches for incrementally training models, enabling them to learn new concepts while effectively retaining previously acquired knowledge [.] The problem of catastrophic forgetting in continual learning impacts the model's performance on previously learned tasks. Numerous efforts have been dedicated to addressing this issue continual learning ( However, most sate-of-the-art methods are trained under the assumption of fully labeled data, neglecting the vast amounts unlabeled data.
Semi-supervised continual learning introduces unlabeled data in CL by partially annotating the trained data each task and aims to alleviate catastrophic forgetting by exploiting these unlabeled data 14) Learning from unlabeled data to improve model generalizability across tasks is natural approach, but its performance significantly relies on the size of the memory buffer [12] [5]. Generative replay methods reduce the need for a memory buffer but incur prohibitive costs for higher resolution images and are limited relatively simple datasets [T 12]. Knowledge distillation methods are proposed to
preserve valuable information in the unlabeled data of the previous model [3] [4]
"ooresponning author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

34
32
J271
30
28
J144
34
32
J266
36
1263
18.08
28 2202
30
25
J.888
30
22
24
30
25
20
26
26
21
18
Avg Avg old
Avg
Avg old
Avg
Avg
Avg old (b) PH distillation (Ows)
Avg
(a) Logits distillton
(b) Feehure distillation
(b) Relation distillation
Figure The performance interference of extra unlabeled data distillation. The blue columns represent experiments only disillation knowledge only labeled data, and the red columns are results with extra unlabeled data distillation based on blue ones. The four approaches are (a) iCaRL, (b) PodNet, (c) R-DFCIL and (d) our persistence homology distillation method. Avg and Avg old mean the average incremental accuracy of all tasks and old tasks, respectively.
Logits or feature distillation approaches focus sample-wist representation 9 Relation distillation methods, as variants of representation distillation, are explored for vulnerable representation of unlabeled data to enhance more expressive knowledge preservation, including class-instance similarity [3]. and local structural similarity [14]. Although these distillation strategies have proven effective, the impact of noisy relations remains significant challenge. As shown Figure the succeeded biased or inaccurate information often interferes with generalization on known classes impact and even impact the entire tasks [16]. Therefore, argue that more powerful distillation that investigates noise insensitive information preservation in unlabeled data is needed for semi-supervised learning continual learning.
To address the aforementioned problem, we explore extracting multi-scale shape and structural information from unlabeled data that is insensitive to feature space bias and noise interference in semi-supervised continual learning. Inspired by the manifold assumption [17] and the homomorphism between manifolds and simplicial complexes (18). we explore summarizing complex vision data into simplicial complexes, and investigate the preservation of multi-scale structural properties despite representation changes. Persistence homology in topological data analysis, as an adaptation of homology to discrete metric spaces, reflects the homology-class evolution across different scales. Its characteristics of invariants homeomorphism and robustness to perturbations provide an adaptable view to explore the structure in unlabeled data.
To this end, we propose new model named Persistence Homology Distillation (PsHD) for semi- supervised continual learning to stabilize learned tasks by leveraging the power of topological features. Firstly, we construct local simplicial complexes in vision data based on the weighted k-hop neighborhood and extract multi-scale structural features using persistent homology. Furthermore, we devise model aeeametee--iddeeeddeee design an acceleration algorithm to mitigate the computation costs. We also demonstrate stability privilege of persistence homology distillation compared to pair-wise similarity distillation strategies 13
distillation loss based on these structural features and also
20 theoretically and experimentally. The primary contributions our work are as follows:
We utilize simplicial complexes to approximate vision data and explore stable topological feature representation of unlabeled data in semi-supervised learning.
We propose a novel persistence homology distillation strategy for SSCL that is insensitive to noise information interference based on the intrinsic topological features, and devise an accelerating algorithm to reduce computation costs.
We demonstrate that our method outperforms existing methods several benchmarks, and highlights the potential of utilizing unlabeled data to overcome catastrophic forgetting.
2 Related Work
Persistence Homology Topological data analysis (TDA) new approach seeking structural or
geometric information from discrete data. Persistent homology (PH) is one of the most popular tools

---

in TDA to capture topologiaal information about connectivity and holes in geometric data. Several impressive applications of persistent homology in complex data are explored [2
Topological features in vision data have been investigated broadly, including leveraging the prior topology knowledge to improve the segmentation results 226 27 and exploiting high-order per- sistence spectral representations in image embedding [27]. These works are all based on explicit object-level topology features. Few insights about TDA in category-level vision data have been investigated. Liu et al. (28] propose a topological prediction model in image dataset, while is not learnable to meet the requirement of deep learning. Madhu and Chepuri [29] introduce the simplicial contrastive loss to preserve more expressive consistency, while only considering neighbors' aggregated representation. So far, there are limited results on learnable intrinsic category features to summarize vision data and investigate the structural stabilization of data distribution.
Continual Learning Continual learning methods can be categorized into three groups. The regularization- baased methods selectively regularizing the variation of essential network parameters 2 or preserve the outputs consistency of prediction functions (1] Replay-based methods store few representative samples for retraining or old knowledge distillation [4 or utilize an
additional generative model to replay previous data 332 33 The parameter isolation methods isolate
parameter subspaces dedicated to each task throughout the network 8)
To address the limitation of continual learning in neglecting tremendous unlabeled data, recent efforts have been directed towards semi-supervised continual learning ( I2 [ [5] [3. Some of these methods leverage the generalizability of unlabeled data to alleviate forgetting by incorporating gradients of unlabeled data [22 or employing contrastive loss among unlabeled data [TS]. Such strategies require a substantial memory buffer to ensure satisfactory performance. To deal with the memory limitation, a generative reply method regularizes discriminator consistency to mitigate catastrophic forgetting [11. Employing meta distribution to consolidate the task-specific knowledge 34] and training an adversarial auto-encoder to reconstruct images [35] extends [[]] further. However, these methods incur prohibitive costs nigher-resolution images. Knowledge distillation strategies have also been explored on unlabeled data to address the forgetting in SSCL, including sample-wise representation [9, class-instancd relationships [13], and local-neighbon neiall-eighborhood similarity [14]
In continual learning scenarios with limited labeled data, the required changes of feature space can often be dramatic [I] and the learned class-related representation is usually biased to labeled data [13 With the unreliable distribution, stabilizing the feature extractor may interfere with accommodating new representations and subsequent task refinement. A more stable representation learning robust to data transformation required for the SSCL.
3 Methodology
3.1 Preliminaries on Persistence Homology
A k-simplex a is a k-dimensional polytope which the convex hull of its k+1 vertexes. single point can be regarded as a 0-simplex, and line segment be defined as I-simplex. A 2-simplex a triangle, and a 3-simplex can be seen as a tetrahedron. The convex hull of any nonempty subset of the k+1 points that define a k-simplex is called a face of the simplex. A simplicial complex K. is a set of simplex that satisfies the following conditions. (1) Any face of a simplex from K. is also in K. (2) The intersection of any two simplices σi, 02 K. is a face of both σ and σ2. A common choice of simplicial complex CC,, is the Vietoris-Rips simplicial complex R(X, r), in which a simplex {21, F2,,... .., Fm, E R(X,r) when dist(x; 22)) r for all S i, S m.
A filtration (Kr of point cloud in & can be constructed from any filter function f Rd IR by considering each simplicial complex to be the sublevel set of f threshold by r, K, = {a Rdd [=( S r). The underlying filter function introduced above is the distance function dx Rd - R, where (y) minndisstl .)}{:: E X) is the distance from y to point set X. To be specific, the Vietoris-Rip simplicial complex R(X, r)) approximates the sublevel set
K, = 6x r)) ty E IR 1x (v) <r) U(,x
(I)
3

---

Homolocy Group
0.6
0.4-
Fllrration Value
Birth
(b)
(c)
Figure 2: Illustration of topological data analysis. (a) Filtration of simplicial complex, (b) correspond- ing persistence barcode, and (c) persistence diagram.
where [xx; r) is ball with radius centered around of subcomplex set
X. Then the filtration is a nested sequence
Ko K1 C... C Km =K,
which reflect multi-scale topological spaces underlying the point data, shown in Figure 2 a). The persistence homology quantifies the changes in topological features as the subcomplexes grow with the increasing threshold of the filter function. The strict definition of persistence homology provided in Appendix A An easier way to understand as follows. The lifetime of each multi-scale topological feature is recorded by persistence barcode, as shown in Figure [)). Finally, the persistence barcode can be converted into a persistence diagram, as shown in Figure 2(c). Dgm {(I,, Ba) R² 2G Y/r U A are scatter plot with the and y respectively depicting the scale Ri at which each topological feature born one subcomplex and dies threshold Yo or is identified with another homology class. A is the diagonal set containing points in PH.
3.2 Persistence Homology Distillation for Semi-supervised Continual Learning
Formulation of SSCL As shown in Figure 3 a), when encountering new data D:, let f be the model to be trained, with both labeled data L and unlabeled data U input into the model. The parameters 0 are updated through the cross-entropy loss Lce and semi-supervised loss LSSL to adapt the model to the new task. We follow DSGD 14 to employ Fixmatch [66 as LSSI. Simultaneously, to prevent catastrophic forgetting, the memory buffer M used to regularize the knowledge consistency between DK()') and DK (h--1) based on the previous model f/- through representation distillation loss Lrd. Now we propose new persistence homology distillation loss for unlabeled data Lnd as part of the continual learning loss. The complete loss function for SSCL as follows:
Lasct LCE + LSSL Erd + XLnd-
Persistence Homology Distillation Existing distillation methods preserve the sample represen- tation, the class-instance similarity between tasks, and local neighbor similarity. This limits the performance when inaccurate representation is inherited or noise occurs in previous tasks, especially when a significant transfer between the feature spaces of two tasks exists. To exploit the intrinsic structural features preservation, which topological invariant regardless of specific representation, propose persistence nomology-based distillation method and design an effective algorithm to achieve stable knowledge accumulation.
The first step of our model is to define the topological similarity among two samples. Given collection image sets, we build the local persistence diagram by considering the weighted k-hop neighborhood N War;, k). To be specific, let G (V, E) be the adjacency graph of memory buffer, where V the set of replayed image samples. W IR NxN X be the adjacency matrix of G such that We = 1 for eij E and 0 for otherwise. As discrete point data, construct the adjacency matrix

---

L
fb
C1S
0.6
D2
u
fb
L
New features
0.
0.
Baub
0.
0.
k-hop
Simplicial Complex
Persistent Diagram
Neighborhood
(K)
L.
X.L
fb
DKK (X")))
Distillated knoliwedge
Old features
Bath
(a)
(b)
Figure 3: Illustration of our proposed persistence homology distillation for cumi-supereieed continual learning. (a) represents the backbone of SSCL, LCg and LSSL are the cross-entropy loss on labeled data and semi-supervised loss on unlabeled data. Lrd means the representation distillation loss on the memory buffer. (b) corresponds our PsHD loss employed on the replied unlabeled samples.
by applying threshold to the cosine similarity 8ij =< hi >=
hehy TA,
1. if s 3 st 3,
Wi = 0,
where h;s hy are the representations of sample xa and æy, respectively. We use the logits h, Rd in this study. Next, we can group the weighted k-hop neighborhood set N(x, k). where for any I, Nax k), the shortest path between T; and xy is at most k. Armed with the distance function dist( = 8ij. we can construct the h-dimensional persistence diagram (N (M k)) by considering the distance function the filtration function according Eq 1-2.
Subsequently, we try preserve this structural information by designing distillation loss, which goes beyond only the pairwise similarity of node features. Given any sample x of the memory buffer, can derive the persistence diagrams of the old task by the old feature extractor Dgm,, fota). The corresponding persistence diagram in the new model is Dgm, (N*t fnew). where the neighborhood set is the same as the previous task. The structural similarity between two tasks is measured with the p- Wasserstein distance between the corresponding persistence diagrams as
dy (fold, fnew) inff((
Σ
| (w))))
(5)
aE DEDgm (NL .foia)
where 2 and y is taken over all bijective maps between Doma (N% fota) and Dgma (Nk, fnew). our analysis p=1. This distance quantifies how much effort needed transform one topological feature representation into another.
known that the worst-case complexity of generating a homology diagram is On where n is the number of the simplices (37] Thus, the complexity of computing local topology similarity for each sample the memory buffer is ( In2.5), which is computationally expensive. This procedure is also redundant since nearby samples typically share similar topological structures. To mitigate the computation cost, we design approximate algorithm to simplify the final distillation loss and accelerate the persistence homology distillation procedure.
Persistence Homology Distillation Acceleration Algorithm
Given the replayed data M, we randomly select one sample In and group its weighted k-hop (Zx neighborhood WV (x1.k). k)). We then select another sample x2 in the remaining samples M (N(x k). and group its weighted k-hop neighborhood set A M(r2, k). This process is repeated until all samples are grouped into their respective neighborhood sets. After this step, we can compute several homology diagrams derived from grouped k-hop neighborhood sets {N(x1, (it,,k), k),..., k)}. We design the final distillation loss as Lnd in SSCL by meaning of all separated local homology diagrams
((12,))
1
Chd
dx (fotdr fnew),
(6)
S
5

---

where I is the cardinality. This procedure is summarized as Algorithm| I in Appendix A.2 By doing so, the complexity is reduced from M to (() We experimentally find that the size of S' is approximate to the number of classes in the memory buffer, so the ([ is approximated be (CCarn This contributes to significant reduction in computation, approximately 10-fold when 1000 unlabeled samples are replayed in CIFAR-100.
Stability Analysis The superiority of proposed persistence homology distillation lies in its insensitivity to noisy and inaccurate representation. According to the stability of persistence homology [38]. we can infer that persistence homology distillation inherently more robust than feature distillation. We also demonstrate its robustness compared to the similarity distillation in Theorem [
Theorem 1. Given the two feature space of point data as Po and Pi, for all p 2 1 and h E Z*, we
have
(M 1M-1) Wpair (Po. (..Dgmm(((P))) (gm,, (
(7)
where Dgm, (P() and Dgm,, (R(P:) are the h--imensional persistence diagram for the Vietoris-Rips filtration the point set Po and Pi respectively. W represents the pair-wise
distance between the point set Wpasr (Po, Pi) in |||u ul [| - ( PP)
where d is bijection between the two set Po and Pi.
The proof is shown in Appendix A.3 This theorem demonstrates that the persistence homology is
robust the perturbations of similarity between samples and inaccurate sample-wise representation.
4 Empirical Study
4.1 Experiment Settings
Datasets.
We evaluate our method on three datasets with different classes and resolutions. CIFAR-
10 (39] is a dataset containing colored images classified into 10 classes, which consists of 5,000
training samples and 1,000 testing samples of size 32 32 per class. CIFAR-100 [39 comprises 500
training images and 100 testing images per class, with the same size with CIFAR- 10. ImageNet- 100
40 subset of ImageNet-
is composed of 100 classes with 1300 images per class for training
and 500 images per class for validation. ImageNet- 100 resembles real-world scenes with a higher resolution of 256+256.
Continual Semi-supervised Setting. For both CIFAR-I0 and CIFAR-100, train the models with different levels of supervision, i.e,, À E {0.8%, 5%, }.%}. For instance, the label ratio corresponds 4, 25, and 125 annotated samples per class in CIFAR-100. Regarding ImageNee-1000 we choose (1%, 7.7%} for label ratio. To build the continual datasets, we use two learning sequence settings the literature NNCSL. 13 and DSGD 14, and divide the datasets into equally disjoint tasks: 5/10/10 tasks for
WWWCCIIAAROO//maaaa 100/mmaeeeet- 100, i.e., 2/10/10 class per task. We also follow the standard incremental setting of CL and assume that only limited space is left for previous data. In other words, the memory buffer can be constructed for both labeled and unlabeled data. Following 13 7,
we set the buffer size as 500 and 2000.
Evaluation Metrics. We evaluate the performance different methods considering the average accuracy over all the classes seen after the last task AAT + Σ or, and average incremental accuracy of each task AIA - Etz AA, [.. Analysis with backward transfer, which presents the degree of forgetting BWT - IT-1 (ar, at,z), is also evaluated in the ablation study. The forgetting is usually reflected as negative BWT.
Baselines. We adopt the semi-super continual learning pipeline of DSGD 14 Two categories of methods: distillation-based method iCaRI[4] and parameter consolidation method DER 7 are considered as the strategies for overcoming catastrophic of labeled data. A common semi-supervised learning method Fixmatch (36) is adopted to learn discriminate representation of unlabeled data.

---

Table 1: Average incremental accuracy and average accuracy of various methods on 5-tasks CIFAR- 10, 10-tasks CIFAR- 100 and 10-tasks ImageNet- 100 settings following the learning sequence [14] with memory buffer size of 2000. The improvements of PsHD compared to the state-of-the-art methods are highlighted in blue color.
CIFAR-10
CIFAR-100
ImageNet-100
Method
5%
25%
5%
25%
%
7.7%
last
avg
last
avg
last
avg
last
avg
last
avg
last
iCaRL
65.9
56.4
65.4
51.4
28.1 1
15.3
44.1
30.7
19.9
12.9
30.8
16.7
iCa Cax_Fix 14
83.7
79.2
82.9
78.8
49.8
31.3
56.9
41.4
26.4
15.6
37.5
21.0
DSGD
83.8
79.1
83.22
79.0
53.4
35.9
58.1
43.1
28.3
19.1
50.5
32.1
PsHD
85.4
81.5
81.4
56.6
38.8
59.4
43.2
29.7
22.4
53.4
35.0
Gain
+1.6
+2.4
+2.9
+2.4
+3.2
+2.9
+1.3
+0.1
+1.4
+3.2
+2.9
+2,9
DER
70.4
65.6
70.9
62.3
32.8
26.5
57.2
48.8
20.8
14.4
41.2
32.6
DER Fix 14
86.1
81.2
83.9
81.4
52.0
44.5
66.7
53.6
29.8
23.3
59.5
51.0
DSGD 14
86.3
81.6
85.0
81.9
57.9
46.7
69.1
58.5
29.8
23.3
59.9
51.5
PsHD
86.0
81.8
88.0
83.4
58.3
47.2
69.5
58.7
30.7
24.2
60.6
52.1
Gain
+3.0
+0.4
+0.4
+0.2
+0.9
+0.7
+0.6
Table 2: Average accuracy of different methods with 5-tasks CIFAR-10 and 10-tasks CIFAR-100 settings following learning sequence of [13 with 500 samples replayed. The data with underline is the best performance within existing methods.
CIFAR-10
CIFAR- 100
Method
Source
0.8%
5%
25%
0.8%
5%
25%
iCaRL 4
CVPR 2017
24.7
35,8
51,4
3.6
11.3
27.6
FOSTER 6
ECCV 2022
43.3
51.9
57.1
4.7
14.1
21.7
X-DER 42
TPAMI 2022
33.4
48.2
58.9
8.9
18.3
23.9
PseudoER
ICCV 2023
56.5
57.0
8.7
11.4
CCIC 151
2RI.2021
54.0
63.3
63.9
11.5
19.5
20.3
PAWS
ICCV 2021
51.8
64.6
65.9
16.1
21.2
19.2
CS
ICCV 2023
64.5
69.6
70.0
23.6
26.2
29.3
NNCSL [3]
ICCV 2023
77.2
77.3
27.4
31.4
35.3
DER
Fix DER_Fix
AAAI 2024
73.4
75.2
76.5
25.4
41.3
46.3
D4
AAAI 2024
73.6
76.1
77.5
25.9
41.6
47.1
PsHD
Ours
74.5.11
77.6.0.
78.0,0.5
27.7+0.3
42.4+08
47.8.07
4.2 Quantitative Results
Comparison with SSCL Methods. Table [ shows the disrupted class order following [14 across the three datasets, and also report the average incremental accuracy demonstrate the effectiveness of the proposed method during the whole evolution. The backbone architectures remain consistent with those used in [14, while the distillation loss on unlabeled samples the memory buffer is replaced by our newly proposed persistent homology distillation loss. We obtain general improvement on the two backbones, demonstrating the intrinsic topological features captured by PH are more efficient than the sample-wise local similarity distillation. In addition, the scale of the sub- graph is an important parameter that should be selected in DSGD [4], which does not occur in the proposed PsHD since its automatic multi-scale characteristic. The results of ImageNet- 100 also demonstrate the effectiveness on a more challenging benchmark with a higher resolution.
For more comprehensive comparison, we also validate our method on the default class sequence. Table 2 shows the results of semi-supervised continual learning on CIFAR-I0 and CIFAR- without shuffle class order. It can be seen that our PsHD outperforms all the methods in the average accuracy across several label ratios on both datasets. As representative relation distillation methods, NNCSL [33 and DSGD [14 show comparative results compared to sample representation distillation. Our method surpasses the two approaches further, indicating the superiority of distilling persistent homology for unlabeled data in semi-supervised continual learning. In addition, NNCSL [13] only

---

considers the rehearsal of labeled data, limiting its potential advantages on unlabeled data, while our methods focus the unlabeled data and complement the class-instance similarity preservation.
Better Adaptability Unlabeled Data Com- pared with Distillation Methods. Table 3 presents a comparison of traditional knowledge distillation and its more recent variants from [48] in the learn- ing tasks of our SSCL problem. The effectiveness of logits and features distillation diminishes gener- ally, particularly as tasks become more challenging and numerous. For instance, the performance CIFAR 100 is lower than that on CIFARIO I0, and a reduced labeled ratio results in decreased accuracy. Although relation and topology distillation are rel- atively effective for SSCL, their stability is still inferior to that of our method.
Table 3: Average accuracy of different knowl-
edge distillation methods applied on SSCL.
CIFARIO CIFAR100
Method
Type
5%
25%
5%
25%
iCaRL 4
logits
79.2 78.8
31.3
41.4
Foster 6
logits
75.0 70.4
24.6
38.8
LUCIR 44
feature
75.2 74.7
32.0
32.6
Podnet 45
feature
57.9
69.0
21.4
21.1
R-DFIL 46
relation 78.5 78.8
34.7
34.2
D4
relation 79.1 79.0
35.9
43.1
TopKD 47
topology 78.7
79.8
35.4
41.9
PsHD
topology 81.5 81.4
38.8
43.2
Stability to Noise Interference. We also validate
the stability of our methods under noise interference. As shown in Table 4 we add Gaussian noise with standard deriviation {0.2, 1, 1.2] on five distillation-baser continual learning methods and our PsHD. The results indicate that our method exhibits greater stability terms both forgetting degree and accuracy, despite the presence of higher noise levels. These findings further confirm the stability conclusions drawn Theorem[]
Table 4: Comparison of distillation methods with Gaussian noise interference on CIFAR10 with 5% supervision. 0 is the standard deviation.
P4dnet(45]
LUCIR 44
R R-DFCIL [46
DSGD
TopKD
PsHD
BWT,
AAT
BWTJ
AAt
BWT,
AAT
BWT|
AAT
BWT,
AAT
BWT
AAt
0.2
31.8
55.6
21.8
71.5
19.2
77.5
18.1
76.7
19.7
76.5
14.0
1.0
44.4
27.9
34.3
57.3
28.2
62.1
27.1
64.2
23.0
67.2
17.7
70.9
1.2
49.8
34.6
36.4
56.4
26.4
61.8
23.7
64.1
22.8
65.8
18.3
67.8
4.3
Parameter Analysis
Ablation Study. We firstly ablate the components of persistence homology distillation through quantitative analysis in Table on both CIFAR-10 and CIFAR-100 across 5% and 25% label ratios. The baseline is based on iCaRL_Fix and DER_Fix, where the iCaRL and DER are two methods traditional continual learning and Fix Fixmatch for semi-supervised learning. LSSI.. and Lnd represent the semi- auiissuereised loss and homology diagrams distillation loss, respectively. The general improvements our structural distillation loss confirm the contributions of our proposed components.
Visualization of the Effectiveness of Persistence Homology Distillation. We also provide the class activation heatmap of old categories in the initial and following tasks during the continual learning process in ImageNet- 1ot-100) 00. These heatmap pictures localize class caass-spccficc discriminative regions, as shown in Figure persistence homology distillation usually drop the class-specific attention region with the task takes advancement, despite the init model having learned a discrininative representation. In addition, base model may even shift focus on the background area in the middle tasks, such as Task 6-9 in 2.a, resulting in unreliable knowledge accumulation. This is because of the insufficient previous data support, memory data under-utilization, and feature space transformation, which lead to attention shift as shown in La, and even attention vanishment as shown in 3.a.
can be seen from (a) part of the three groups that the model without our
In contrast, the introduction of our persistence homology distillation is able to preserve the discrimi- native region along with the whole continual learning, as depicted in (b) part, which indicates that preserving persistence homology that captures the class-wise structural information encourages the model to focus on the class-specific region continually.
worth noting that even if the initial model is not capable of noticing the class- region, as shown in 3.a, which may result from insufficient supervision, our method still empowers the following model training to activate the potential class-relevant region since it focuses on the global distribution of each class and pushes the
8

---

Table 5: Ablation study of proposed persistence homology distillation
Method
LSSr.
Lnd
CIFARIO 5%
CIFARIO
25%
CIFARI00 5%
CIFARI00 25%
avg
last
avg
last
avg
avg
iCaRL
V
v
85.3.15
83.7
80.8.16
79.2
85,4.16
82,9
80.5
78.8
56.4,2.
54.2
38.8,2.7
36.1
59.240s
58.4
42.2-11
41.1
DER
86.1
81.2
84.9
81.4
57.8
46.3
68.1
57.2
86.2.a.
81.8.aa
87.9.30
83.4.2a
58.2,04
47.2,09
68.3.0a
57.7.0.
TaskO
Task1
Task2
Task3
Taske
Task5
Task6
Task7
Task8
Task9
1. Attentioo shift.
R
3. Attention deficiency 2. Attertionndca docay
(8)
QEAND
(
Blssoo
(b)
Figure 4: Visualization of activation heatmap during the continual learning process, where the cate-
gories belong to TaskO. 1-3(a) correspondence to PsHD without the Lnd, and 1-3(b) correspondence to PsHD with Led. The red area localizes class-specine discriminative regions.
semantic similar samples be related in representation, as shown in 3.b. Additional visualization of PsHD is depicted in Appendix A.5)
Effect of the Weight of Persistence Homology Loss. We evaluate the influence of loss weight in Eq. across the three datascts, and report the results in Figure 5 can seen that a higher weight indicates higher average incremental accuracy and lower forgetting degree generally. In addition, our method improve the baseline generally within a reasonable range. Too much higher weight persistence homology loss, for instance bigger than 1, is not supposed to be better for the trade-off problem between old and new tasks. We choose for CIFARIO and CIFAR100, and 1.5 for ImageNet100 in experiments.
Effect of k-simplex in Persistence Homology. The persistence diagram calculation considers lift-span of /--dimensional holes that existed in the nested simplicial complexes data distribution. We verify the effectiveness of dimension h in the homology diagram by considering O-dimensional holes and both 0, 0.l-dimensional holes in CIFAR- 10 and CIFAR- 100. The results shown in Figure 5 can be seen that considering the persistence of both 0, -i--mmnsiiooal holes are more effective in CIFAR- 10, while CIFAR-100 100 slightly prefers the simple O-dimensional holes persistence preserved.
Effect of Memory Buffer Distribution. To investigate the necessity of replaying the unlabeled data, we evaluate the effect of memory buffer allocation of labeled and unlabeled data on CIFAR- 100 of 5% and 25% label ratios Table where ratios represent the percentage labeled data in the memory buffer. The general improvements indicate the percentage of labeled data has important effect on the performance. can be observed that replacing 20%-40% labeled samples with unlabeled samples improves 1.49%, 1.27% in 5%. We experimentally choose 0.6-0.9 in the following settings.

---

CIFAR10 3%
CIRA R100 5%
ImageNett 100 7.7%
58
130, DWT
H10 Avg
BWWLBWT
HOL_AVg
50
85.5
HO, Avg
HU HWT
HOL_AN
HOT BWT
Avg
58
57
84.5
30
Aug!
54
BWN
55
20
82.5
45
54
53
0
0.1
0.5
1.2
1.5
81.5
1.2
1.5
44
Figure Effectiveness h-simplex features persistent homology. HO 1vo_ and HO1 Avg represent the average incremental accuracy based on considering O-dimensional holes and 0,1-dimensional holes persistence. BWT evaluates the forgetting degree.
Table 6: Effect of data allocation in memory buffer
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
5%
53.94
54.60
55.43
54.63
55.21
54,45
54.57
54.30
54.21
54.52
25%
65.32
65.82
65.48
65.45
65.50
65.44
65.17
65.3
65.00
56.10
4.4 Efficiency Analysis
Memory Reduction. Our method is also memory-efficient for SSCL. As shown in Table 7 the proposed PsHD that incorporates unlabeled data surpasses methods relying solely on labeled data, such as NNCSL, by an average of 3.9% across six SSCL settings. To further ver- ify the memory efficiency, we reduced the memory buffer size by 60% to 2000, achieving an average improvement 1.5% compared to NNCSL across the six semi-supervised continual learning tasks.
Table 7: Average accuracy of different methods following
learning sequence of [13] with memory buffer size 5120.
represents that the size of memory buffer is 2000.
CIFAR-10
CIFAR-100
Method
0.8%
5%
25%
0.8%
5%
25%
PscudoFR 13
55.4
70.0
71.5
15.1
24.9
30.1
CCIC 15
55.2
74.3
84.7
12.0
29.5
44.3
C3
64.3
73.1
73.9
23.7
41.8
50.3
NNCSL
73.7
79.3
81.0
27.5
46.0
56.4
PsHD
82.3
82.5
87.2
28.9
47.7
57.5
PsHD
81.5
84.6
28.3
46.0
56.5
Computation Efficiency. A newly topology-based distillation method TopKD 47), which is similar with our method, is proposed for model compression. We prove that the complexity proportion of TopKD and our method is b2, where is the number of old classes. The specific derivation clarified in Appendix A.44 This value indicates the relative complexity of TopKDD grows polynomially when the class number increases, highlighting our computation superiority.
We evaluate the memory storage, model parameters and training time between methods and state-of-the-art methods in Appendix [A.4 Table indicating the superiority, Despite the acceleration algorithm, a limitation of our method is that the computation of the persistence diagram relies on the Guidh package, which operates on the CPU. We plan to explore the learnable network RipsNet [49 to approximate the persistence diagram and further accelerate the computation in the following work.
5 Conclusion
We proposed Persistence Homology Distillation (PsHD) to address catastrophic forgetting and noise sensitivity Semi-Supervised Continual Learning (SSCL). First, we utilized persistence homology to capture stable topological feature representation of unlabeled data in semi-supervised learning. Next, we proposed novel persistence homology distillation strategy for SSCL that is insensitive to noise information interference based on the intrinsic topological features and devised an accelerating algorithm to reduce computation costs. Finally, general improvements in three benchmarks demonstrated the efficiency of PsHD in overcoming catastrophic forgetting in SSCL.
10

---

Acknowledgments and Disclosure of Funding
This work was supported in part by the National Science and Technology Major Project under Grant 2022ZD01 16500, in part by the National Natural Science Foundation of China under Grants 62436002, 62476195, 61925602, U23B2049, 62222608, and 62406219, part by Tianjin Natural Science Funds for Distinguished Young Scholar under Grant 23JCJQJC00270, in part by Zhejiang Provincial Natural Science Foundation China under Grant LD24F020004, and part by China Postdoctoral Science Foundation Tianjin Joint Support Program under Grant Number 2023T014TJ.
References
[1] Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE Trunsactions on Puttern Analysis and Machine Intelligence, 2024.
[2] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Barbbkk-Barwinkaa,, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):5211-322 2017.
[3] Liyun Wang, Mingtian Zhang, Zhongfan Jia, Qian Li, Chenglong Bao, Kaisheng Ma, Jun Zhu, and Zhong. Afec: Active forgetting negative transfer in continual learning. Advances in Neural Information Processing Systems,
2021.
[4] Sylvestre Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. iCaRL: Incremental classifier and representation learning. Proceedings 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017-JJnuary:5533-555
2017.
[5] David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[6] Fu-Yun Wang, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: Feature boosting and compression for class-incremental learning, arXiv preprint urXiv:2204.04662. 2022.
[7] Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3014-3023, 2021.
[8] Da-Wei Zhou, Qi-Wei Wang, Han-Jia Ye, and De-Chuan Zhan. A model 603 exemplars: Towards memory-efficient class-incremental learning. In ICLR, 2023.
[9] James Smith, Jonathan Balloch, Yen-Chang Hsu, and Zsolt Kira. Memory-efficient semi- supervised continual learning: The world is its own replay buffer. In 2021 International Joint Conference Neural Networks (IJCNN, pages 1-8. IEEE, 2021.
[10] Aras Asaad and Sabah Jassim. Topological data analysis for image tampering detection. In Digital Forensics and Watermarking: 16th International Workshop, IWDW 20/7, Magdeburg, Germany, August 23-25, 2017, Proceedings 16, pages 136-146. Springer, 2017.
[11] Liyuan Wang, Kuo Yang, Chongxuan Li, Lanqing Hong, Zhenguo Li, and Jun Zhu. Ordisco: Effective and efficient usage of incremental unlabeled data for semi-supervised continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5383-5392. 2021.
[12] Yan Luo, Yongkang Wong, Mohan Kankanhalli, and Qi Zhao. Learning predict gradients for semi-supervised continual learning. arXiv preprint arXiv:2201.09196 2022.
[13] Zhiqi Kang, Enrico Fini, Moin Nabi, Elisa Ricci, and Karteek Alahari. A soft nearest-neighhor framework for continual supi-super learning. In Proceedings of the IEFF/CVE Interna- tional Conference on Computer Vision, pages 11868-1188 11877, 2023.
11

---

[14] Yan Fan, Yu Wang, Pengfei Zhu, and Qinghua Hu. Dynamic sub- -ab-rraph distillation for robust semi-supervised continual learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 1927-11935, 2024.
[15] Matteo Boschini, Pietro Buzzega, Lorenzo Bonicelli, Angelo Porrello, and Simone Calderara. Continual semi-supervised learning through contrastive interpolation consistency. Pultern Recognition Letters, 162:9-144 14, 2022.
[16] Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? Advances in Neural Information Processing Systems, 2021.
[17] Luke Melas-Kyriazi, The mathematical foundations of manifold learning. arXiv preprint arXiv:2011.01307, 2020.
[18] Tamal Krishna Dey and Yusu Wang. Computational topology for data analysis. Cambridge University Press, 2022.
[19] Songlin Dong, Xiaopeng Hong, Xiaoyu Tao, Xinyuan Chang, Xing Wei, and Yihong Gong. Few-shot class-incremental learning via relation knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1255-1263, 2021.
[20] Qiankun Gao, Chen Zhao, Bernard Ghanem, and Jian Zhang. R-dfcil: Relation-guided represen- tation learning for data-free class incremental learning, In European Conference on Computer Vision, pages 423 4399. Springer, 2022.
[21] Qi Zhao, Ze Ye. Chao Chen, and Yusu Wang. Persistence enhanced graph neural network. In International Conference on Artificial Intelligence and Statistics, pages 2896-2906. PMLR, 2020.
[22] Qi Zhao and Yusu Wang. Learning metrics for persistence-based summaries and applications for graph classification. Advances in Neural Information Processing Systems, 32, 2019.
[23] Christoph Hofer, Florian Graf, Bastian Rieck, Marc Niethammer, and Roland Kwitt. Graph filtration learning. In International Conference on Machine Learning, pages 4314-4323, PMLR, 2020.
[24] Zuoyu Yan, Tengfei Ma, Liangcai Gao, Zhi Tang, and Chao Chen. Link prediction with persistent homology: An interactive view. In International conference machine learning. pages 11659-11669, PMLR, 2021.
[25] Zixuan Cang, Lin Mu, and Guo-Wei Wei. Representability of algebraic topology for biomolecules in machine learning based scoring and virtual screening. PLoS computational biology, 14( (4(1)ee10059)) 2018.
[26] James R Clough, Nicholas Byrne, Ilkay Oksuz, Veronika A Zimmer, Julia A Schnabel, and Andrew P King. topological loss function for deep-learning based image segmentation using persistent homology. IEEE transactions on pattern analysis and machine intelligence, 44(12):8766-8778, 2020.
[27] Hongliang He, Jun Wang, Pengxu Wei, Fan Xu, Xiangyang Ji. Chang Liu, and Jie Chen. Toposeg: Topology-awar aware nuclear instance segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 21307-21316. 2023.
[28] Meng Liu, Tamal K Dey, and David Gleich. Topological structure of complex predictions. Nature Machine Intelligence, 5(12):1382-1338) 2023.
[29] Hiren Madhu and Sundeep Prabhakar Chepuri. Toposrl: Topology preserving self-supervised simplicial representation learning. Advances in Neural Information Processing Systems, 36, 2024.
[30] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEFE Transactions on Pattern Analysis and Machine Intelligence, 40(12):9955-2947, 2017.
12

---

[31] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In Computer Eccvn 2020: 16th European conference, Glasgow, UK, August 23-28, 2020, proceedings, part XX 16, pages 86-102. Springer, 2020.
[32] Craig Atkinson, Brendan McCane, Lech Szymanski, and Anthony Robins. Pseudo-recursal: Solving the catastrophic forgetting problem in deep neural networks. arXiv preprint arXiv:1802.03875. 2018.
[33] Gido M Van de Ven, Hava T Siegelmann, and Andreas S Tolias. Brain-inspired replay for continual learning with artificial neural networks. Nature communications, 11(1):4069 2020.
[34] Dhanajit Brahma, Vinay Kumar Verma, and Piyush Rai. Hypernetworks for continual semi- supervised learning. arXiv preprint arXiv:21 2021.
[35] Alexis Lechat, Stéphane Herbin, and Frédéric Jurie. Semi-supervised class incremental learning. In 2020 25th International Conference on Pattern Recognition (ICPR), pages (0383-10389, IEEE, 2021.
[36] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raf- fel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. Advances in Neural Information Process- ing Systems, 33:596-608 608, 2020.
[37] Nina Otter, Mason A Porter, Ulrike Tillmann, Peter Grindrod, and Heather A Harrington. A roadmap for the computation of persistent homology. EPJ Data Science, 6:1-38, 2017.
[38] Primoz Skraba and Katharine Turner. Wasserstein stability for persistence diagrams. arXiv preprint 2rXiv:2006.16824 2020.
[39] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[40] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255. Icee, 2009.
[41] Da-Wei Zhou, Qi- Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Deep class-incres ciass-incrementll learning: A survey. arXiv preprint arXiv:2302.03645 2023.
[42] Matteo Boschini, Lorenzo Bonicelli, Pietro Buzzega, Angelo Porrello, and Simone Calderara. Class-incremental continual learning into the extended der-verse. IEEE transactions on pattern analysis and machine intelligence, 45(5):5497-5512, 2022.
[43] Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael Rabbat. Semi-supervised learning of visual features by non-parametrically predicting view assignments with support samples. In Proceedings of the IEEF/CVE Interna- tional Conference on Computer Vision, pages 8443-8452, 2021.
[44] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 831-839, 2019.
[45] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In Computer vision-ECCV 2020: 16th European conference, Glasgow, UK, August 23-28, 2020, proceedings, part XX 16, pages 86-102. Springer, 2020.
[46] Qiankun Gao, Chen Zhao, Bernard Ghanem, and Jian Zhang. R-dfcil: Relation ciaatinn-uuied represen- tation learning for data-frec class incremental learning. In European Conference on Computer Vision, pages 423 439. Springer, 2022.
[47] Jungeun Kim, Junwon You, Dongjin Lee, Ha Young Kim, and Jae-Hun Jung. Do topological characteristics help knowledge distillation? Forty-first International Conference on Machine Learning.
13

---

[48] Songze Li, Tonghua Su, Xuyao Zhang, and Zhongjie Wang. Continual learning with knowledge distillation: A survey. Authorea Preprints, 2024.
[49] Thibault de Surrel, Felix Hensel, Mathieu Carrière, Théo Lacombe, Yuichi Ike, Hiroaki Kurihara, Marc Glisse, and Frédéric Chazal. Ripsnet: a general architecture for fast and robust estimation of the persistent homology of point clouds. In Topological, Algebraic and Geometric Learning Workshops 2022, pages 96-106. PMLR, 2022.
[50] Daniel Freedman and Chao Chen. Algebraic topology for computer vision. Computer Vision, pages 239-268, 2009.
14

---

A
Appendix supplemental material
Content
Preliminaries of Persistence Homology
A.2 Algorithm of Persistence Homology Distillation
A.3 Proof of Stability Theorem
A.4 Efficiency Analysis of Proposed Algorithm
A.5 Visualization of Effectiveness PsHD
A.1 Preliminaries
Persistence Homology. Now we explain how persistence homology works. It is easy to obtain an inclusion map K; + Kj < homomorphism of the corresponding homology groups
from Eq. 2 These maps also induce group
rr Ha(K,) - Halk)),<i<j<<
(8) where Ha(K;) and Ha(Ky) homology groups of the dth order. A homology class h is born at time fa Ha(K.) but & img(F2 Provided born at fis dies at time fa only if (h) of img(FF map). These maps reveal which feature features persist (a feature refers homology class, corresponding to component or n-dimensional hole). An easier way to understand is as follows. A new homology class is born time fa only it does not appear at the existing features, and dies at time fj only if it still persists at time fj-1 and is merged into other features at time fz. The time between death and birth feature called its lifetime or persistence.
while F img(F) [50]. ('img' refers to the image of
A.2 Algorithm of Persistence Homology Distillation
We illustrate our persistence homology distillation in Algorithm
Algorithm 1 Persistence Homology Distillation
Input: Replayed batch of unlabeled data: B% Parameter: Embedding encoder of current task fnew and old task fold,
1: E < B / 0,S 0
2: repeat
3:
RandomSamplee(E)
4:
(Snew :Joid)
5: Group the weighted k-hop neighborhood set (xx; k) by fold
6: E+ E,, k), + S-SSU U xi, <- i+1
7:
Compute the Wasserstein distance dyę (Jatd, fnew) according to Eq.
8: until empty
9: Minimize the total loss Lnd
1 Exes dAy (folds fnew)
[S]
A.3 Proof of Stability Theorem
Theorem 1.
Given the two feature space of point data Po and Pi, for all p 2 1 and h Zt,
have
W, W,(Dgm (R(Pa)) Dgma (R(P)))) < ( Wpair (Po. Pi),
k
where Dgm,, (R(Po) and (RR(1) )) are the h-dimensional persistence diagram for the Vietoris-
Rips filtration on the point set Po and Pı respectively. Wrearr represents the pair-wist wise distance
between the point set Wpair (Po, Pi) = inf (2.. |||e - ull - [[o(w) d(w)] (p) where ei a bijection within the same set, such as Po - Po or P, - Pi. and & is bijection between the two set Po and P..
15

---

Proof. Let o: Po + P1 be the bijection which achieves the minimum of :P: Wp (Po. P1) inf( Σ
||u w11 [(((()) <(v)] )(
a.uEPo
Relabel the points in Pu = f F and P1 = fy...... yud } so that x(x;) = y1- Let
K be the complete simplicial complex on M vertices {01,-.. UM ). Define functions f.g K + R by f( 8/ : U,,l) the time when [Ti:: Pi,.". _ is included in ((Pp) and a((%% Vi1=" :. (iy]) the time when (Wig W ,ii, is included in R(Pi).
Suppose h 2 then
viul)) g(1ing Vi. whel) (mexx( - =2:1133 max maxi[lus Me
3.1
S max
|| -aall -1lvi, well
S max Σ || -xall - |1
ya |||-
Since K the complete simplicial complex over M vertices, each edge [1s, Uz] appears in (x-1) h-simplices, we only need decide which extra k-1 vertives to include.
Based on the cellular stability theorem [8
Wp wp((Dng (Dgma (Dgm)(((Po))) Dgm,, (R(P:)))
Σ
If((et) U Us2)) 9((22o Ubp. .rxl)) P
[iigr ,05a]
Σ
[[[[1gg Dips vi .I) g(vvio War. Ube (1)) D)
Σ
max Jit
|l,i - Fig || - -111yr, - -iill
Σ
11 xz, || 1 1/k P
max 3.l
[Migi Migtil
Σ
Σ
|| - xz. 11 -118re, - 3i
max
[
le I" mln)
Σ
Σ
| xig |1 - 1lurs 33.
max
lE|vig 1m Mieesl
M
|ag - xiel
-1193 - 3n
k
#jEPo
M 2
||ar mill
11ys
34 1P
k
2, EP
M
Wpair (Po, P.)..
For h = 0 the calculations are easier as the vertex values are all 0.
Wp((Dgmo((PP)) (gm <
f[Uus, 1y]) - g(2, u
|| xzll - -11B- 8
< W W.Pair pair (Po. P.)..
16

---

A.4 Efficiency of Proposed Algorithm
As the similar copology-based distillation methods, ours computation is smaller than the TopKD [7]. The main reason is because of the smaller number of persistence homology points u in the persistence homology Dgma 2. ffatd) U A of Eq. The number of w proportional simplices while we can deriviate its upper bound in our method 2 PsHD b - (Mi/h)) and TopKD U HTTopD >E= (A)). where M is the replayed samples in each batch, b is approximate to class number and K is the highest number of distilled simplex. We experimentally set K to be 3, since I-simplex derive when i and dead when i = 3. The upper bound ratio is b2
number n since HP complexity is (
). It is hard to approximate
Proof.
U D.pKoD M M(M- 1)
M(M 1)(M - 2)
2
++ M(M/b 1)
M(Mbb 1)(M/b 2)
6
u3 HoTTopKD
6b + (M - 1)36² + (M 1M(M - 2)6²
u3
6b (M b)3b + (M b)(M 2b)
M.HHD
562 + M26
5b2 M2
b²(b - M, M >> 5).
When the number of old class b raises the continual learning process, relative computational complexity grows quadratically with b. Heavy computation usually accumulates during the whole learning process, so this ratio demonstrating the superiority of our methods.
In addition, we also evaluate the memory storage, model parameters and training time between our methods and ofate-of-the-eee methods Table X When reducing the replayed examplers 2000, our method still achieves the best performance 46.0%. Compared to the best indices, our method lessen the training time by 10.9%, example buffer by 60.9%, and memory size by 28.4%. Therefore, while existing strong baselines excel in different aspects, NNCSL in accuracy, DER_ Fix in training time, and DSGD relatively balanced computation costs and accuracy, our method achieves the overall least resources overhead and the highest accuracy.
Table 8: Effeectiveness and efficiency comparison on CIFAR100_51 The memory size is the sum of the storage of model parameters and replayed examples.
Thain.()
Parameters(m)
Example
Memory Size (MB)
Acc(%)
NNCSL.
12.1
11.8
5120
56.58
46.0
DER Fix
9.1
4.6
5120
30,80
43.61
DSGD
4.6
5120
30,80
44.61
PsHD
14.2
4.6
5120
30.80
47.7
PsHD
8.1
4.6
2000
22.07
46.0
A.5 Visualization of Effectiveness of PsHD
We provide additional visualizations of PsHD in Figure Expect for the issue in Figure the attention bias shows that the initial model fails to recognize the accurate class-specific regions, making it difficult for subsequent models consistently focus on the correct areas. By incorporating persistent homology features, the model is encouraged to direct attention to the appropriate regions.
17

---

TaskO
Taskl
Task2
Task3
Task4
Task5
Task6
Task7
Task8
Task9
Attention shift
(a)
(b)
Attention dacay (b)
(a)
8
Attortion
(3)
deficiencc
(a)
Attention
(b)
Figure 6: Visualization activation heatmap during the continual learning process, where the cate- gories belong TaskO. (a) part correspondence PsHD without the Lua, and (b) part correspondence to PsHD with Led. The red area localizes class-specific discriminative regions.
18

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See Abstract and Section 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section 4.4
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

Justification:
See Appendix A.3.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section Section 4 and Supplemental Material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
See Abstract, Section 4.1 and Supplemental Material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //ippe.cc/pullic/guuii wwwiii/gguie///eeeeeeoecc CoodeSubeiiieeee ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Section 4.1 and Supplemental Material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We share the same settings with compared methods.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
21

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section 4.1 and Supplemental Material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't i make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal
impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinfornation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
22

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The original paper related to the experiment results and data are all cited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. www.com/caao datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
23

---

Researchers should communicate the details of the dataset/codeemmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
24


Reviews:
[{"review_id": "RzyOWkGZ9e", "reviewer": "Reviewer_AVV3", "summary": "This paper proposed to preserve intrinsic structural information with the use of persistent homology, so as to improve knowledge distillation and memory replay in semi-supervised continual learning. The authors provided an efficient acceleration algorithm to reduce computational overheads and theoretically demonstrated its stability. Extensive experiments demonstrate the effectiveness of the proposed method.", "questions": "I think this is essentially a solid paper. My major concerns lie in the effectiveness and efficiency. Please refer to the Weaknesses.", "limitations": "In Checklist the authors claimed that they have discussed the limitations in Section 5. However, Section 5 only presents the Conclusion without any discussion of the limitations.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well organized and easy to follow. The motivation is clearly presented. Most of the related work has been discussed.\n\n2. The design of persistent homology for knowledge distillation is reasonable to me. Although it seems to be an incremental design, the authors have provided many adaptations, such as the acceleration algorithm and the theoretical analysis.\n\n3. The experimental validation is essentially extensive. It covers different setups of semi-supervised continual learning, ablation study, visualization, etc.", "weaknesses": "1. Although the authors have provided extensive experiments, the effectiveness of the proposed method seems to be limited. It offers less than 1% improvements in a majority of cases with relatively simple datasets (e.g., CIFAR-10 and CIFAR-100). This may be due to the classic NP-hard problem in selecting a few data to represent the entire dataset.\n\n2. Although the authors have provided an acceleration algorithm to reduce computational overheads and conceptually analyzed it with their own method, I suggest to compare the overall resource overheads (storage and computation) of their method with other strong baselines (e.g., NNCSL, DER_Fix, and DSGD).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**A1**. The limited replied 500 samples restrict the improvement space, while our method reduces the degree of forgetting (BWT) by a substantial and balanced margin compared to the newly strong methods. Furthermore, our methods demonstrate effective utilization of replayed unlabeled samples, as evidenced by the substantial improvements over strong baselines when the memory buffer size is increased. \n\n**(1) Lower forgetting degree with comparable incremental accuracy.** As shown in Table 1 of the attached PDF file, our method surpasses the NNCSL by 6.15\\% with a decreased average BWT of 18\\%, outperforms DER\\_Fix by 1.63\\% with a decreased average BWT of 6.52\\% forgetting on average, and exceeds DSGD by 0.89\\% with a decreased average BWT rate 3.35\\% in the four benchmarks. \n\n**(2) More balanced forgetting degree.** The BWT our method is under 27 in these benchmarks, avoiding the high forgetting rate observed with NNCSL. The overall BWT is lower than state-of-the-art with even higher incremental accuracy. These results show that our method PsHD achieves a better trade-off between adapting old tasks and preserving new tasks. \n\n**(3) Sufficient utilization of replayed unlabeled samples.** As shown in Table 1, when the number of replayed sampled increases to 5120, our method leverages the topology structure information of the unlabeled data more effectively, surpassing NNCSL and DSGD by 3.28\\% and 2.07\\% on average across the four benchmarks, respectively. \n\n**Table 1. Comparison on CIFAR10 and CIFAR100 with 5120 examples replayed**\n| Methods | CIFAR10_5% | CIFAR10_25% | CIFAR100_5% | CIFAR100_25% |\n|-------|------------|-------------|-------------|--------------|\n| Ours  | 82.5       | 87.2        | 47.5        | 58.6         |\n| NNCSL | 79.3       | 81.0        | 46.0        | 56.4         |\n| Compare_Ours | **+3.2**   | **+6.2**        | **+1.5**        | **+2.2**         |\n| DER_FIX | 80.3      | 78.2        | 43.6        | 56.3         |\n| DSGD  | 81.2       | 84.4        | 44.6        | 57.3         |\n| Compare_Ours | **+1.3**  | **+2.8**       | **+2.9**       | **+1.3**        |\n\n**A2.** Our method exhibits better performance, fewer memory budgets, and lower computation costs compared to these strong baselines. We are sorry for not fully expressing these advantages. The detailed verification is illustrated as follows.\n\n**(1) Highest accuracy with aligned memory size.** To verify the effectiveness and efficiency of our PsHD, we provide a comprehensive comparison of the overall resource overheads in Table 2, including the computation time and storage costs. The experiments are conducted on the same server for a fair comparison. Referring to the storage costs with aligned example 5120, our method achieves the highest accuracy at 47.5\\% with the aligned memory size of 30.8 MB. It also surpasses the NNCSL by 1.5\\% while using 45\\% less memory, due to our smaller model buffer. \n\n**(2) Optimal solution in terms of both effectiveness and efficiency.** When reducing the example size to 2000, our method still achieves the best performance 46.0\\% while reducing training time by 35.2\\%, examples buffer size by 60.9\\%, and memory size by 28.4\\% compared to the best indices, which do not belong to the same method. Therefore, while existing strong baselines excel in different aspects —NNCSL in accuracy, DER\\_Fix in training time, and DSGD is relatively balanced computation costs and accuracy, our method achieves the overall state-of-the-art performance with the least resource overheads and the highest accuracy.  \n\n\n**Table 2. Effectiveness and efficiency comparison on CIFAR100_5%. The memory size is the sum of the storage of model parameters and replayed examples.**\n\n| Methods | Training time(h) | Parameters(m) | Example | Memory size (MB) | Acc(\\%) |\n|-------|-----------|---------------|---------|------------------|---------|\n| NNCSL | 67        | 11.8          | 5120    | 56.58            | 46.01    |\n| DER_Fix | 34.1    | 4.6           | 5120    | 30.80            | 43.61   |\n| DSGD  | 40.8      | 4.6           | 5120    | 30.80            | 44.61   |\n| PsHD  | 40.4      | 4.6           | 5120    | 30.80            | **47.50** |\n| PsHD  | **22.1**  | 4.6           | **2000**| **22.07** |     _46.02_  |\n\n\nWe thank the reviewer for the insightful suggestion to highlight the efficiency comparison,  and we have reorganized our expression  to highlight the superiority of the proposed method.\n\n**A3.** One limitation of the proposed method is that the partial computation of persistence homology is conducted on the CPU, which increases training time. We have include this limitation in the Conclusion. \n\n**Further improvement plan on limitation.** Although our acceleration algorithm mitigates this issue without disrupting end-to-end training, there is still room for improvement. The primary reason for this limitation is that the Python package Gudhi has not been developed for GPU use. To address this problem, we plan to implement persistence homology computation on the GPU, which is expected to further reduce training time.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GhjLO4HEWd", "reviewer": "Reviewer_K7Tp", "summary": "The paper proposes a persistence homology knowledge distillation for continual learning (PsHD). PsHD loss is calculated using a ''memory buffer'' between a previous variant of a network and a new one. Experiments show some improvement w.r.t. baselines. Ablation studies are provided.\nThe main issue of the paper is limited novelty, also some essential details are missing (see below).", "questions": "1. How do you differentiate PsHD?\n2. You claim that PsHD helps to handle noise, but there is no noise in experimental settings.\nThe claim \"however, traditional distillation strategies often fail in unlabeled data with inaccurate or noisy information\" is not proved also. \n3. What is the difference between a memory buffer and unlabeled data? To my understanding, for PsHD loss one only needs unlabeled data.\n4. Is the SSL part of loss a critical part of your method? Is seems to be completely independent of the main idea (PsHD loss) and, probably, gives to your method an artificial advantage w.r.t. others.\n\n**Other**\n\n1. In Figure 2, two diagrams seem to be identical, while the top simplicial complex has one hole, but the bottom one - two holes.\n2. The Wasserstein-p distance (line 164) equation is not correct, because the subscript $\\infty$ relates to Wasserstein-$\\infty$ distance.\n3. line 214: typo CIAFR", "limitations": "-", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Novelty: this is a first application of persistent homology to continual learning.\n2. Experiments are correct, ablation studies are provided.\n3. The manuscript is well organized, the idea is easily comprehensible.\n4. Visualizing of attentions maps helps to reveal how the method improves continual learning.", "weaknesses": "1. The idea that knowledge distillation can help continual learning is not new, see [2]. Given this, the novelty of the paper is small.\nDo you have an ablation with a traditional KD [1] and its more recent variants from [2]?\n2. A relevant reference [3] is missing. How you paper is related to [3], is your method better?\n3. I can't find an explicit equation for $L_{CL}$.\n\n[1] Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.\n[2] Li, S., Su, T., Zhang, X., & Wang, Z. (2024). Continual Learning with Knowledge Distillation: A Survey. Authorea Preprints.\n[3] Kim, J., You, J., Lee, D., Kim, H. Y., & Jung, J. H. Do Topological Characteristics Help in Knowledge Distillation?. In Forty-first International Conference on Machine Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**A1.** Traditional CL methods are not always effective for unlabeled data, as they assume the knowledge of previous models is accurate. This specific challenge of SSCL has been explored in previous SSCL methods (NNCSL, DSGD, etc). As suggested by the reviewer, our ablation experiments provide additional verification of this motivation for overcoming catastrophic forgetting of unlabeled data. \n\n**(1)\tNegative influence with Traditional KD [1] on SSCL.**  The performance decreases when employing traditional KD [1] directly on SSCL, which is the iCaRL method in our comparison method. As shown in Table 2 of PDF file, the introduction of logits distillation on unlabeled data (iCarL\\_U) leads to 4.97\\% decrease on average compared to distillation only on labeled data (iCaRL). We also conduct pseudo-label distillation on unlabeled data based on DER, resulting in 5.23\\% accuracy decrease (DER\\_U) on average. This indicates that inaccurate representation preservation has a negative effect on SSCL. \n\n**(2) Diminished effectiveness of KD variants from [2] for SSCL.** Table 3 of PDF file provides comprehensive ablation experiments of traditional KD [1] and its more recent variants from [2] on our SSCL problem. The effectiveness of logits and feature distillation diminishes, especially as tasks become harder and more numerous, such as in CIFAR100\\_5\\%.  \nAlthough relation and topology distillation are relatively effective for SSCL, their stability, computation costs, and accuracy require better balancing.\n\nGiven the limitations of existing KD methods in SSCL, an adaptive knowledge distillation strategy for SSCL should be further proposed.\n\n**A2.** We acknowledge we didn't seen the reference [3] TopKD, so the citation and comapration are missed even our method is better. The idea of topology feature distillation is similar while ours requires less computation and is more adaptive to SSCL.\n\n**(1) Lower Computation Costs.** Heavy computation usually accumulates during the whole learning process.  Our lower superior computation cost is mainly because of  smaller number of persistence homology points in the PH, which is $u$ in Equation (5). The number of $u$ is proportional to simplices number $n$ since HP complexity is $\\mathcal{O}(n^{2.5})$. $n$ is hard to approximate, while we can compute the its upper bound in our PsHD $\\sum_{t=0}^{M/b}\\binom{M/b}{t}$ and TopKD $\\sum_{t=0}^{M}\\binom{M}{t}$, where $M$ is the replayed samples in each batch and $b$ is approximate to class number. The ratio is $2^{M-M/B}$, indicating a significant reduction of distillation computation. \n\n**(2) Better Adaptability to Unlabeled Data.**  As shown in Table 3 of PDF file, applying TopKD to CIFAR10\\_5\\% based on iCaRL results in a decrease in accuracy from 79.2\\% to 78.7\\%, indicating TopKD's reliance on accurate representations. In contrast, our method improves iCaRL and outperforms TopKD by an average of 2.28\\% across four benchmarks. Furthermore, our training time on CIFAR10\\_5\\% is 7.1 hours, compared to TopKD's 16.5 hours, reducing training time by 56.9\\%. \n\n**A3.** $L_{Cl}$ is the continual learning loss, which corresponds to the persistence homology distillation loss in our method. It varies among different types of CL methods. We will change $ L\\_{Cl}$  to $L\\_{hd}$  to avoid confusion.  \n\n**A4.** According to the loss of Equation (5), PsHD is back propagated along with the persistence point $u$ in the persistence diagram. So the gradient backpropagation flow is: \n$L_{hd}\\rightarrow u=[u_s,u_e]\\rightarrow[dia(\\sigma_s), dia(\\sigma_e)]\\rightarrow [dia(v_1,v_2), dia(v_1,v_2,v_3)]$. \n\n**(1) Illustration of differentiating PsHD.** $u=[u_s,u_e]$ represents a lifespan of one simplex $\\sigma_s=\\\\{v_1,v_2\\\\}$ exists in $u_s=dia(\\sigma_s)$ and disappear or fused to higher dimension simplex $\\sigma_e=\\\\{v_1,v_2,v_3\\\\}$ in $u_e=dia(\\sigma_e)$. $dia(\\sigma)$ is the diameter of simplex $\\sigma$, which is the maximum distance of any two samples in $\\sigma$, and $v_i$ represents the sample. The loss is finally optimized to update the representation.\n\n**(2) End-to-End training of our PsHD.** The origin Gudhi hasn’t been developed on pytorch, so the end-to-end training is unprocurable directly. Our method achieves this progress by making the differentiation on PsHD along the chosen samples. \n\n**A5.** We supplement further ablation experiments on noise interference. As shown in Table 4 of attached PDF file, we add Gaussian noise with standard deriviation \\{0.2,1,1.2\\} on 5 distillation-based continual learning methods iCaRL, Foster, Podnet, LUCIR, and R-DFCIL. It can be seen that our method is stable on the forgetting degree and accuracy despite a higher noisy interpretation, which is consistent with the conclusion of Theorem 1. \n\n**A6.** The unlabeled data is part of the memory buffer, with the remaining data being labeled. PsHD loss is applied only to the unlabeled data, while labeled data is distilled through the traditional KD. Thus, our method is compatible with the traditional KD.  \n\n**Necessity of replaying both labeled and unlabeled data.** Labeled data provides accurate representations for discrimination, while unlabeled data has the potential to represent the entire dataset. Traditional knowledge distillation strategies are usually efficient in labeled data. Our PsHD loss is applied only to the unlabeled data, reserving intrinsic information and reducing computation costs, as illustrated in Answer 2. \n\n**A7.** The semi-supervised loss (SSL) used in our method follows the approach from DSGD, which is based on the SSL method Fixmatch. The SSL loss is not the primary motivation for our PsHD, as its effects in SSCL have already been explored in DSGD. We will clarify this point in the  Section 3.2. \n\n**A8-10.** The simplicial complex corresponding to the diagrams in Figure 2 is simplified, which leads to persistence feature inconsistency. We will update this figure to ensure greater consistency. We will correct the two typo issues.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j52V3k6BCw", "reviewer": "Reviewer_2waA", "summary": "The paper proposes a new method PsHD to preserve intrinsic structural information in semi-supervised continual learning. The method proposes to uses distillation and cross-entropy loss on the continual learning samples.", "questions": "Please in weakness", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1. I think the paper presents quite comprehensive experiments with different settings. In all of the experiments, the work demonstrate at least marginal improvement compared to previous methods\n2. I think figure.3 looks very interesting and seems to justify the paper's motivation.", "weaknesses": "1. I think the paper can be further polished in terms of writing, presentation and clarity. For example. I think Figure.4 can be further improved for clarity. \n2. The results raise some concerns regarding the proposed method. In Table 4, we observe that changing the data allocation ratio results in performance change within 1.5% for 5% and within 1% for 25% setting. Also in Table.4, optimal data ratio are different for the two settings. We also observe in Table 1 that the proposed method leads the previous methods by less than 1%. It makes the author's claim for their ``superiority'' performance quite unsupported. \n3. This brings my third concern to the work: Some language in the paper seems overclaimed. The paper tried to adopt many large words like ``significant'' and ``superior'' while the performance does not support such claims.\n4. In Figure.4, larger lambda seem to help the model performance, then why not train with larger lambda? Maybe there is an elbow effect but there needs more experiment to show that.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**A1.** We will make substantial revisions to enhance the overall clarity and readability: \n\n**(1) Provide detailed results to verify the description of our advantages.** As shown in Table 1. of attached PDF file, we provide additional comparison of the forgetting rate  (BWT) among strong baselines and our method to demonstrate its effectiveness in alliviate catastrophic forgetting. Our method achieves 3.4% forgetting mitigation with 0.9% accuracy increasement on average  compared with DSGD, and  18\\% forgetting mitigation with 6.15\\% accuracy increasement on average compared to NNCSL. Moreover, the effeciency comparison is also provided in Table 2, and the stability contrast to noise interference is supplemented in Table 4. \n\n**(2) Redesigned Figure 4 to make it more informative and readable.** As shown in Figure 1 of the attached PDF file, we conducted additional ablation experiments on larger distillation loss weight $\\lambda$, specifically  {1.2, 1.5, 2} to further verify the effect of proposed homology distillation loss in overcoming catastrophic forgetting. We selecte the optimal  weights $\\lambda=1$ for the three dataset that  balance the forgetting (BWT) and average incremental accuracy (Avg) best.  Additionally, we expand the ablation study of persistence homology dimension to determine appropriate homology featuresfor distillation, using 0-simplex for the simple dataset CIFAR10, and 0,1-simplex for complex data, such as CIFAR100 and ImageNet-100. Lastly, we added legends to each diagram for better readability.\n\n\n**A2.** \nSelecting an optimal ratio of labeled to unlabeled data in a limited memory buffer to represent an entire dataset is an open problem in semi-supervised continual learning with different degrees of supervision. We firstly offer experimentally valid allocation ratios for several benchmarks. \nRegarding the limited improvements of Table 1, we enhance the results by providing the degree of forgetting (BWT). We further provide evidence of the efficiency and stabilization of PsHD compared to newly strong baselines. \n\n\n**(1) First consideration of ideal allocation labeled vs. unlabeled ratio in SSCL.** Our experiments indicate that indiscriminate utilization of unlabeled data can diminish accuracy.  As shown in Table 2 of the PDF file, the introduction of logits distillation on unlabeled data (iCarL_U) leads to 4.97% decrease on average compared to distillation only on labeled data (iCaRL). These diminishes are different across supervision ratios, since higher supervision can ensure confident representation of unlabeled data. Therefore, the ideal allocation ratio remains unresolved in existing SSCL methods. We provide experimentally valid allocation ratios for labeled data allocation ratios between 0.6 and 0.9 for a general reproducibility and transferability.\n\n**(2) Enhanced forgetting rate in Table 1 and better knowledge distillation on unlabeled data.**  As shown in Table 1 of the attached PDF file, our method surpasses the NNCSL by 6.15\\% with 18\\% forgetting on average, outperforms DER\\_Fix by 1.63\\% with 6.52\\% forgetting on average, and exceeds DSGD by 0.89\\% with 3.35\\% forgetting on average. \nMoreover, our methods exhibit better adaption in distillation knowledge in unlabeled data, achieving a 35.2\\% reduction in training time, a 60.9\\% reduction in examples buffer, and a 28.4\\% memory size reduction compared to the best indexes in CIFAR100\\_5\\%. \n\n**(3) More stable distillation for SSCL.** The training stability of our method can be verified through comprehensive ablation experiments. As shown in Table 4 of the attached PDF file, we add Gaussian noise with weight \\{0.2,1,1.2\\} to five typical distillation-based continual learning methods: iCaRL, Foster, Podnet, LUCIR, and R-DFCIL. Our method demonstrates stability in both the degree of forgetting and accuracy, even under higher noise levels. These results are consistent with our Theorem 1.\n\nTo sum up, our method PsHD effectively mitigates catastrophic forgetting of unlabeled data and achieves a better trade-off between adapting old tasks and preserving new tasks in SSCL.\n\n**A3.** We include more detailed experimental results to emphasize the strengths of our method and validate our conclusions. Additionally, to ensure rigor, we will also improve the modifier of experiment results and illustrate the limitations at the same time. Such as use 'comparable' or 'competitive' to illustrate  the accuracy improvement less than 1%, and clarify the limited improvement on 500 replayed samples. \n\n\n**A4.** We supplement more experiments to validate the effect of distillation weight lambda value and persistence dimension H0 and present the results in Figure 1 of the attached PDF file. Higher distillation weight results in lower forgetting (BWT) and higher average incremental accuracy (Avg). The optimal persistence homology distillation weights are also highlighted in each benchmark. Additionally, we improve the arrangement of Figure 4 for better readability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for recognizing novelty (AVV3, K7Tp, 2waA), well-organization (AVV3, K7Tp), significance and reproducibility (AVV3,  K7Tp, 2waA), good performance (AVV3, K7Tp, 2waA), and comprehensive comparisons (AVV3, K7Tp, 2waA) of proposed PsHD. 
 
**Reviewer AVV3's Questions and Our Responses:**

(1) For the question of  limited performance in Table 1, we providd additional comparison on the degree of forgetting (BWT) in Table 1 of attached PDF file, validating the superiority of proposed method on overcoming catastrophic forgetting. 

(2) For the question of effectiveness and efficiency demonstration, we supplement detailed verification results in Table 2 of AVV3 rebuttal area. 

(3) For the question of limitation clarficaiton, we clarified the CPU computation limitation and propose a further improvement strategy. 

**Reviewer K7Tp's Questions and Our Responses:**

(1) For  the results of  traditional KD [1] and its varients from [2] employed on SSCL, we provide the evidence of negative influence of traditional KD [1] on SSCL in Table 2 of attached PDF file, and include experiment comparison of representative knowledge distillation strateties in [2] on the same benchmark, with results shown in Table 3 of the attached PDF file.


(2) For the missed comparison of reference [3], we acknolwedge the issue since its the same period work. We also conduct comparison experiment on four benchmarks in Table 3 of attached PDF file. 

(3) For the stability evaluation of our methods, we add Gaussian noise interference experiments in Table 4 of attached PDF file. 

(4) For more specific technique questions, such as how differentiate  PsHD, we also clarify the details in the K7Tp rebuttal area. 

**Reviewer 2waA 's Questions and Our Responses:**

(1) For the unsufficient evidence for supporting the superiority, we supplement more experiments results to valid the conclusion from Table 1-4 of attached PDF file. 

(2) For the unclear presentation of Figure 4, we conducte additional ablation experiments on larger distillation loss weight $\lambda$ and  add legends on each diagram for readability, the improved figure is depicted in Figure 1 of  attached PDF file.

(3) For the language problem, we improve the modifier of experiment results and illustrate the limitations at the same time. Such as use 'comparable' or 'competitive' to illustrate the accuracy improvement less than 1\%.

Reference

[1] Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
 
[2] Li, S., Su, T., Zhang, X., & Wang, Z. (2024). Continual Learning with Knowledge Distillation: A Survey. Authorea Preprints. 

[3] Kim, J., You, J., Lee, D., Kim, H. Y., & Jung, J. H. Do Topological Characteristics Help in Knowledge Distillation?. In Forty-first International Conference on Machine Learning.

Meta-review: The paper leverages persistent homology to enhance knowledge distillation and memory replay in semi-supervised continual learning, introducing an efficient algorithm to reduce overhead and demonstrating its stability and effectiveness through experiments. However, the improvement is quite incremental in many cases.

This paper is borderline for acceptance. After the rebuttal, all reviewers have given a score of 5. The previously raised issues regarding experiments and unclear concepts have been adequately addressed. To make a final decision, I carefully reviewed the paper, initial reviews, rebuttal, and all discussions. My assessment aligns with the reviewers' opinions: after revisions, the paper does not contain any obvious errors or major weaknesses, but the overall experimental impact and contribution to the field remain limited. 

I am currently leaning toward acceptance but will discuss further with the SAC.
Decision: Accept (poster)