Persistence Homology Distillation for Semi-supervised Continual Learning

Authors: yan  fan
yu  wang
pengfei  zhu
dongyue  chen
qinghua  hu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 18:24:38

Abstract: Semi-supervised continual learning (SSCL) has attracted significant attention for addressing catastrophic forgetting in semi-supervised data. Knowledge distillation, which leverages data representation and pair-wise similarity, has shown significant potential in preserving information in SSCL. However, traditional distillation strategies often fail in unlabeled data with inaccurate or noisy information, limiting their efficiency in feature spaces undergoing substantial changes during continual learning. To address these limitations, we propose Persistence Homology Distillation (PsHD) to preserve intrinsic structural information that is insensitive to noise in semi-supervised continual learning. First, we capture the structural features using persistence homology by homological evolution across different scales in vision data, where the multi-scale characteristic established its stability under noise interference. Next, we propose a persistence homology distillation loss in SSCL and design an acceleration algorithm to reduce the computational cost of persistence homology in our module. Furthermore, we demonstrate the superior stability of PsHD compared to sample representation and pair-wise similarity distillation methods theoretically and experimentally. Finally, experimental results on three widely used datasets validate that the new PsHD outperforms state-of-the-art with 3.9% improvements on average, and also achieves 1.5%  improvements while reducing 60% memory buffer size, highlighting the potential of utilizing unlabeled data in SSCL. Our code is available: https://github.com/fanyan0411/PsHD.

Main paper: 
 introduction: 
 Continual learning involves developing practical approaches for incrementally training models, enabling them to learn new concepts while effectively retaining previously acquired knowledge [1] . The problem of catastrophic forgetting in continual learning impacts the model's performance on previously learned tasks. Numerous efforts have been dedicated to addressing this issue in continual learning [2] [3] [4] [5] [6] [7] [8] . However, most state-of-the-art methods are trained under the assumption of fully labeled data, neglecting the vast amounts of unlabeled data.Semi-supervised continual learning introduces unlabeled data in CL by partially annotating the trained data in each task and aims to alleviate catastrophic forgetting by exploiting these unlabeled data [9] [10] [11] [12] [13] [14] . Learning from unlabeled data to improve model generalizability across tasks is a natural approach, but its performance significantly relies on the size of the memory buffer [12, 15] . Generative replay methods reduce the need for a memory buffer but incur prohibitive costs for higher resolution images and are limited to relatively simple datasets [11, 12] . Knowledge distillation methods are proposed to Figure 1 : The performance interference of extra unlabeled data distillation. The four approaches are (a) iCaRL, (b) PodNet, (c) R-DFCIL and (d) our persistence homology distillation methods. Avg and Avg_old mean the average incremental accuracy of all tasks and old tasks, respectively. preserve valuable information in the unlabeled data of the previous model, referred to as the teacher model [13, 14] .Logits or feature distillation approaches focus on sample-wise representation [9] . Relation distillation methods, as variants of representation distillation, are explored for vulnerable representation of unlabeled data to enhance more expressive knowledge preservation, including class-instance similarity [13] , and local structural similarity [14] . Although these distillation strategies have proven effective, the impact of noisy relations remains a significant challenge. As shown in Figure 1 , the succeeded biased or inaccurate information often nterferes with the generalization on known classes impact and even impact the entire tasks [16] . Therefore, we argue that a more powerful distillation that investigates noise insensitive information preservation in unlabeled data is needed for semi-supervised learning continual learning.To address the aforementioned problem, we explore extracting multi-scale shape and structural information from unlabeled data that is insensitive to feature space bias and noise interference in semi-supervised continual learning. Inspired by the manifold assumption [17] and the homomorphism between manifolds and simplicial complexes [18] , we explore summarizing complex vision data into simplicial complexes, and investigate the preservation of multi-scale structural properties despite representation changes. Persistence homology in topological data analysis, as an adaptation of homology to discrete metric spaces, reflects the homology-class evolution across different scales. Its characteristics of invariants on homeomorphism and robustness to perturbations provide an adaptable view to explore the structure in unlabeled data.To this end, we propose a new model named Persistence Homology Distillation (PsHD) for semisupervised continual learning to stabilize learned tasks by leveraging the power of topological features. Firstly, we construct local simplicial complexes in vision data based on the weighted k-hop neighborhood and extract multi-scale structural features using persistent homology. Furthermore, we devise a model parameter-independent distillation loss based on these structural features and also design an acceleration algorithm to mitigate the computation costs. We also demonstrate the stability privilege of persistence homology distillation compared to pair-wise similarity distillation strategies [13, 19, 20] theoretically and experimentally. The primary contributions of our work are as follows:• We utilize simplicial complexes to approximate vision data and explore stable topological feature representation of unlabeled data in semi-supervised learning.• We propose a novel persistence homology distillation strategy for SSCL that is insensitive to noise information interference based on the intrinsic topological features, and devise an accelerating algorithm to reduce computation costs.• We demonstrate that our method outperforms existing methods on several benchmarks, and highlights the potential of utilizing unlabeled data to overcome catastrophic forgetting. 
related work: 
 Persistence Homology Topological data analysis (TDA) is a new approach to seeking structural or geometric information from discrete data. Persistent homology (PH) is one of the most popular tools in TDA to capture topological information about connectivity and holes in geometric data. Several impressive applications of persistent homology in complex data are explored [21] [22] [23] [24] [25] .Topological features in vision data have been investigated broadly, including leveraging the prior topology knowledge to improve the segmentation results [26, 27] and exploiting high-order persistence spectral representations in image embedding [27] . These works are all based on explicit object-level topology features. Few insights about TDA in category-level vision data have been investigated. Liu et al. [28] propose a topological prediction model in image dataset, while is not learnable to meet the requirement of deep learning. Madhu and Chepuri [29] introduce the simplicial contrastive loss to preserve more expressive consistency, while only considering neighbors' aggregated representation. So far, there are limited results on learnable intrinsic category features to summarize vision data and investigate the structural stabilization of data distribution.Continual Learning Continual learning methods can be categorized into three groups. The regularization-based methods selectively regularizing the variation of essential network parameters [2, 3] or preserve the outputs consistency of prediction functions [30, 31] . Replay-based methods store few representative samples for retraining or old knowledge distillation [4] [5] [6] , or utilize an additional generative model to replay previous data [32, 33] . The parameter isolation methods isolate parameter subspaces dedicated to each task throughout the network [7, 8] .To address the limitation of continual learning in neglecting tremendous unlabeled data, recent efforts have been directed towards semi-supervised continual learning [9, 12, 11, 15, 13] . Some of these methods leverage the generalizability of unlabeled data to alleviate forgetting by incorporating gradients of unlabeled data [12] or employing contrastive loss among unlabeled data [15] . Such strategies require a substantial memory buffer to ensure satisfactory performance. To deal with the memory limitation, a generative reply method regularizes discriminator consistency to mitigate catastrophic forgetting [11] . Employing meta distribution to consolidate the task-specific knowledge [34] and training an adversarial auto-encoder to reconstruct images [35] extends [11] further. However, these methods incur prohibitive costs on higher-resolution images. Knowledge distillation strategies have also been explored on unlabeled data to address the forgetting in SSCL, including sample-wise representation [9] , class-instance relationships [13] , and local-neighborhood similarity [14] .In continual learning scenarios with limited labeled data, the required changes of feature space can often be dramatic [1] , and the learned class-related representation is usually biased to labeled data [13] .With the unreliable distribution, stabilizing the feature extractor may interfere with accommodating new representations and subsequent task refinement. A more stable representation learning robust to data transformation is required for the SSCL.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: See Section 4.4Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries on persistence homology: 
 A k-simplex σ is a k-dimensional polytope which is the convex hull of its k+1 vertexes. A single point can be regarded as a 0-simplex, and a line segment can be defined as a 1-simplex. A 2-simplex is a triangle, and a 3-simplex can be seen as a tetrahedron. The convex hull of any nonempty subset of the k+1 points that define a k-simplex is called a face of the simplex. A simplicial complex K is a set of simplex that satisfies the following conditions. (1) Any face of a simplex from K is also in K.(2) The intersection of any two simplices σ 1 , σ 2 ∈ K is a face of both σ 1 and σ 2 . A common choice of simplicial complex K r is the Vietoris-Rips simplicial complex R(X, r), in which a simplex σ = {x 1 , x 2 , . . . , x m } ∈ R(X, r) when dist(x i , x j ) < r for all 1 ≤ i, j ≤ m.A filtration {K r } r∈R of a point cloud in R d can be constructed from any filter function f : R d → R by considering each simplicial complex to be the sublevel set of f threshold by r, K r = {x ∈ R d |f (x) ≤ r}. The underlying filter function introduced above is the distance function δ X : R d → R, where δ X (y) = min{dist(y, x)|x ∈ X} is the distance from y to point set X. To be specific, the Vietoris-Rips simplicial complex R(X, r) approximates the sublevel set where B(x, r) is a ball with radius centered around x ∈ X. Then the filtration is a nested sequence of subcomplex setEQUATIONEQUATIONwhich reflect multi-scale topological spaces underlying the point data, as shown in Figure 2 (a). The persistence homology quantifies the changes in topological features as the subcomplexes grow with the increasing threshold of the filter function. The strict definition of persistence homology is provided in Appendix A.1. An easier way to understand is as follows. The lifetime of each multi-scale topological feature is recorded by persistence barcode, as shown in Figure 2 (b). Finally, the persistence barcode can be converted into a persistence diagram, as shown in Figure 2(c) .Dgm = {(x σ , y σ ) ∈ R 2 :x σ < y σ } ∪ △ are a scatter plot with the x and y axes respectively depicting the scale x σ at which each topological feature is born one subcomplex and dies in threshold y σ or is identified with another homology class. △ is the diagonal set containing points in PH.Formulation of SSCL As shown in Figure 3 (a), when encountering new data D t , let f t θ be the model to be trained, with both labeled data L and unlabeled data U input into the model. The parameters θ are updated through the cross-entropy loss L CE and semi-supervised loss L SSL to adapt the model to the new task. Simultaneously, to prevent catastrophic forgetting, the memory buffer M is used to regularize the knowledge consistency between DK(h t ) and DK(h t-1 ) based on the previous model f t-1 θ through a continual learning loss L CL . The complete loss function for SSCL is as follows:EQUATIONwhere we follow DSGD [14] to employ Fixmatch [36] as the semi-supervised loss L SSL . Now we propose a new persistence homology distillation loss L hd as L CL .Persistence Homology Distillation Existing distillation methods preserve the sample representation, the class-instance similarity between tasks, and local neighbor similarity. This limits the performance when inaccurate representation is inherited or noise occurs in previous tasks, especially when a significant transfer between the feature spaces of two tasks exists. To exploit the intrinsic structural features preservation, which topological invariant regardless of specific representation, we propose a persistence homology-based distillation method and design an effective algorithm to achieve stable knowledge accumulation.The first step of our model is to define the topological similarity among two samples. Given a collection of image sets, we build the local persistence diagram by considering the weighted k-hop neighborhood N (x i , k). To be specific, let G = (V, E) be the adjacency graph of memory buffer, where V is the set of replayed image samples. W ∈ R N ×N be the adjacency matrix of G such that W ij = 1 for e ij ∈ E and 0 for otherwise. As a discrete point data, we construct the adjacency matrix by applying a threshold to the cosine similarityEQUATIONwhere h i , h j are the representations of sample x i and x j , respectively. We use the logits h i ∈ R d in this study. Next, we can group the weighted k-hop neighborhood set N (x i , k), where for any x j ∈ N (x i , k), the shortest path between x i and x j is at most k. Armed with the distance function dist(x i , x j ) = 1 -s ij , we can construct the h-dimensional persistence diagram Dgm h (N (x i , k)) by considering the distance function as the filtration function according to Eq (1)- (2) .Subsequently, we try to preserve this structural information by designing a distillation loss, which goes beyond only the pairwise similarity of node features. Given any sample x of the memory buffer, we can derive the persistence diagrams of the old task by the old feature extractor Dgm h (N k x , f old ). The corresponding persistence diagram in the new model is Dgm h (N kx , f new ), where the neighborhood set is the same as the previous task. The structural similarity between two tasks is measured with the p-Wasserstein distance between the corresponding persistence diagrams asEQUATIONwhere p ≥ 1 and γ is taken over all bijective maps between Dgm h (N k x , f old ) and Dgm h (N k x , f new ). In our analysis p=1. This distance quantifies how much effort is needed to transform one topological feature representation into another.Persistence Homology Distillation Acceleration Algorithm It is known that the worst-case complexity of generating a homology diagram is O(n 2.5 ), where n is the number of the simplices [37] . Thus, the complexity of computing local topology similarity for each sample in the memory buffer is O(|M |n 2.5 ), which is computationally expensive. This procedure is also redundant since nearby samples typically share similar topological structures. To mitigate the computation cost, we design an approximate algorithm to simplify the final distillation loss and accelerate the persistence homology distillation procedure.Given the replayed data M , we randomly select one sample x 1 and group its weighted k-hop neighborhood set N (x 1 , k). We then select another sample x 2 in the remaining samples M \N (x 1 , k), and group its weighted k-hop neighborhood set N (x 2 , k). This process is repeated until all samples are grouped into their respective neighborhood sets. After this step, we can compute several homology diagrams derived from grouped k-hop neighborhood sets {N (x 1 , k), . . . , N (x l , k)}. We design the final distillation loss as L hd in SSCL by meaning of all separated local homology diagramsEQUATIONwhere | • | is the cardinality. This procedure is summarized as Algorithm 1 in Appendix A.2. By doing so, the complexity is reduced from O(|M |n 2.5 ) to O(|S|n 2.5 ). We experimentally find that the size of S is approximate to the number of classes in the memory buffer, so the O(|S|n 2.5 ) is approximated to be O(C M n 2.5 ). This contributes to a significant reduction in computation, approximately 10-fold when 1000 unlabeled samples are replayed in CIFAR-100.The superiority of our proposed persistence homology distillation lies in its insensitivity to noisy and inaccurate representation. According to the stability of persistence homology [38] , we can infer that persistence homology distillation is inherently more robust than feature distillation. We also demonstrate its robustness compared to the similarity distillation in Theorem 1.Theorem 1. Given the two feature space of point data as P 0 and P 1 , for all p ≥ 1 and h ∈ Z + , we haveEQUATIONwhere Dgm h (R(P 0 )) and Dgm h (R(P 1 )) are the h-dimensional persistence diagram for the Vietoris-Rips filtration on the point set P 0 and P 1 respectively. W pair p represents the pair-wise distance between the point set W pair p(P 0 , P 1 ) = inf ϕ ( u,v∈P0 | ∥v -u∥ -∥ϕ(v) -ϕ(u)∥ | p ) 1 p), where ϕ is a bijection between the two set P 0 and P 1 .The proof is shown in Appendix A.3. We consider the features of point data as the extracted embedding through our backbone network. This theorem demonstrates that the persistence homology is robust to the perturbations of similarity between samples and inaccurate sample-wise representation.Theorem 1. Given the two feature space of point data as P 0 and P 1 , for all p ≥ 1 and h ∈ Z + , we haveEQUATIONwhere Dgm h (R(P 0 )) and Dgm h (R(P 1 )) are the h-dimensional persistence diagram for the Vietoris-Rips filtration on the point set P 0 and P 1 respectively. W pair p represents the pair-wise distance between the point set W pair p(P 0 , P 1 ) = inf ϕ ( u,v∈P0 | ∥v -u∥ -∥ϕ(v) -ϕ(u)∥ | p ) 1 p), where ψ is a bijection within the same set, such as P 0 or P 1 , and ϕ is a bijection between the two set P 0 and P 1 .Proof. Let ϕ : P 0 → P 1 be the bijection which achieves the minimum ofW pair p (P 0 , P 1 ) = inf ϕ ( u,v∈P0 ∥u -v∥ -∥ϕ(u) -ϕ(v)∥ p ) 1 pRelabel the points in P 0 = {x 1 , . . . , x M } and P 1 = {y 1 , . . . , y M } so that ϕ(x i ) = y 1 . Let K be the complete simplicial complex on M vertices {v1 , • • • , v M }. Define functions f, g : K → R by f ([v i0 , v i1 , • • • , v i k ]) the time when [x i0 , x i1 , • • • , x i k ] is included in R(P 0 ) and g([v i0 , v i1 , • • • , v i k ]) the time when [y i0 , y i1 , • • • , y i k ] is included in R(P 1 ). Suppose h ≥ 1, then |f ([v i0 , v i1 , • • • , v i k ]) -g([v i0 , v i1 , • • • , v i k ])| = (max j,l { x ij -x i l } -max j,l { y ij -y i l } ≤ max j,l x ij -x i l -y ij -y i l ≤ max j l x ij -x i l -y ij -y i l .Since K is the complete simplicial complex over M vertices, each edge[v i , v j ] appears in M -2 k-1h-simplices, we only need to decide which extra k-1 vertives to include.Based on the cellular stability theorem [38] ,W p (Dgm h (R(P 0 )), Dgm h (R(P 1 ))) p ≤ [vi 0 ,••• ,vi k ] |f ([v i0 , v i1 , • • • , v i k ]) -g([v i0 , v i1 , • • • , v i k ])| p + [vi 0 ,••• ,vi k+1 ] |f ([v i0 , v i1 , • • • , v i k+1 ]) -g([v i0 , v i1 , • • • , v i k+1 ])| p ≤ [vi 0 ,...,vi k ] max j,l x ij -x i l -y ij -y i l p + [vi 0 ,••• ,vi k+1 ] max j,l x ij -x i l -y ij -y i l p ≤ [vi 0 ,...,vi k ] max j l∈[vi 0 ,••• ,vi k ] x ij -x i l -y ij -y i l p + [vi 0 ,••• ,vi k+1 ] max j l∈[vi 0 ,••• ,vi k+1 ] x ij -x i l -y ij -y i l p ≤ xj ∈P0 M -2 k -1 x l ∈P0 ∥x j -x l ∥ -∥y j -y l ∥ p + xj ∈P0 M -2 k x l ∈P0 ∥x j -x l ∥ -∥y j -y l ∥ p ≤ M -1 k W pair p (P 0 , P 1 ) p .For h = 0 the calculations are easier as the vertex values are all 0.W p (Dgm 0 (R(P 0 )), Dgm 0 (R(P 1 ))) ≤ i<j |f ([v i , v j ]) -g(v i , v j )| p = i<j | ∥x i -x j ∥ -∥y i -y j ∥ | p ≤ W pair p (P 0 , P 1 ) p .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiment settings: 
 Datasets. We evaluate our method on three datasets with different classes and resolutions. CIFAR-10 [39] is a dataset containing colored images classified into 10 classes, which consists of 5,000 training samples and 1,000 testing samples of size 32 * 32 per class. CIFAR-100 [39] comprises 500 training images and 100 testing images per class, with the same size with CIFAR-10. ImageNet-100 [40] , a subset of ImageNet-1000, is composed of 100 classes with 1300 images per class for training and 500 images per class for validation. ImageNet-100 resembles real-world scenes with a higher resolution of 256*256.Continual Semi-supervised Setting. For both CIFAR-10 and CIFAR-100, we train the models with different levels of supervision, i.e., λ ∈ {0.8%, 5%, 25%}. For instance, the label ratio corresponds to 4, 25, and 125 annotated samples per class in CIFAR-100. Regarding ImageNet-100, we choose {1%, 7.7%} for label ratio. To build the continual datasets, we use two learning sequence settings in the literature NNCSL [13] and DSGD [14] , and divide the datasets into equally disjoint tasks: 5/10/10 tasks for CIAFR-10/CIFAR-100/ImageNet-100, i.e., 2/10/10 class per task. We also follow the standard incremental setting of CL and assume that only limited space is left for previous data. In other words, the memory buffer can be constructed for both labeled and unlabeled data. Following [13, 7] , we set the buffer size as 500 and 2000.Evaluation Metrics. We evaluate the performance of different methods considering the average accuracy over all the classes seen after the last task AA T = 1 T T i=1 a T,i , and average incremental accuracy of each task AIA = 1 T -1 T t=2 AA t [1] . Analysis with backward transfer, which presents the degree of forgetting BW T= 1 T -1 T -1 t=1 (a T,t -a t,t), is also evaluated in the ablation study. The forgetting is usually reflected as negative BW T .Baselines. We adopt the semi-supervised continual learning pipeline of DSGD [14] . Two categories of methods: distillation-based method iCaRl [4] and parameter consolidation method DER [7] are considered as the strategies for overcoming catastrophic of labeled data. A common semi-supervised learning method Fixmatch [36] is adopted to learn discriminate representation of unlabeled data. Table 1 : Average incremental accuracy and average accuracy of various methods on 5-tasks CIFAR-10, 10-tasks CIFAR-100 and 10-tasks ImageNet-100 settings following the learning sequence [14] , with a memory buffer size of 2000. The improvements of PsHD compared to the state-of-the-art methods are highlighted in blue color. Comparison with SSCL Methods. Table 1 shows the disrupted class order following [14, 41] across the three datasets, and also report the average incremental accuracy to demonstrate the effectiveness of the proposed method during the whole evolution. The backbone architectures remain consistent with those used in [14] , while the distillation loss on unlabeled samples in the memory buffer is replaced by our newly proposed persistent homology distillation loss. We obtain a general improvement on the two backbones, demonstrating the intrinsic topological features captured by PH are more efficient than the sample-wise local similarity distillation. In addition, the scale of the sub-graph is an important parameter that should be selected in DSGD [14] , which does not occur in the proposed PsHD since its automatic multi-scale characteristic. The results of ImageNet-100 also demonstrate the effectiveness on a more challenging benchmark with a higher resolution.For a more comprehensive comparison, we also validate our method on the default class sequence. Table 2 shows the results of semi-supervised continual learning on CIFAR-10 and CIFAR-100 without shuffle class order. It can be seen that our PsHD outperforms all the methods in the average accuracy across several label ratios on both datasets. As representative relation distillation methods, NNCSL [13] and DSGD [14] show comparative results compared to sample representation distillation.Our method surpasses the two approaches further, indicating the superiority of distilling persistent homology for unlabeled data in semi-supervised continual learning. In addition, NNCSL [13] only considers the rehearsal of labeled data, limiting its potential advantages on unlabeled data, while our methods focus on the unlabeled data and complement the class-instance similarity preservation. Better Adaptability to Unlabeled Data Compared with Distillation Methods. Table 3 presents a comparison of traditional knowledge distillation and its more recent variants from [48] in the learning tasks of our SSCL problem. The effectiveness of logits and features distillation diminishes generally, particularly as tasks become more challenging and numerous. For instance, the performance on CIFAR100 is lower than that on CIFAR10, and a reduced labeled ratio results in decreased accuracy.Although relation and topology distillation are relatively effective for SSCL, their stability is still inferior to that of our method.Stability to Noise Interference. We also validate the stability of our methods under noise interference. As shown in Table 4 , we add Gaussian noise with standard deriviation {0.2, 1, 1.2} on five distillation-based continual learning methods and our PsHD. The results indicate that our method exhibits greater stability in terms of both forgetting degree and accuracy, despite the presence of higher noise levels. These findings further confirm the stability conclusions drawn in Theorem 1. Ablation Study. We firstly ablate the components of persistence homology distillation through quantitative analysis in Table 5 on both CIFAR-10 and CIFAR-100 across 5% and 25% label ratios. The baseline is based on iCaRL_Fix and DER_Fix, where the iCaRL and DER are two methods in traditional continual learning and Fix is Fixmatch for semi-supervised learning. L SSL and L hd represent the semi-supervised loss and homology diagrams distillation loss, respectively. The general improvements of our structural distillation loss confirm the contributions of our proposed components.Visualization of the Effectiveness of Persistence Homology Distillation. We also provide the class activation heatmap of old categories in the initial and following tasks during the continual learning process in ImageNet-100. These heatmap pictures localize class-specific discriminative regions, as shown in Figure 4 . It can be seen from (a) part of the three groups that the model without our persistence homology distillation usually drop the class-specific attention region with the task takes advancement, despite the init model having learned a discriminative representation. In addition, the base model may even shift focus on the background area in the middle tasks, such as Task 6-9 in 2.a, resulting in unreliable knowledge accumulation. This is because of the insufficient previous data support, memory data under-utilization, and feature space transformation, which lead to attention shift as shown in 1.a, and even attention vanishment as shown in 3.a.In contrast, the introduction of our persistence homology distillation is able to preserve the discriminative region along with the whole continual learning, as depicted in (b) part, which indicates that preserving persistence homology that captures the class-wise structural information encourages the model to focus on the class-specific region continually. It is worth noting that even if the initial model is not capable of noticing the class-specific region, as shown in 3.a, which may result from insufficient supervision, our method still empowers the following model training to activate the potential class-relevant region since it focuses on the global distribution of each class and pushes the Effect of the Weight of Persistence Homology Loss. We evaluate the influence of loss weight λ in Eq. 3 across the three datasets, and report the results in Figure 5 . It can be seen that a higher weight indicates higher average incremental accuracy and a lower forgetting degree generally. In addition, our method improve the baseline generally within a reasonable range. Too much higher weight on persistence homology loss, for instance bigger than 1, is not supposed to be better for the trade-off problem between old and new tasks. We choose 1 for CIFAR10 and CIFAR100, and 1.5 for ImageNet100 in our experiments.Effect of k-simplex in Persistence Homology. The persistence diagram calculation considers the lift-span of h-dimensional holes that existed in the nested simplicial complexes data distribution. We verify the effectiveness of dimension h in the homology diagram by considering 0-dimensional holes and both 0,1-dimensional holes in CIFAR-10 and CIFAR-100. The results are shown in Figure 5 .It can be seen that considering the persistence of both 0,1-dimensional holes are more effective in CIFAR-10, while CIFAR-100 slightly prefers the simple 0-dimensional holes persistence preserved.Effect of Memory Buffer Distribution. To investigate the necessity of replaying the unlabeled data, we evaluate the effect of memory buffer allocation of labeled and unlabeled data on CIFAR-100 of 5% and 25% label ratios in Table 6 , where ratios represent the percentage of labeled data in the memory buffer. The general improvements indicate the percentage of labeled data has an important effect on the performance. It can be observed that replacing 20%-40% labeled samples with unlabeled samples improves 1.49%, 1.27% in 5%. We experimentally choose 0.6-0.9 in the following settings. Memory Reduction. Our method is also memory-efficient for SSCL. As shown in Table 7 , the proposed PsHD that incorporates unlabeled data surpasses methods relying solely on labeled data, such as NNCSL, by an average of 3.9% across six SSCL settings. To further verify the memory efficiency, we reduced the memory buffer size by 60% to 2000, achieving an average improvement of 1.5% compared to NNCSL across the six semi-supervised continual learning tasks.Computation Efficiency. A newly topology-based distillation method TopKD [47] , which is similar with our method, is proposed for model compression. We prove that the complexity proportion of TopKD and our method is b 2 , where b is the number of old classes. The specific derivation is clarified in Appendix A.4. This value indicates the relative complexity of TopKD grows polynomially when the class number increases, highlighting our computation superiority.We evaluate the memory storage, model parameters and training time between our methods and state-of-the-art methods in Appendix A.4 Table 8 , indicating the superiority. Despite the acceleration algorithm, a limitation of our method is that the computation of the persistence diagram relies on the Guidh package, which operates on the CPU. We plan to explore the learnable network RipsNet [49] to approximate the persistence diagram and further accelerate the computation in the following work. 
conclusion: 
 We proposed Persistence Homology Distillation (PsHD) to address catastrophic forgetting and noise sensitivity in Semi-Supervised Continual Learning (SSCL). First, we utilized persistence homology to capture stable topological feature representation of unlabeled data in semi-supervised learning. Next, we proposed a novel persistence homology distillation strategy for SSCL that is insensitive to noise information interference based on the intrinsic topological features and devised an accelerating algorithm to reduce computation costs. Finally, general improvements in three benchmarks demonstrated the efficiency of PsHD in overcoming catastrophic forgetting in SSCL. Persistence Homology. Now we explain how persistence homology works. It is easy to obtain an inclusion map K i → K j , 0 < i < j < m from Eq. 2. These maps also induce a group homomorphism of the corresponding homology groups 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The original paper related to the experiment results and data are all cited. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: EQUATIONwhere H d (K i ) and H d (K j ) are homology groups of the d th order. A homology class h is born at timef i if h ∈ H d (K i ) but h / ∈ img(F i-1,j d ). Provided h is born at f i , h dies at time f j only if F i,j-1 d (h) / ∈ img(F i-1,j-1 d ) while F i,j d (h) ∈ img(F i-1,j d) [50] . ('img' refers to the image of a map). These maps reveal which feature features persist (a feature refers to a homology class, corresponding to a component or a n-dimensional hole). An easier way to understand is as follows. A new homology class is born at time f i only if it does not appear at the existing features, and it dies at time f j only if it still persists at time f j-1 and is merged into other features at time f j . The time between death and birth of a feature is called its lifetime or persistence.We illustrate our persistence homology distillation in Algorithm 1.Input: Replayed batch of unlabeled data: B o u . Parameter: Embedding encoder of current task f new and old task f old ,1: E ← B o u , i ← 0, S ← ∅ 2: repeat 3:x i ← RandomSample(E) 4: (f new xi , f old xi ) 5:Group the weighted k-hop neighborhood set N (x i , k) by f old xi 6:E ← E\N (x i , k), S ← S ∪ x i , i ← i + 1 7:Compute the Wasserstein distance d N k x (f old , f new ) according to Eq. 5 8: until E is empty 9: Minimize the total lossL hd = 1 |S| x∈S d N k x (f old , f new ) A.As the similar topology-based distillation methods, ours computation cost is smaller than the TopKD. The main reason is because of the smaller number of persistence homology points u in the persistence homology Dgm h (N k x , f old ) ∪ △ of Eq. 5. The number of u is proportional to simplices number n since HP complexity is O(n 2.5 ). It is hard to approximate n, while we can deriviate its upper bound in our methodU K n,P sHD = b K i=1 M/b i and TopKD U K n,T opKD = K i=1 M i, where M is the replayed samples in each batch, b is approximate to class number and K is the highest number of distilled simplex. We experimentally set K to be 3, since 1-simplex derive when i = 2 and dead when i = 3. The upper bound ratio is b 2 .Proof.U 3 n,T opKD = M + M (M -1) 2 + M (M -1)(M -2) 6 , U 3 n,P sHD = M + M (M/b -1) 2 + M (M/b -1)(M/b -2) 6 , U 3 n,T opKD U 3 n,P sHD = 6b 2 + (M -1)3b 2 + (M -1)(M -2)b 2 6b 2 + (M -b)3b + (M -b)(M -2b) = 5b 2 + M 2 b 2 5b 2 + M 2 ≈ b 2 (b → M, M >> 5).When the number of old class b raises in the continual learning process, the relative computational complexity grows quadratically with b. Heavy computation usually accumulates during the whole learning process, so this ratio demonstrating the superiority of our methods.In addition, we also evaluate the memory storage, model parameters and training time between our methods and state-of-the-art methods in Table 8 . When reducing the replayed examplers to 2000, our method still achieves the best performance 46.0%. Compared to the best indices, our method lessen the training time by 10.9%, example buffer by 60.9%, and memory size by 28.4%. Therefore, while existing strong baselines excel in different aspects, NNCSL in accuracy, DER_Fix in training time, and DSGD relatively balanced computation costs and accuracy, our method achieves the overall least resources overhead and the highest accuracy. We provide additional visualizations of PsHD in Figure 6 . Expect for the issue in Figure 4 , the attention bias shows that the initial model fails to recognize the accurate class-specific regions, making it difficult for subsequent models to consistently focus on the correct areas. By incorporating persistent homology features, the model is encouraged to direct attention to the appropriate regions. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The paper leverages persistent homology to enhance knowledge distillation and memory replay in semi-supervised continual learning, introducing an efficient algorithm to reduce overhead and demonstrating its stability and effectiveness through experiments. However, the improvement is quite incremental in many cases.

This paper is borderline for acceptance. After the rebuttal, all reviewers have given a score of 5. The previously raised issues regarding experiments and unclear concepts have been adequately addressed. To make a final decision, I carefully reviewed the paper, initial reviews, rebuttal, and all discussions. My assessment aligns with the reviewers' opinions: after revisions, the paper does not contain any obvious errors or major weaknesses, but the overall experimental impact and contribution to the field remain limited. 

I am currently leaning toward acceptance but will discuss further with the SAC.

Reviews: review: paper_summary: paper_summary: This paper proposed to preserve intrinsic structural information with the use of persistent homology, so as to improve knowledge distillation and memory replay in semi-supervised continual learning. The authors provided an efficient acceleration algorithm to reduce computational overheads and theoretically demonstrated its stability. Extensive experiments demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. Although the authors have provided extensive experiments, the effectiveness of the proposed method seems to be limited. It offers less than 1% improvements in a majority of cases with relatively simple datasets (e.g., CIFAR-10 and CIFAR-100). This may be due to the classic NP-hard problem in selecting a few data to represent the entire dataset.

2. Although the authors have provided an acceleration algorithm to reduce computational overheads and conceptually analyzed it with their own method, I suggest to compare the overall resource overheads (storage and computation) of their method with other strong baselines (e.g., NNCSL, DER_Fix, and DSGD).
questions: questions: I think this is essentially a solid paper. My major concerns lie in the effectiveness and efficiency. Please refer to the Weaknesses.
limitations: limitations: In Checklist the authors claimed that they have discussed the limitations in Section 5. However, Section 5 only presents the Conclusion without any discussion of the limitations.
paper_summary: paper_summary: The paper proposes a persistence homology knowledge distillation for continual learning (PsHD). PsHD loss is calculated using a ''memory buffer'' between a previous variant of a network and a new one. Experiments show some improvement w.r.t. baselines. Ablation studies are provided.
The main issue of the paper is limited novelty, also some essential details are missing (see below).
strength_weakness: strength_weakness: 1. The idea that knowledge distillation can help continual learning is not new, see [2]. Given this, the novelty of the paper is small.
Do you have an ablation with a traditional KD [1] and its more recent variants from [2]?
2. A relevant reference [3] is missing. How you paper is related to [3], is your method better?
3. I can't find an explicit equation for $L_{CL}$.

[1] Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
[2] Li, S., Su, T., Zhang, X., & Wang, Z. (2024). Continual Learning with Knowledge Distillation: A Survey. Authorea Preprints.
[3] Kim, J., You, J., Lee, D., Kim, H. Y., & Jung, J. H. Do Topological Characteristics Help in Knowledge Distillation?. In Forty-first International Conference on Machine Learning.
questions: questions: 1. How do you differentiate PsHD?
2. You claim that PsHD helps to handle noise, but there is no noise in experimental settings.
The claim "however, traditional distillation strategies often fail in unlabeled data with inaccurate or noisy information" is not proved also. 
3. What is the difference between a memory buffer and unlabeled data? To my understanding, for PsHD loss one only needs unlabeled data.
4. Is the SSL part of loss a critical part of your method? Is seems to be completely independent of the main idea (PsHD loss) and, probably, gives to your method an artificial advantage w.r.t. others.

**Other**

1. In Figure 2, two diagrams seem to be identical, while the top simplicial complex has one hole, but the bottom one - two holes.
2. The Wasserstein-p distance (line 164) equation is not correct, because the subscript $\infty$ relates to Wasserstein-$\infty$ distance.
3. line 214: typo CIAFR
limitations: limitations: -
paper_summary: paper_summary: The paper proposes a new method PsHD to preserve intrinsic structural information in semi-supervised continual learning. The method proposes to uses distillation and cross-entropy loss on the continual learning samples.
strength_weakness: strength_weakness: 1. I think the paper can be further polished in terms of writing, presentation and clarity. For example. I think Figure.4 can be further improved for clarity. 
2. The results raise some concerns regarding the proposed method. In Table 4, we observe that changing the data allocation ratio results in performance change within 1.5% for 5% and within 1% for 25% setting. Also in Table.4, optimal data ratio are different for the two settings. We also observe in Table 1 that the proposed method leads the previous methods by less than 1%. It makes the author's claim for their ``superiority'' performance quite unsupported. 
3. This brings my third concern to the work: Some language in the paper seems overclaimed. The paper tried to adopt many large words like ``significant'' and ``superior'' while the performance does not support such claims.
4. In Figure.4, larger lambda seem to help the model performance, then why not train with larger lambda? Maybe there is an elbow effect but there needs more experiment to show that.
questions: questions: Please in weakness
limitations: limitations: NA

Comments: title: Answer
comment: We sincerely appreciate the reviewer's response. Meanwhile, we are very grateful for the reviewer's recognition of our responses and work, while increasing the final rating to ‘borderline accept’. Great thanks again!
I thank the authors for their rebuttal. After considering the rebuttal and other reviewers' comments, I will raise my score as the rebuttal addresses my concerns.
We sincerely appreciate the reviewer for fast reply. Meanwhile, we are very grateful for the reviewer's recognition of our responses and work, while increasing the final rating to ‘borderline accept’. Great thanks again!
I appreciate the detailed answer. Authors did additional experiments to address my questions. Both clarifications and experiments must be included into manuscript. After some consideration, I'm raising my score. While I don't see evident mistakes in the paper, the final score is defined by overall novelty/impact of the paper.
Dear AC and reviewers,

We greatly thank for your time and effort to review our paper, while giving many valuable suggestions. As a kind reminder, we provided the detailed responses to the comments. Particularly, we sincerely thank the Reviewer AVV3 for very fast reply and recognition. We know and understand the reviewers are very busy, and sincerely hope Reviewer K7Tp and Reviewer 2waA could take a little time to review our responses, while let us know whether you have further questions. We are willing to try our best to answer these questions. Great thanks!

Best wishes
We sincerely thank the reviewer for very fast reply. Meanwhile, we are very grateful that the reviewer recognizes our responses and work. Great thanks again!
I thank the authors for their rebuttal. After considering the rebuttal and other reviewers' comment, I keep my score unchanged.