TopoFR: A Closer Look at Topology Alignment on Face Recognition

Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-24 08:57:37

Abstract: The field of face recognition (FR) has undergone significant advancements with the rise of deep learning. Recently, the success of unsupervised learning and graph neural networks has demonstrated the effectiveness of  data structure information. Considering that the FR task can leverage large-scale training data, which intrinsically contains significant structure information, we aim to investigate how to encode such critical structure information into the latent space. As revealed from our observations, directly aligning the structure information between the input and latent spaces inevitably suffers from an overfitting problem, leading to a structure collapse phenomenon in the latent space. To address this problem, we propose TopoFR, a novel FR model that leverages a topological structure alignment strategy called PTSA and a hard sample mining strategy named SDE. Concretely, PTSA uses persistent homology to align the topological structures of the input and latent spaces, effectively preserving the structure information and improving the generalization performance of FR model. To mitigate the impact of hard samples on the latent space structure, SDE accurately identifies hard samples by automatically computing structure damage score (SDS) for each sample, and directs the model to prioritize optimizing these samples. Experimental results on popular face benchmarks demonstrate the superiority of our TopoFR over the state-of-the-art methods. Code and models are available at: https://github.com/modelscope/facechain/tree/main/face_module/TopoFR.

Main paper: 
 introduction: 
 Face recognition (FR) is a critical biometric authentication technique that is widely applied in various applications, including electronic payments, smartphone lock screens, and video surveillance. In recent years, convolutional neural networks (CNNs) have achieved remarkable success in FR task, thanks to their powerful ability to autonomously extract face features from images. Existing studies on FR primarily focuses on constructing more discriminative face features by developing margin-based loss functions Deng et al. (2019) ; Wang et al. (2018b) ; Kim et al. (2022) and powerful network architectures Chang et al. (2020) ; Schroff et al. (2015) ; Zhong & Deng (2021) . Recently, the success of unsupervised learning Wu et al. (2023a) ; Moor et al. (2020) ; Jiang et al. (2021) ; Lopes & Pedronette (2023) and graph neural networks Wong & Vong (2021) ; Wu et al. (2023b) has demonstrated the importance of data structure information in improving model generalization. However, to the best of our knowledge, how to effectively mine the potential structure information in large-scale face data has not investigated. Thus, in this paper, we extend our interests on building a cutting-edge FR framework through exploiting such powerful and substantial structure information.First, we use Persistent homology (PH) Edelsbrunner et al. (2008) ; Barannikov (1994) , a mathematical tool used in topological data analysis Carlsson (2009) to capture the underlying topological structure of complex point clouds, to investigate the evolution trend of structure information in existing FR framework and illustrate 3 observations: (1) as the amount of data increases, the topological structure of the input space becomes more and more complex, as verified in Figures 1(a )-1(d) (this observation is also consistent with the conclusions proposed in some theoretical works that conducted topological research on random points Bobrowski & Kahle (2018) ; Kahle (2011) ;(2) as the amount of data increases, the topological structure discrepancy (measured via topologically relevant distance Moor et al. (2020) between the persistence diagrams of two spaces) between the input space and the latent space becomes increasingly larger, as verified in Figure 2a; (3) The results in dataset respectively, and compute their persistence diagrams using PH, where H j represents the jth dimension homology. In topology theory, if the number of high-dimensional holes in the space is more, then the underlying topological structure of the space is more complex. Persistence diagram is a mathematical tool to describe the topological structure of space, where the j-th dimension homology H j in persistence diagram represents the j-th dimension hole in space. As shown in Figures 1(a )-1(d), as the amount of face data increases, the persistence diagram contains more and more high-dimensional homology (e.g., H 3 and H 4 ), which indicates that the input space contains an increasing number of high-dimensional holes. Therefore, this phenomenon also demonstrates a growing complexity in the topological structure of the input space.Figure 2b demonstrate that as the depth of the network increases, the topological structure discrepancy becomes progressively smaller. This finding also provides an explanation for why models with more complex structure achieve higher FR accuracy. Based on the above observations, we can infer that in FR tasks with large-scale datasets, the structure of face data will be severely destroyed during training, which limits the generalization ability of FR models in practical application scenarios. To this end, we propose to improve the generalization performance of FR models by preserving the structure information.However, we experimentally find that directly using PH to align the topological structures of the input space and the latent space may cause the model to suffer from structure collapse phenomenon.Concretely, under this experimental setting, we have 2 following quantitative and visualization results: (1) As shown in Figure 2c , the topological structure discrepancy drops to 0 dramatically during early training. (2) As illustrated in Figure 2d , when evaluating on the IJB-C benchmark Maze et al. (2018) , there exists a significant structure information gap between the input space and the latent space. These typical overfitting phenomena indicates the latent space fails to preserve the structure information of input space accurately.To remedy this issue, we propose a superior FR model named TopoFR that leverages a Perturbationguided Topological Structure Alignment (PTSA) strategy to better preserve the topological structure information of the input space in corresponding latent face features. PTSA first employs a diverse data augmentation (DDA) mechanism to randomly augment training samples, effectively perturbing the latent space and increasing its structure diversity. Then PTSA utilizes an invariant structure alignment (ISA) mechanism to align the topological structures of the original input space and the perturbed latent space, resulting in face features with stronger generalization ability. ) and found that i) directly using PH to align the topological structures will cause the discrepancy to drops to 0 dramatically; ii) whereas using our PTSA strategy promotes a smooth convergence of structure discrepancy. (d): Aligning the topological structures directly using PH (batch size=128) will lead to significant discrepancy when evaluating on IJB-C benchmark. Our PTSA strategy effectively mitigates this overfitting issue, resulting in smaller structure discrepancy during evaluation.hard samples with significant structure damage, SDE can gradually guide these samples back to their reasonable positions, thereby improving the generalization ability of FR model.In summary, the main contributions are listed as follows:1) To the best of knowledge, we are the first to explore the topological structure alignment in FR task. We propose a novel topological structure alignment strategy called PTSA to effectively align the structures of the original input space and the perturbed latent space.2) A novel hard sample mining strategy named SDE is introduced to mitigate the adverse impact of hard samples on the latent space structure.3) Our experimental results show that the proposed method outperforms state-of-the-art methods on several face benchmark datasets, e.g., by employing ResNet-100 as the backbone, we achieve an accuracy of 96.95% on "TAR@FAR=1e-4" of the IJB-C benchmark using MS1MV2 training set. 3 BACKGROUND: PERSISTENT HOMOLOGY PH is a computational topology method that quantifies the changes in the topological invariants of a Vietoris-Rips complex as a scale parameter ρ is varied. In this section, we briefly introduce some key concepts of PH. Further details on PH can be found in Refs. Edelsbrunner et al. (2008; 2000) . 
related works: 
 Notation. X := {x i } n i=1 represents a point cloud and µ : X × X → R denotes a distance metric over X . Matrix M represents the pairwise distances (i.e., Euclidean distance) between points in X .Vietoris-Rips Complex. The Vietoris-Rips complex Vietoris (1927) is a special simplicial complex constructed from a set of points in a metric space, and it can be used to approximate the topology of the underlying space. For 0 ≤ ρ < ∞, we represent the Vietoris-Rips complex of point cloud X at scale ρ as V ρ (X ), which contains all simplices (i.e., subsets) of X , and each component of point cloud X satisfies a distance constraint: µ(x i , x j ) ≤ ρ for any i, j. Moreover, the Vietoris-Rips complex satisfies a nesting relation, i.e., V ρi ⊆ V ρj for any ρ i ≤ ρ j , which allows us to track the evolution progress of simplical complex as the scale ρ increases. It is worth noting that V ρ (X ) and V ρ (M) are equivalent because constructing the Vietoris-Rips complex only requires distance.Homology Group. The homology group is an algebraic structures that analyzes the topological features of a simplicial complex in different dimension j, such as connected components (H 0 ), cycles/loops (H 1 ), voids/cavities (H 2 ), and higher-dimensional features (H j , j ≥ 3). By tracking the changes in topological features (H j ) of the Vietoris-Rips complex as the scale ρ increases, it is possible to gain insight into the multi-scale topological information of the underlying space.Persistence Diagram and Persistence Pairing. The persistence diagram D is a multi-set of points (b, d) in the Cartesian plane R 2 , which encodes information about the lifespan of topological features. Specially, it summarizes the birth time b and death time d information of each topological feature in a homology group, where birth time b signifies the scale at which the feature is created and death time d refers to the scale at which it is destroyed. The difference between the death and the birth times is the lifetime of the homology group l f = |d -b|. The persistence pairing γ contains indices (i, j) corresponding to simplices r i , r j ∈ V ρ (X ) that create and destroy the topological features identified by (b, d) ∈ D, respectively. 
methodology: 
 In this paper, we propose a novel framework named TopoFR for constraining the FR model to preserve the topological structure information of the input space in their latent features. The architecture of our TopoFR model is depicted in Figure 3 . It consists of two components: a feature extractor F and an image classifier C. Mathematically, given an input image x, the latent feature extracted by F is denoted as f = F(x) ∈ R l , and the classification probability predicted by C is denoted as g = C(f ) ∈ R K , where l represents the feature dimension and K denotes the number of classes. The entropy of the classification prediction probability g can be represented as E(g) = -K k=1 g k log g k , where g k is the probability of predicting a sample to class k.As mentioned in Section 1, directly applying PH to align the topological structures of the input space and the latent space can cause the FR model to encounter structure collapse phenomenon. To remedy this problem, we propose a Perturbation-guided Topological Structure Alignment (PTSA) strategy that includes two mechanisms: diverse data augmentation and invariant structure alignment. L arc ( x i , y i ) = -log e s(cos(θ y i +m))e s(cos(θ y i +m))EQUATIONwhere y i indicates the class label of the original image x i , s is a scaling hyperparameter, θ k i is the angle between the k-th class center and feature, and m > 0 denotes an additive angular margin.Invariant Structure Alignment (ISA). Given a mini-batch of original training samples X = {x i } n i=1 , we denote the augmented batch samples as X = { x i } n i=1 . For the augmented samples, we denote the latent features extracted by F as Z = f i n i=1. During forward propagation, we can construct the Vietoris-Rips complexes V ρ (X ) and V ρ ( Z) for point clouds X and Z respectively, based on their respective pairwise distance matrix (i.e., Euclidean distance) M X and M Z . Then we can compute the persistent homology of V ρ (X ) and V ρ ( Z), and obtain their corresponding persistence diagrams D X , D Z and persistence pairings γ X , γ Z , respectively.Ideally, no matter how the face image is perturbed, the position of the encoded face feature in the latent space should remain unchanged, and the topological structure of the perturbed latent space should also be consistent with the original input space. To this end, we choose to align the original input space X with the perturbed latent space Z to achieve this goal. To align the topological structure of two spaces, prior studies usually utilize bottleneck distance or p-Wasserstein distance to measure the discrepancy Turner et al. (2014) ; Mileyko et al. (2011) between persistence diagrams. However, these two distance metrics are sensitive to outliers Horak et al. (2021) ; Bertsekas (1981) and will increase the training time of the FR models, rendering them unsuitable for FR tasks with extremely large-scale datasets.To mitigate this issue, we turn to retrieve the persistence diagrams values by subsetting the corresponding pairwise distance matrix with edge indices provided by the persistence pairings Zomorodian (2010); Moor et al. (2020) , i.e., D X ≃ M X [γ X ] and D Z ≃ M Z [γ Z ]. By comparing the difference between two topologically relevant distance matrices from both spaces, we can quickly and stably compute the discrepancy between their persistence diagrams, providing an efficient solution for structure alignment of FR models driven by large-scale datasets. We formulate the ISA loss as follows: EQUATIONEQUATIONwhereEQUATIONU(0, Ω) is a uniform distribution defined on [0, Ω], π is a prior probability, and Σ is the variance of Gaussian distribution N (a|0, Σ). In this mixed model, the uniform distribution term U and the Gaussian distribution term N + respectively model the hard samples and easy samples. Then the posterior probability that the sample x i to be hard (i.e., high-uncertainty) can be computed as follows:EQUATIONIn Equation ( 5), when the classifier prediction probability is close to uniform distribution, the posterior probability of a sample belonging to hard data will be very high, i.e., (g i → [ 1 K , 1 K , • • • , 1 K ], h φ ( x i ) → 1), otherwise it is very low. By this mean, the prediction uncertainty of sample x i can be measured by a quantitative probability h φ ( x i ).Assume E( g i ) = (-1) ϵi E( g i ), ϵ i ∼ B(1, 0.5), where B is a Bernoulli distribution Dai et al. (2013) , then variable E( g i ) obeys the following statistical distribution:EQUATIONIn this way, the parameter set φ = {π, Σ, Ω} of GUM can be estimated via the Expectation-Maximization (EM) algorithm Coretto & Hennig (2016) with the following iterative formulas: t) , Ω (t) )EQUATIONEQUATIONwhereη 1 = n i=1 h (t+1) φ ( xi) 1-π (t+1) E( g i ) n i=1 (1 -h (t+1) φ ( x i )) , η 2 = n i=1 h (t+1) φ ( xi) 1-π (t+1) ( E( g i )) 2 n i=1 (1 -h (t+1) φ ( x i )).Structure Damage Score (SDS). Compared to correctly classified samples, misclassified samples usually have larger difficulty and have greater destructive effects on the latent space's topological structure. Therefore, misclassified samples need to receive more attention from the model during training. Inspired by the Focal loss Lin et al. (2017) , we design a probability-aware scoring mechanism ω( x i ) that combines prediction uncertainty and prediction accuracy to adaptively compute SDS for each sample x i :EQUATION)where λ is a temperature coefficient, and g gt i represents the prediction probability of ground truth. Specifically, SDE assigns higher SDS to hard samples and lower SDS to easy samples, which effectively balances the contribution of each sample to the training objective. By assigning higher scores to hard samples, the model is encouraged to focus more on learning these challenging samples, which can ultimately improve the generalization performance of the FR system. Formally, the SDS weighted classfication loss L cls can be defined as:EQUATIONDuring training, to minimize the objective L cls , the model needs to optimize both the SDS ω and the loss L arc , which brings two benefits: (1) Minimizing L arc can encourage the model to capture face features with greater generalization ability from diverse training samples. (2) Minimizing SDS ω can alleviate the damage of hard samples to the latent space's topological structure, which is beneficial to the preservation of topological structure information and the construction of clear decision boundary.To summarize, the overall objective of TopoFR can be formulated as follows:minEQUATION)where α is hyper-parameter that balances the contributions of the loss L cls and the loss L sa .  
results on mainstream benchmarks: 
 Results on LFW, CFP-FP and AgeDB-30. We employ ResNet-100 He et al. (2016) as the backbone, and adopt MS1MV2 and Glint360K to train our models respectively. To demonstrate the universality of our method, we also provide detailed experimental results of our TopoFR † model trained by CosFace Wang et al. (2018b) . The results are reported in Table 1 .As stated in Refs. Kim et al. (2022) ; Meng et al. (2021) , the performances of existing FR models on these three benchmarks have reached saturation. On MS1MV2 training set, we note that our TopoFR and TopoFR † models still achieve performance gains. Specially, compared with some cuttingedge competitors (e.g., BroadFace) on these three benchmarks, the proposed TopoFR † trained on MS1MV2 still achieves performance gains on the pose-invariant (0.2% improvement on CFP-FP) and age-invariant (0.04% improvement on AgeDB-30) face verification and becomes a SOTA model.On Glint360K training set, the proposed TopoFR becomes SOTA model and surpasses ArcFace by 0.04%, 0.39% and 0.41% on three benchmarks, respectively. It is worth mentioning that regardless of the training set used, our TopoFR and TopoFR † models exhibit almost no fluctuation in accuracy across these three benchmarks, which indicates the stability and robustness of our TopoFR. Results on IJB-C. We train our TopoFR on MS1MV2 and Glint360K respectively, and compare with SOTA competitors on IJB-C benchmark, as reported in Table 2 . On MS1MV2 training set, our models obtain the best results on "TAR@FAR=1e-4" under different backbones. Specially, our R100 TopoFR model outperforms the advanced method R100 AdaFace Kim et al. (2022) and achieves SOTA performance (96.95%) on "TAR@FAR=1e-4". Furthermore, under the ResNet-50 backbone, our TopoFR and TopoFR † models greatly surpass the previous SOTA method AdaFace by +0.22% and +0.13% on "TAR@FAR=1e-4" respectively, and even beat most ResNet-100 based competitors.On Glint360K training set, all our models achieve significant performance gains on IJB-C benchmark when compared with the original baselines. For instance, R100 TopoFR outperforms R100 ArcFace by +0.48% and +0.56% on "TAR@FAR=1e-5" and "TAR@FAR=1e-4" respectively. In addition, R200 TopoFR achieves SOTA performance (96.54% and 97.74%) on "TAR@FAR=1e-5" and "TAR@FAR=1e-4" respectively, and becomes a SOTA model. More importantly, the accuracy with a smaller fluctuation range indicates not only the robustness of our models, but also the stability of our strategies under different backbones.Due to the limitation of page size, more ablation experiments and analysis are placed on Appendix. TopoFR-D and TopoFR-A, indicating that preserving the topological structure information using our PTSA strategy can greatly improve FR model's generalization. (4) TopoFR outperforms TopoFR-F and TopoFR-G, which not only demonstrates the effectiveness of our SDE strategy, but also indicates that the prediction uncertainty ω 1 is complementary to the Focal loss ω 2 in mining hard samples.To visually demonstrate the effectiveness of GUM in mining hard samples, we present the estimated Gaussian density of the prediction entropy during training in Figure 4 . These curves show that the entropy of misclassified face samples (represented by black crosses) usually have rather low Gaussian density (i.e., high posterior probability h φ ), thus can be easily detected. Note that even if some misclassified samples have small entropy (i.e., high Gaussian density and low posterior probability h φ ), their SDS ω can still be corrected by the Focal loss ω 2 .3) Generalization Performance of PTSA: To demonstrate the superior generalization ability of our PTSA strategy in preserving structure information, we investigate the topological structure discrepancy between the input space and the latent space of TopoFR and its variant TopoFR-A on IJB-C benchmark. Note that TopoFR-A directly utilizes PH to align the topological structures of two spaces. The results illustrated in Figure 5 indicate that: 1) Directly using PH to align the topological structures of two spaces does not effectively reduce the structure discrepancy, as the model suffer from the structure collapse phenomenon; 2) PTSA strategy can effectively align the topological structures of two spaces and address this structure collapse phenomenon. Remarkably, Figures 5c and 5d show that our TopoFR models trained on Glint360K dataset almost perfectly preserve struc- ture information of the input space in their latent features, thereby verifying the generalization ability of PTSA strategy. 
conclusion: 
 This paper proposes a novel FR framework called TopoFR that aims to encode the critical structure information hidden in large-scale face dataset into the latent space. Specially, TopoFR leverages two key strategies: structure alignment strategy PTSA and hard sample mining strategy SDE. PTSA employs persistent homology to reduce the topological structure discrepancy between the input space and latent spaces, effectively mitigate structure collapse phenomenon and preserving structure information. SDE accurately identifies hard samples by adaptively computing SDS for each sample, and guides the model to prioritize optimizing these samples, mitigating their adverse impact on the latent space structure. Comprehensive experiments and analysis demonstrate the superiority of our proposed TopoFR on various popular face benchmarks. 
 
 Appendix: A.1 IMPLEMENTATION DETAILS Our models are trained using Pytorch on 4 NVIDIA Tesla A100 GPUs, and a mini-batch of 128 images is assigned for each GPU. We crop all images to 112×112, following the same setting as in ArcFace Deng et al. (2019) . For the backbone, we adopt ResNet50, ResNet100 and ResNet200 He et al. (2016) as modified in Deng et al. (2019) . We follow Deng et al. (2019) to employ ArcFace (s = 64 and m = 0.5) as the basic classification loss to train the TopoFR model. For the TopoFR † model trained by CosFace Wang et al. (2018b) , we set the scale s to 64 and the cosine margin m of CosFace to 0.4. To optimize the models, we use Stochastic Gradient Descent (SGD) optimizer with momentum of 0.9 for both datasets. The weight decay for MS1MV2 is set to 5e-4 and 1e-4 for Glint360K. The initial learning rate is set to 0.1 for both datesets. In terms of the balance coefficient α, we choose α = 0.1 for experiments on R50 TopoFR, and α = 0.05 for experiments on R100 TopoFR and R200 TopoFR. During training, we apply DDA mechanism with a certain probability. Specially, for an original input sample x, the probability of it undergoing DDA is ξ, and the probability of it remaining unchanged is 1 -ξ. For the hyper-parameter ξ, we choose ξ = 0.2. We utilize the Ripser package Tralie et al. (2018) to compute persistence diagrams. Notably, in our method, we focus on preserving the 0-dimension homology H 0 in the topological structure alignment loss L sa . Because preliminary experiments demonstrated that using the 1-dimension or higher-dimension homology only increases model's training time without clear performance gains.1) Effectiveness of DDA: To demonstrate the effectiveness of DDA in expanding the structure diversity of the latent space, we employ MS1MV2 as the training set, and compare TopoFR-P with its multiple variants on the IJB-C benchmark. The variants of TopoFR-P are as follows: (1) TopoFR-P (A i ′ ), the variant denotes that only one data augmentation operation A i ′ is included in the list A.(2) TopoFR-P (A i ′ + A j ), the variant indicates that two data augmentation operations A i ′ and A j are included in the list A.(3) TopoFR-P (A i ′ + A j + A j ′ ), the variant denotes that three data augmentation operations A i ′ , A j and A j ′ are included in the list A.The results gathered in Table 4 demonstrate that using a variety of data augmentation strategies can more effectively perturb the topological structure of the latent space and mitigate the structure collapse phenomenon, leading to a significant improvement in the generalization performance of FR model. EQUATIONranging over all bijections between sets of persistent intervals in diagrams D X and D Z , and ∥•∥ ∞ denotes the ∞-norm. Equivalently, the p-Wasserstein distance is defined asL p (D X , D Z ) = (inf κ:D X →D Z ϖ∈D X ∥ϖ -κ(ϖ)∥ p ∞ ) 1/p. Furthermore, some studies Cuturi (2013); Carriere et al. (2017) ; Kerber et al. (2017) ; Chen & Wang (2021) have also attempted to accelerate the computation of p-Wasserstein distances between persistence diagrams. However, the Bottleneck distance metric and p-Wasserstein distance metric are sensitive to outliers Horak et al. (2021) ; Bertsekas (1981) ; Som et al. (2018) and will increase the training time of the FR models. This makes them unsuitable for FR tasks with extremely large-scale datasets.To demonstrate the stability and computational efficiency of our ISA strategy, we employ MS1MV2 as the training set, and compare TopoFR and its two variants on the IJB-C benchmark. The variants of TopoFR are as follows: (1) TopoFR-B, the variant uses Bottleneck distance to compute the discrepancy between persistence diagrams. (2) TopoFR-W, the variant adopts 1-Wasserstein distance to measure the discrepancy between persistence diagrams. We provide the average training time per epoch of these models and their recognition performance on "TAR@FAR=1e-4" of the IJB-C benchmarks. The results presented in Table 6 reflect the following observations: (1) TopoFR outperforms TopoFR-B and TopoFR-W on "TAR@FAR=1e-4", indicating the robustness of our ISA to outliers. (2) Compared with TopoFR-B and TopoFR-W, our proposed TopoFR has a shorter training time, demonstrating the computational efficiency of ISA. 4) Parameter Sensitivity: To demonstrate the effect of the hyper-parameters α (i.e., the balance coefficient of the ISA loss L sa ) and ξ (i.e., the probability of DDA mechanism), we conduct additional experiments by setting different values of α and ξ, respectively. We use MS1MV2 dataset to train the R50 TopoFR and R100 TopoFR models, and evaluate their performance on the IJB-C benchmark. The results on "TAR@FAR=1e-4" are shown in Figures 6a and 6b . We can observe that as α increases, the accuracy first rises and then falls. This is because an overly large α will cause the model to focus more on the structure alignment and less on the classification learning of the samples during training. Additionally, setting the parameter ξ to a high value may significantly disturb the topological structure of the latent space, which can make it difficult to achieve convergence of the ISA loss L sa .We conduct a visual comparison experiment to visualize some hard samples that can be correctly classified by our method but cannot be correctly classified by existing method. The visualization results illustrated in Figure 7 weight (i.e., SDS ω ) to each sample based on its prediction uncertainty ω 1 and prediction accuracy ω 2 . Specially, SDE will assign higher SDS ω to hard samples, which can encourage the model to extract robust face features from these challenging samples, thereby effectively improving the model's generalization performance.For detailed training time analysis, please refer to the Table 7 . Due to the introduction of the structure alignment strategy PTSA and hard sample mining strategy SDE, our TopoFR models require longer training time (1.16x). Specially, compared to the vanilla R50 ArcFace model, our R50 TopoFR model requires about 2 seconds more training time per 100 steps, which does not significantly increase the training time but brings a large performance gain. And our R100 topoFR model requires about 3 seconds extra training time per 100 steps than the vanilla R100 ArcFace model, which is not a major increase in traing time but leads to a significant performance advantage. While for inference computation head, our method performs consistently with that of vanilla ArcFace model, since we adopt the same network architecture and data pre-process module.Moreover, the results in Table 7 indicate that introducing SDE strategy (i.e., GUM) leads to significant performance improvements with only a small increase in training time (i.e., R50 backbone: 0.2s / 100 steps, R100 backbone: 0.25s / 100 steps), which is reasonable. Therefore, the addition of GUM does not bring too much computational burden and does not significantly increase the training time.  


Decision: The rebuttal provided clarifications about the proposed method and its analysis that were useful for assessing the paper's contribution and responded adequately to most reviewer concerns. All reviewers recommend acceptance after discussion (with two weak accepts and one accept), and the ACs concur. The final version should include all reviewer comments, suggestions, and additional clarifications from the rebuttal.

Reviews: review: paper_summary: paper_summary: The paper introduces a new method to improve the topological structures of facial features in the latent space. By exploring the topological structure alignment in face recognition, the authors propose a new structural alignment strategy PTSA to align the structures of origin input space and feature space. The experimental results on various benchmarks have verified the effectiveness of the proposed method.
strength_weakness: strength_weakness: - While I acknowledge the competitive results of the proposed approach, the performance improvement of the method in several benchmarks remain minor. For example, the results of the method on IJB-B and IJB-C benchmarks are minor compared to prior methods on the same backbone. 

- The technical approach seems to be an incremental one. In particular, the authors adopt the ArcFace framework and introduce two additional modules on top of the ArcFace method, including Perturbation-guided Topological Structure Alignment and Structure Damage Estimation. The first module is basically a metric to compare the feature distribution between feature space and original input space, similar to the Gromov-Wasserstein distance. However, the ArcFace loss itself is also an efficient approach to learning the facial feature representation in the latent space. I am not sure whether the Perturbation-guided Topological Structure Alignment significantly improves the structure information or not.

- Why the authors claim the proposed approach improve the topological structure of the feature space, I could not see this point has been verified via either experimental results or visualization. It would be better if the authors visualize the feature distributions of faces on the latent space (via PCA or T-SNE).


- The author claims that "to the best of our knowledge, how to effectively mine the potential structure information in large-scale face data has not investigated". I do not agree with this claim since the other methods, e.g., ArcFace, SphereFace, CoseFace, etc, are also considered as approaches to improve the structures/distributions of facial features in the deep latent space via the well design loss functions. I think the statements made by the authors are over-claimed.
questions: questions: Please refer to my weakness section. In addition, I have another question related to the Perturbation-guided Topological Structure Alignment. Are the persistence diagrams and persistence pairings computation in the ISA loss differentiable for backpropagation? Can the authors detail this computational process?
limitations: limitations: The authors have discussed their limitations in the appendix. However, it will be better if the authors discuss the broader impact of the proposed method.
paper_summary: paper_summary: This paper uses the topological structure alignment in face recognition tasks, and it proposes a Perturbation-guided Topological Structure Alignment (PTSA) strategy to align the topology of input image space and latent space. In this paper, Persistent Homology(PH) is used to verify that the complexity of input space topology will increase rapidly when there are many face images. In addition, this paper also puts forward some ideas and implementation of data mining.
strength_weakness: strength_weakness: - Formula 7 may require a more specific explanation.
- The meaning of H0 in Figure 1, along with the distribution style and other differences, is best explained.
- The hard sample mining strategy SDE in this paper seems to have little correlation with Perturbation-guided Topological Structure Alignment.
questions: questions: - Is this topological alignment strategy only for face recognition tasks?
- Is the process of alignment computationally intensive?
limitations: limitations: Yes
paper_summary: paper_summary: This paper proposes a novel framework named TopoFR for face recognition (FR). The authors first discover that existing FR models struggle to effectively preserve the structure information hidden in FR dataset, and provide three specific observations: 

(i) the topological structure in the input pixel space becomes increasingly complex as the amount of data increases; 

(ii) the structure difference between pixel space and latent feature space increases as the amount of data increases;
 
(iii) the structure difference between two spaces decreases as the network architecture becomes deeper. 

Considering that FR training datasets are typically massive and contain rich structure information, this paper aims to leverage these intrinsic structure information to enhance FR model's generalization performance. However, the authors notice that directly aligning the structures between latent space and input pixel space will result in inferior performance which might be overfitting. To solve this issue, TopoFR introduces a PTSA strategy to encode the structure information of input pixel space into latent space. Specifically, PTSA utilizes RSP to perturb the latent space structure and employs ISA to match the topological structures of two spaces, effectively mitigating structure discrepancy. Moreover, a novel hard sample mining strategy SDE is proposed to identify hard samples and minimize their impact on the latent space structure. Extensive experiments on various FR benchmarks demonstrate that TopoFR outperforms state-of-the-art methods. In addition, detailed ablation studies and analysis are conducted to provide a comprehensive understanding of the proposed method.
strength_weakness: strength_weakness: There are some concerns that I would like the authors to address in order to enhance the overall comprehensiveness of the paper.
 
1) It is not clear if this method can be applied to recent proposed loss, e.g., AdaFace [1]. I recommend adding experiments to further validate the universality of the proposed method.
 
2) [Minor Comments]: Although the authors have discussed the model’s training and inference efficiency in Appendix, I suggest including comparisons of the model's GFLOPs. This would provide a more intuitive representation of the model’s computational cost.
 
Ref[1] Kim M, Jain A K, Liu X. Adaface: Quality adaptive margin for face recognition[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 18750-18759.
questions: questions: 1) EM algorithm is sensitive to the initialization of its parameters values. In Eq.(7), how to determine the initial values of parameters $\pi$, $\Sigma$, and $\Omega$ in the GUM statistical distribution?
 
2) In Eq.(2), does the ISA function loss $\mathcal{L}_{sa}$ guide the optimization of the entire network or only specific components of the network?
 
3) What distance criterion is used to construct the topological structure space of the data in Vietoris-Rips complex during model forward propagation? 
 
4) In addition, in topological data analysis, different dimensions of topological holes analyze the underlying data structure information at different levels. In ISA loss, what dimension of topological holes does the persistent homology (PH) capture and analyze?
 
5) I am very curious about the reasons why the structure differences between the input pixel and latent spaces decrease as the network becomes deeper.
limitations: limitations: It's not discussed whether this method can be applied in other related FR methods, e.g. AdaFace. I suggest to further discuss this to increase the overall quality of this paper.

Comments: title: Response to Reviewer 68Et
Response to Reviewer SWuk
Feedback to Author Rebuttal
Response to Reviewer GA4X
comment: Dear Reviewer 68Et:

We thank your response and appreciation of our work and rebuttal. We will make sure to incorporate the discussions into our revision to enable it to be a high-quality paper.

Best Regards, Authors of 505.
Thanks for your responses. I have changed my rating to Weak Accept.
Dear Reviewer SWuk:

Thank you for your recognition of our work. We appreciate your time and efforts in reviewing our work. We will incorporate your valuable suggestions into the revised version to ensure that it becomes a high-quality paper.

Best Regards, Authors of 505.
Thank you very much for your rebuttal. It has addressed my concerns.  I hope you can update your answers in the revised paper. It will help to improve the paper quality. I decided to increase my score to 6.
Dear Reviewer GA4X:

Thank you for the positive feedback. We appreciate your efforts in reviewing our work. We will reflect your suggestions in the revised version to enable it to be a high-quality paper.

Best Regards, Authors of 505.
Thanks to the authors for the comprehensive response to my concerns.

I have carefully read all the reviews and responses. The authors have addressed all of my concerns with clear explanations and additional experiment results, especially about the experiment of AdaFace + PTSA + SDE, making the benefits and novelty of the proposed method more convincing. 

I have raised the rating to reflect the comments.