TopoFR: A Closer Look at Topology Alignment on Face Recognition
Authors: Jun Dan
Yang Liu
Jiankang Deng
Haoyu Xie
Siyuan Li
Baigui Sun
Shan Luo
Venue: NeurIPS 2024 poster (2024)
Keywords: Face Recognition
Structure Alignment
Face Perception and Understanding
Primary area: machine_vision
TL;DR: Investigate structure alignment on face recognition.

Abstract: The field of face recognition (FR) has undergone significant advancements with the rise of deep learning. Recently, the success of unsupervised learning and graph neural networks has demonstrated the effectiveness of  data structure information. Considering that the FR task can leverage large-scale training data, which intrinsically contains significant structure information, we aim to investigate how to encode such critical structure information into the latent space. As revealed from our observations, directly aligning the structure information between the input and latent spaces inevitably suffers from an overfitting problem, leading to a structure collapse phenomenon in the latent space. To address this problem, we propose TopoFR, a novel FR model that leverages a topological structure alignment strategy called PTSA and a hard sample mining strategy named SDE. Concretely, PTSA uses persistent homology to align the topological structures of the input and latent spaces, effectively preserving the structure information and improving the generalization performance of FR model. To mitigate the impact of hard samples on the latent space structure, SDE accurately identifies hard samples by automatically computing structure damage score (SDS) for each sample, and directs the model to prioritize optimizing these samples. Experimental results on popular face benchmarks demonstrate the superiority of our TopoFR over the state-of-the-art methods. Code and models are available at: https://github.com/modelscope/facechain/tree/main/face_module/TopoFR.

TopoFR: A Closer Look at Topology Alignment on Face Recognition
Jun Dan 11, Yang Liu 223 Jiankang Deng", Haoyu Xie25, Siyuan Li²,
Baigui Sun 125, Shan Luo
!Zeejiang University Paacehain Community King's College London *Imperil College London Allibba Group
danjundzju. edu. en, {yang. 1iu, shan 1konnnll........ ac uk, j deng16@imperiaa ill. uk xiehaoyu. Sahhyyllibbba-lae com, aunbaigui85@gmailn com
Abstract
The field of face recognition (FR) has undergone significant advancements with the rise of deep learning. Recently, the success of unsupervised learning and graph neural networks has demonstrated the effectiveness of data structure infor- mation. Considering that the FR task can leverage large-scale training data, which intrinsically contains significant structure information, to encode such critical structure information into the latent space. As revealed from our observations, directly aligning the structure information between the input and latent spaces inevitably suffers from an overfitting problem, leading to a structure collapse phenomenon in the latent space. To address this problem, we propose TopoFR, a novel FR model that leverages topological structure alignment strategy called PTSA and a hard sample mining strategy named SDE. Concretely, PTSA uses persistent homology to align the topological structures of the input and latent spaces, effectively preserving the structure information and improving the generalization performance of FR model. To mitigate the impact of hard samples on the latent space structure, SDE accurately identifies hard samples by automatically computing structure damage score (SDS) for each sample, and directs the model to prioritize optimizing these samples. Experimental results popular face benchmarks demonstrate the superiority of our TopoFR over the state-of the-trrt methods. Code and models are available at: htt//maiini/eeeee hwtpe/aaeeeaeeeeeeeeeeeeeeeeelee wwdule/Topoolooooo
aim investigate how
Introduction
Face recognition (FR) is critical biometric authentication technique that is widely applied in various applications. In recent years, convolutional neural networks (CNNs) have achieved remarkable success in FR task, thanks to their powerful ability to autonomously extract face features from images. Existing studies FR primarily focuses constructing developing margin- Recently, the success of unsupervised learning [10
discriminative face features by and powerful network architectures [6]
loss functions 2 3
7
g
14] and graph neural networks [5 16
17) has demonstrated the importance of data structure information in improving model generalization. However, the best of our knowledge, how effectively mine the potential structure information in large-scale face data has not investigated. Thus, in this paper, we extend our interests on building a cutting-edge FR framework through exploiting such powerful and substantial structure information.
First, we use Persistent Homology (PH) 20 21, a mathematical tool used in topological data analysis 22 to capture the underlying topological structure of complex point clouds, investigate
Equal Contribution, Corresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

155
140
172
18
Aud
omn
2011
1000 images
5000 images
10000 images
100000 images
(a)
(d)
Figure 1: We sample 1000 (a), 5000 (b), 10000 (c) and 100000 (d) face images from the MS1MV2 dataset respectively, and compute their persistence diagrams using PH, where H2 represents the j-th dimension homology. Persistence diagram (18) is mathematical tool to describe the topological structure of space, where the j-th dimension homology H, in persistence diagram represents the j-th dimension hole in space. In topology theory, if the number of high- sigh--imensionll holes in the space more, then the underlying topological structure of the space more complex [9]. As shown in Figure (d), as the amount of face data increases, the persistence diagram of the input space contains more and more high-dimensional holes (e.g., H3 and Ha). Therefore, this phenomenon demonstrates a growing complexity in the topological structure of the input space.
the evolution trend of structure information in existing FR framework and illustrate 3 interesting observations: (I) as the amount of data increases, the topological structure of the input space becomes more and more complex, as verified in Figures D (2) as the amount of data increases, the topological structure discrepancy between the input space and the latent space becomes increasingly larger, as verified in Figure 2a (3) The results Figure 2b demonstrate that as the depth of the network increases, the topological structure discrepancy becomes progressively smaller. This finding also provides explanation for why models with complex structure achieve higher FR accuracy. Based on the above observations, we can infer that FR tasks with large-scale datasets, the structure of face data will be severely destroyed during training, which limits the generalization ability of FR models in practical application scenarios. To this end, we propose to improve the generalization performance of FR models by preserving the structure information.
However, we experimentally find that directly using PH to align the topological structures of input space and the latent space may cause the model suffer from structure collapse phenomenon. Concretely, under this experimental setting, we have 2 following quantitative results: (1) As shown in Figure 2c] the topological structure discrepancy drops to 0 dramatically during early training. (2) As depicted in Figure 2d when evaluating on the IJB-C benchmark [23], there exists a significant structure information gap between the input space and the latent space. These typical overfitting phe- nomena indicate the latent space fails to preserve the structure information of input space accurately.
To remedy this issue, we propose superior FR model named TopoFR that leverages Perturbation- guided Topological Structure Alignment (PTSA) strategy to adequately preserve the topological structure information of the input space in corresponding latent face features. PTSA first employs random structure perturbation (RSP) mechanism perturb the latent space and effectively increase its structure diversity. Then PTSA utilizes invariant structure alignment (ISA) mechanism to align the topological structures of the original input space and the perturbed latent space, resulting in face features with stronger generalization ability
Moreover, in practical FR scenarios, the training dataset typically includes some low-quality face samples (i.e., hard samples) that are prone to being encoded into abnormal positions close to the decision boundary the latent space (24 6 26), significantly destroying the topological structure of the latent space and affecting the alignment of structure. To address this issue, we propose novel hard sample mining strategy named Structure Damage Estimation (SDE). SDE adaptively assigns structure damage score (SDS) to each sample based its prediction uncertainty and prediction

---

Faschen dO NNCEOR PISA
Bedt an Artthee +
Frcelet 30 Andece
Boriin Artfoct FTSA
Structure Discrejancy on 1MV2
Stricture Discrepancy on NS1M2
Number of Interations (r 10)
structure Biscrepancy ga-C
(b)
(C)
(d)
Figure 2: (a): We investigate the relationship between the amount of data and the topological structure discrepancy by employing ResNet-50 ArcFace model [T] to perform inferences on MSIMV2 training set. Inferences conducted for 1000 iterations with batch sizes of 256, 1024, and 2048, respectively. Histograms are used to approximate these discrepancy distributions. (b): We investigate the relationship between the network depth and the topological structure discrepancy by performing inference on MSI MSI training set (batch size=128) using ArcFace models with different backbones. (c): We investigate the trend of topological structure discrepancy during training (batch size=128) and found that i) directly using PH to align the topological structures will cause the discrepancy to drops to dramatically; ii) whereas using our PTSA strategy promotes a smooth convergence of structure discrepancy. (d): Aligning the topological structures directly using PH will lead to significant discrepancy when evaluating on IJB-C benchmark. Our PTSA strategy effectively mitigates this overfitting issue, resulting in smaller structure discrepancy during evaluation.
probability. By prioritizing the optimization of hard samples with significant structure damage, SDE can gradually guide these samples back their reasonable positions, thereby improving the generalization ability of FR model.
In summary, the main contributions are listed as follows:
1) To the best of knowledge, we are the first to explore the topological structure alignment in FR task. We propose novel topological structure alignment strategy called PTSA to effectively align the structures of the original input space and the perturbed latent space.
2) A novel hard sample mining strategy named SDE is introduced to mitigate the adverse impact of hard samples on the latent space structure.
3) Experimental results show that the proposed method outperforms SOTA methods on various face benchmarks. Notably, our TopoFR has secured the second place in the ICCV21 MFR-Ongoing challenge (77 until the submission of this work (May 22 '24, academic track): htth::::tttttttit/h/aaa # com/#/leaderboarce academic indicating the robustness and generalization of our method.
2
Related Works
28 29 have achieved remarkable Notably, the extraction of robust deep facial embeddings has raised considerable interest within the research community. Among them, CNNs framework are representative methods, using two primary methods: metric learning-based and margin- softmax approaches. The former utilizes loss functions like Triplet loss [7]. Tuplet loss [35], and Center loss [66] to learn discriminative face features, while the latter aims to incorporate margin penalty into the softmax loss framework, including methods such as ArcFace (1) CosFace [2], AM-sofimax [37], and SphereFace [38). Recent studies have explored
Face Recognition (FR), Convolutional Neural Networks (CNNs)
advancements in tasks related to facial recognition [3 31
35 34
various techniques, including adaptive parameters [3 mining 4
39] 40 learning acceleration
vision transformer architecture 9 8) and data uncertainty 255 6 26
to further enhance
models performance on large-scale datasets.
Persistent Homology (PH). Over the past decade, PH has shown significant advantages in multiple various such as signal processing 44). video analysis 44]
[46) neuroscience 48), disease diagnosis 49 and evaluation of embedding strategies [50 51 In the field of machine learning, some studies
52 53 54
have shown that integrating topological representations into network can enhance model's pecognition/ssemmetaaiii performance.
[55] proposes topology distance for the evaluation of GANs.
3

---

Mini-batch
Feature Extractor
Classifier
GausiaaBlur
Grayseale
Latent Features
Lila
Prediction
Entropy
Randona Calodditter
Random Strueture Perturbation
Probability
Re- Weight
Focal Loss
GUM
Invariant Structure Alignment
Sa
W2
W
SDS
Global overview of our proposed TopoFR. X represents the multiplication operation. denotes the probability of applying RSP to each training sample.
Figure 3:
3 Background: Persistent Homology
PH a computational topology method that quantifies the changes in the topological invariants of Vietoris-Rin complex as a scale parameter p is varied. In this section, we introduce some key concepts of PH. Further details on PH can be found in Refs. (20 [6].
Notation. X' == {x;) represents a point cloud and μ x x X - R denotes a distance metric over X. Matrix M represents the pairwise distances (i.e. Euclidean distance) between points in X.
Vietoris-Rige Complex. The Vietoris-Ripp complex [57] is a special simplicial complex constructed from set of points in metric space, and it be used to approximate the topology of the underlying space. For 0 S p < 0o, we represent the Vietoris Rips complex of point cloud X' at scale as V, (X)), which contains all simplices (i.e., subsets) of X, and each component of point cloud X' satisfies a distance constraint: p(z1, x;;) S for any i. Moreover, the Vietoris-Ripp complex satisfies nesting relation, i.e., Vo, Vo; for any P; S Pý. which allows us track the evolution progress of simplical complex the scale p increases. It is worth noting that (X)) and (MM) are equivalent because constructing the Vietoris-Ripe complex only requires distance.
Homology Group. The homology group 58 is an algebraic structures that analyzes the topological features of simplicial complex in different dimension j. such as connected components (Ho), cycles (H,), D. voids (H2), and higher-dimensional features (H,, 2 3). By tracking the changes in topological features (Hiz) of the Vietoris-Rips complex as the scale p increases, it is possible to gain insight into the multi-scale topological information of the underlying space.
Persistence Diagram and Persistence Pairing [59]. The persistence diagram is multi-set of points (b, d) in the Cartesian plane R², which encodes information about the lifespan of topological features. Specially, it summarizes the birth time b and death time d of each topological feature, where birth time b signifies the scale at which the feature is created and death time d refers to the scale at which it is destroyed. The persistence pairing 7 contains indices (i. corresponding to simplices rs E (>(') that create and destroy the topological features identified by (b, d) E DD respectively.
Methodology
In this paper, we propose novel framework named TopoFR for constraining the FR model to preserve the topological structure information of the input space in their latent features. The architecture of our TopoFR model is depicted in Figure It consists of two components: a feature extractor F and an image classifier C. Mathematically, given an input image r, the latent feature extracted by F is denoted f F(x) E TR' and the classification probability predicted by C is denoted as = C(J) E R where I represents the feature dimension and K denotes the number of classes. The entropy of the classification prediction probability can be represented as E(g) -k 9 loggg where g² is the probability of predicting sample to class k.

---

4.1 Perturbation-guidddd Topological Structure Alignment
As mentioned in Section | directly applying PH to align the topological structures of the input space and the latent space can cause the FR model to encounter structure collapse phenomenon. To remedy this problem, we propose a Perturbation-guin -ggided guided Topological Structure A Alignment (PTSA) strategy that includes two mechanisms: random structure perturbation and invariant structure alignment.
Random Structure Perturbation (RSP). PTSA first utilizes the RSP mechanism to randomly perturb the structure of the latent space. Specially, it utilizes a data augmentation list A [A1,Az A3, that includes four common data augmentation operations, namely Random Erasing A1, Gaussian Ag. Grayscale A3 and ColorJitter A1. For each training sample 24, RSP will randomly select an operation A, from A to perturb it, i.e., = A, (z,). Then the perturbed sample I; will be fed into the model for supervised learning, which effectively increases the structure diversity of the latent space. In our model, we adopt ArcFace loss 1 as the basic classification loss:
e=(cos(07 +m]]
Carc(T....) vo) = llog @=(coo(0? +m])
1k e cos
where y; the class label of the original image F;, s is scaling parameter, 6 is the angle between the k-th class center and feature, and m denotes an additive angular margin. During training, we apply the RSP mechanism to each sample z; with a probability & of 0.2.
Invariant Structure Alignment (ISA). Given a mini-batch of original training samples X' = {ax J denote the perturbed batch samples X-
{Ta) For the perturbed samples, we denote the latent features extracted by as 2 f i=1 During forward propagation, we can construct the Vietoris- Rips complexes (X') and (2) for point clouds X' and 2 respectively, based on their respective pairwise distance matrix M and M². Then we can utilize persistent homology to
analyze the topological structures of (x)) and V2)) and obtain their corresponding persistence diagrams {D* D2 and persistence pairings { , respectively.
Ideally, matter how the face image is perturbed, the position of the encoded face feature in the latent space should remain unchanged, and the topological structure of the perturbed latent space should also be consistent with the original input space. To this end, choose to align the original input space X with the perturbed latent space Z to achieve this goal. Prior studies usually utilize bottleneck distance or Wasserstein distance to measure the topological structure discrepancy [60] [8] between two spaces by comparing the differences in persistence diagrams. However, these two metrics are sensitive to outliers in persistence diagrams
(55 61 and will significantly increase models' training time, rendering them unsuitable for FR tasks with extremely large-scale datasets, as verified in Table in the Appendix.
To mitigate this issue, we turn retrieve the persistence diagrams values by subsetting the corre- sponding pairwise distance matrix with edge indices provided by the persistence pairings 162 63 64),
i.., Dx = [> and 2 2 [-ē]. By comparing the difference between two topologically relevant distance matrices from both spaces, we can quickly and stably compute the discrepancy
between their persistence diagrams, providing an efficient solution for structure alignment of FR models driven by large-scale datasets. We formulate the ISA loss as follows:
Lso (Dx,D3 ) (| 2 bx -
(2)
Notably, in the field of FR, most existing works do not include any data augmentation operations, as this would introduce more unidentifiable face images (i.e., destroying the fidelity of each face), which generally hurts the FR model's generalization ability, as verified in Refs. B 65). In this work, we do not employ these data augmentations to simply augment data scale. Instead, we use them to increase the latent space's structure diversity, effectively addressing the structure collapse problem. As a result, our PTSA strategy can reap the benefits of data augmentations while mitigating their potential negative effects (see Figure 2, Table 3 and Figure 5 for further analysis.).
5

---

4.2 Structure Damage Estimation
In practical FR scenarios, low-quality face samples, also known as "hard samples", are commonly included in the training set. These hard samples tend to be encoded in abnormal positions near the decision boundary in the latent space [25] 6 66 67], which will disrupt the latent space's s topoloicac structure and further hinder the aligmment of structures. To address this issue, we propose a novel hard sample mining strategy called Structure Damage Estimation (SDE). SDE is specifically designed to identify hard samples with serious structure damage within the training set accurately. By prioritizing the learning of these hard samples and guiding them back to the reasonable positions during optimization, SDE aims to mitigate the adverse impact of hard samples on the latent space's 's topological structure.
Prediction Uncertainty. Hard samples are typically distributed near the decision boundary, thus have a high prediction uncertainty (i.e., large entropy of the classifier prediction) and are more likely to be misclassified by the classifier [68] 69 70 71]. Conversely, easy samples are usually located far from the decision boundary and have relatively low prediction uncertainty. To model the difficulty of each sample, we introduce binary random variable E {0, 1 for each sample ã; to indicate whether the sample is hard or easy by values of and O, respectively. Then the probability that sample ã; belongs to hard samples (i.e., with large prediction uncertainty) can be defined as (T;)) P& (u4 1 where yp represents the parameter set. According to the cluster assumption [72 73 we believe that samples with higher prediction entropy are more disruptive the latent space's topological structure. Therefore, we propose to model the distribution of the entropy E(T:) ) for each training sample T. using the Ganssian-uniform mixture (GUM) model, statistical distribution that is robust to outliers [74 75 76):
p (E(G))) = xN* (E(G))) E) + (
(3)
where
[a Σ), a a22 0. 10,
N (a
a O.
((00.)) ?) is uniform distribution defined on [0, is prior probability, and Σ is the variance of Gaussian distribution Maao, Σ). In this mixed model, the uniform distribution term U and the Gaussian distribution term * respectively model the hard samples and easy samples. Then the posterior probability that the sample T; to be hard (i.e. high-uncertainty can be computed as follows:
(1- (1,,) he P. (u 1| TV+ (B (1- (0,
(5)
In Equation (5), when the classifier prediction is close uniform distribution or when the prediction probabilities for multiple classes are nearly equal, the posterior probability of a sample belonging to
hard data will be very high, i.e., (ā: + I K ... Hence, the prediction uncertainty of sample x; can be measured by a quantitative probability htp (T.).
h + 1), otherwise it is relatively low.
Assume ((A)) = (-1)* F(O.). B(1, 0.5), where B is Bernoulli distribution 77 then the
variable E(g+) obeys the following statistical distribution:
(T.))..) = T. Σ) + (1 n (-
In this way, the maximum likelihood model of Equation (6) can be formulated as: max (IG))) Then, the parameter set 9 {,, E, of GUM can be estimated via E, i=1 the Expectation-Maxiiizzate (EM) algorithm 788 with the following iterative formulas:
(1 - (- ((0))
((+1) ΣR (1
h
(++1)
(((+1)(F))) p
~ )(0 + +11 (() (-00))) 200)
x(41) Σ (1 1
(1(11)
(3r) (E(()))
024+1) = V3(m2 - -
Σn (1 n1 (1+1) (Tx))

---

where
1 E
(Z.) 1-x(++1) (E(g.))2 E
((ge
(1+ 1-m((+1)
1/1 Ei= hi (z.))
112
- - 16+ (i.))
Specifically, at each iteration, EM alternates between evaluating the expected log rigllikelhood (E-step) and updating the parameter set (M-step). In Equation (7), the E-step aims to evaluate the posterior probability h of an sample ž; to be hard sample using the iterative formula hig A({+1)/ (āz), where (f (++1) 1) denotes the EM iteration index. The M-step updates the parameter set using the iterative
formulas (1+11, ( and (2(++))
Structure Damage Score (SDS), Compared to correctly classified samples, misclassified samples usually have larger difficulty and have greater destructive effects on the latent space's topological structure. Therefore, misclassified samples need to receive more attention during training. Inspired by the Focal loss [79]. we design a probability-awan scoring mechanism w(z;) that combines prediction uncertainty h.(() and prediction accuracy adaptively compute SDS for each sample X;:
w (T:) wi (Z.)) x (Z:)) (1 + (T+))) (1 - (1-9.").
(8)
where À is temperature coefficient, and D.e represents the prediction probability of ground truth. Specifically, SDE assigns higher SDS to hard samples and lower SDS to easy samples, which effectively balances the contribution of each sample to the objective. By assigning higher scores to hard samples, the model is encouraged to focus more on learning these challenging samples, boosting the FR system's generalization. Formally, the SDS weighted classfication loss Lele is defined as:
Lole (ā+) Care y1)
During training, to minimize the objective Cole. the model needs to optimize both the SDS w and the loss Lare. which brings two benefits: (1) Minimizing Lare can encourage the model to capture face features with greater generalization ability from diverse training samples. (2) Minimizing SDS w can alleviate the damage of hard samples to the latent space's 's topological structure, which is beneficial to the preservation of topological structure information and the construction of clear decision boundary.
4.3 Model Optimization
To summarize, the overall objective of TopoFR can be formulated as follows:
min Lets a.
(10)
F.C
where a is hyper-parameter that balances the contributions of the loss Lots and the loss Lea. Detailed parameter sensitivity analysis can be found in Figure in the Appendix.
Experiments
5.1 Datasets.
i) For training, we employ three distinct datasets, namely MSIMV2 [1 (5.8M facial images, 85K
identities), Glint360K [41] (17.IM facial images, 360K identities), and WebFace42M (80) dataset (42.5M facial images, 2M identities). ii) For evaluation, we adopt LFW 8T. AgeDB-30 [82].
CFP-FP 833, CPLFW 84 CALFW 85 .JB-C (23). IJB-B (86) and the ICCV-2021 Masked Face Recognition Challenge (MFR-Ongoing) models.
27 as the benchmarks to test the performance of our
Notably, the MFR-Ongoing 27 is the most authoritative and comprehensive competition for evaluat- ing FR models' generalization performance. It contains not only the existing popular test sets, such as IJB-C, but also its own MFR benchmarks, such as Mask, Children, and Multi-Racia test sets. Due page size limitation, more training settings and experimental results are placed on Appendix.
5.2 Results on Mainstream Benchmarks
Results on MFR-Ongoing. We employ WebFace42M as training set, and compare our TopoFR with SOTA competitors MFR-Ongoing challenge, as reported in Table | For a fair comparison, all
7

---

Table 1: Verification accuracy (%) the MFR-Ongoii benchmark.
MFR
DB-C
Methed
Training Data
Venue
Mask Children African Caucasian South Asian Fast Asian MR-AIl
le-5 1e-4
R200, Partial PC 43
CVPR22
91.87
97.79
98.70
98.54
89.52
97.70
96.93 97.97
R200, UniFace (87
WehFace42M
ICCV23
92.43
93.11
98.14
98.98
98.84
90.01
97.92
96.68 97.91
R200, TopoFR
NeurlPS24
93.96
93.57
97.97
98.71
98.98
92.85
98.13
97.10 98.01
Table 2: Verification accuracy (%) LFW, CFP-FR, AgeDB-30, IJB-C and UBBB benchmarks.
IIB-C
UB-B
Training Data
Method
Venue
LFW
CFP-FP
AgeDB-30
Ic-3
Io-4
R50, ArcFace
CVPR19
99.68
97.11
97.53
88.36
92.52
91.66
R50, MagFace
CVPR21
99.74
97.47
97.70
88,95
93.34
91.47
R50, AdaFace
3
CVPR22
99.82
97.86
97.85
96.27
94.42
R50, TopoFR
NeurIPS24
99.83
98.24
98.23
94.79
96.42
95.13
R50. TopoFR
NeurIP224
99.83
98.2 28.24
98.25
94.71
96.49
95,14
R100, Costaac
2
CVPR18
99.78
98.26
98.17
92.68
95.56
94.01
R100. ArcFace
CVPR19
99.77
98.27
98.15
92.69
95.74
94.09
R100, MV-Soltmax
88
AAA120
99.80
98.28
97.95
95.20
93.60
R100, URL (65
CVPR20
99.78
98.64
95,00
96.60
R100, BroadFace
ECCV20
99.85
98.63
98.38
94.59
96.38
94.97
MSIMV2
R100, CurricularFace
CVPR20
99.80
98.37
98.32
96.10
R100, MagFacc+
S
CVPR21
99.83
98.46
98.17
94.08
95.97
94.51
R100, SCF-ArcFace
25
CVPR21
99.82
98.40
98.30
94.04
96.09
94.74
R100, DAM Curricularaace
500
ICCV21
96.20
95.12
R100, ElasticFace-Coss
91
CVPR22
99.80
98.73
98.28
95.43
R100, AdaFacc
3
CVPR22
99.82
98.49
98.05
96.89
95.67
TransFace-B 9
ICCV23
99.82
98.39
98.27
94.15
96.55
R100. TopoFR
NeurIPS24
99.85
98.83
98.42
95.28
96.96
95,70
R100, TopoFR
NeurlP$24
99.85
98.71
98.42
95.23
96.95
95.70
R200, Arcluce
CVPRT9
99.79
98.44
98.19
94,67
96.53
95.18
R200, AdaFace 3
CVPR22
99.83
98.76
98.28
94.88
96.93
95.71
TransFace-1
ICCV23
99.83
98.65
98.23
94.55
96.59
R200. TopoFR
NeurPS24
99.85
99.09
98.54
95.19
97.12
95577
R200, TopoFR
NeurlPS24
99.85
99.05
98.52
95.15
97.08
95.82
R50, ArcFace 1
CVPRT9
99.78
98,77
98.28
95.29
96.81
95,
R50, AdaFace 3
CVPR22
99.82
99.07
98.34
95.58
96.90
95.66
R50, TopoFR
NcurIP224
99.85
99.28
98.47
95.99
97.27
95.96
RI00, Arclaace
CVPR19
99.81
99.04
98.31
95.38
96.89
95.69
R100, AdaFace B
CVPR22
99.82
99,20
98.58
96.24
97.19
95.87
Glint360K
TransTace-B [
ICCV23
99.85
99.17
98.53
96.18
97.45
R100. TopoFR
NeurIPS24
99.85
99.43
98.72
96.57
97.60
96.34
R200, ArcPace
CVPR19
99.82
99.14
98.49
95.71
97.20
95.89
R200, Adulacce 3
CVPR22
99.83
99.24
98.61
95.96
97.33
96.12
TransFace-l.
ICCV23
99.85
99.32
98.62
96.29
97.61
R200, TopoFR
Neur1PS24
99.87
99.45
96.71
96.56
compared models adopt ResNet-200 as the backbone. Specially, our TopoFR surpasses the SOTA competitors UniFace and Partial FC in multiple metrics, implying the superiority of our method. Until the submission of this work (May 22 '24), the proposed TopoFR ranks second place on the academic track of the MFR-Ongen OFR-Onnoing leaderboard: http://iccv2:: http://www...........oeoo httmm////eedebbbrrddaaooo academidd
Results on LFW, CFP-FP and AgeDB-30. We adopt MSIMV2 and Glint360K to train our models, respectively. The results are reported in Table To showcase the universality of our method, we also provide detailed experimental results of TopoFR model trained by CosPace 2 As stated in Refs. 3 5), the performances of existing FR models on these three benchmarks have reached saturation. 1) On MS MV22 training set, we note that our TopoFR and TopoFR' models still obtain accuracy improvement and outperform SOTA competitors (e.g., AdaFace (3] and TransFace (9). 2) On Glint360K training set, our TopoFR become SOTA models and surpass AdaFace and TransFace.
Results on IJB-C and LJB-B. We train our TopoFR on MSIMV2 and Glint360K respectively. and compare with SOTA methods on JJBC and IJB-B benchmarks, as reported in Table 1) On MSIMV2 training set, our models obtain the best results under different backbones. For instance, our R50 TR and R50 TopoFR models greatly surpass SOTA method AdaFace and even beat most R R100-based competitors. 2) On Glint360K training set, all our models significantly outperform the cutting-ed edge competitor AdaFace and achieve SOTA performance. More importantly, our TopoFR works better than VIT based SOTA method TransFace, implying the superiority of our method.

---

173
correcty clas F barpa
175
cornccty clea sirsla miss ssalhol vemple
miccus find canple
000
0.0
Entropy
Entropy
Entropy
Entropy
(a) R50, MSIMV2
(b) R50, Glint360K
(c) R100, MSIMV2
(d) R100, Glint360K
Figure 4: The estimated Gaussian density (blue curve) w.F the entropy of classification prediction. Green marker and black marker represent the entropy of correctly classified sample and misclas- sified sample, respectively.
TopoFR
TopoFk
TopeFR
ToooFk
Structure Discrepancy UB-C (a) R100, MSIMV2
Structure Discrepancy (B-c
Structure Discrepancy (B-C
Stricture Discrepancy BB-C
(b) R200, MSIMV2
(c) R50, Glint360K
(d) R100, Glint360K
Figure 5:
The topological structure discrepancy of TopoFR and variant TopoFR-A under different backbones and training datasets (i.e., [Backbone, Training dataset]). Variant TopoFR-A directly utilizes PH to align the topological structures of two spaces. Notably, our TopoFR models trained with Glint360K dataset almost perfectly align the topological structures of the input space and the latent space on the IJB-C benchmark (i.e., the blue histogram almost converges a straight line).
5.3 Analysis and Ablation Study
Due to the limitation of page size, more ablation experiments and analysis are placed on Appendix.
1) Contribution of Each Component: To investigate the
contribution of each component in our model, we employ MS1MV2 as the training set, and compare ArcFace (baseline), and four variants of TopoFR on the IJB-C benchmark. The
Table 3: Ablation study.
Training Data
Method
IBB-C(le4)
variants of TopoFR are as follows: (1) TopoFR-R, the variant only adds RSP mechanism to ArcFace. (2) TopoFR-A. based on ArcFace, the variant simply aligns the structure of input space and latent space without using RSP. (3) TopoFR-P. the variant fully introduces the PTSA strategy into ArcFace. (4) TopoFR- based on TopoFR-P, the variant only uses prediction uncertainty w1 modeled by GUM to reiweight each sample. (5) TopoFR-F, based on TopoFR-P, the variant simply applies Focal loss W2 to re-weight each sample.
R50, Arce
92.52
R50, TopoFR
92.44
R50, TA
MSIMV2
R50, R.P
95.34
R50, TopoFR.P
95.40
R50, TopaFR
96.23
R50, TopoFR
96.49
The results gathered in Table some observations: (1) Compared with ArcFace, the accuracy of TopoFR-R R is clearly reduced due to the addition of more unidentifiable face images, which hurts the FR model's generalization ability. (2) TopoFR A outperforms ArcFace, indicating that directly aligning the two spaces can slightly boost model's performance, but it inevitably encounters structure collapse issue. (3) TopoFR-P greatly surpasses TopoFR-R and TopoFR-A, implying that preserving the structure information can greatly improve FR model's generalization.
(4) TopoFR outperforms To and TopoFR-G. GGG, which not only demonstrates the effectiveness of SDE strategy, but also indicates that the prediction uncertainty Wi is complementary Focal loss wy in mining hard samples.
To visually demonstrate the effectiveness GUM in mining hard samples, we present the estimated Gaussian density of the prediction entropy during training in Figure These curves show that the entropy of misclassified face samples (represented by black crosses) usually have rather low Gaussian density (i.e., high posterior probability h2),, thus be easily detected.
2) Effectiveness GUM:

---

Note that even if some misclassified samples have small entropy (i.e., high Gaussian density and low posterior probability hep). their SDS can still be corrected by the Focal loss w2.
3) Generalization of PTSA:
To show the superior generalization ability of our PTSA strategy in preserving structure information,
investigate the topological structure discrepancy between the
input and the latent spaces of TopoFR and its variant TopoFR-A on IJB-C benchmark. Note that Ao directly utilizes PH to align the topological structures of two spaces. The results in Figure 3 indicate that: 1) Directly using PH to align the topological structures of two spaces does effectively reduce the structure discrepancy, as the model encounters the structure collapse issue; 2) PTSA strategy can effectively align the topological structures of two spaces and address this structure collapse issue. Remarkably, Figures (aneee//sddaddddnn [d][5d]show that our TopoFR models trained on Glint360K almost perfectly preserve structure information of input spaces in their latent features, thereby verifying the generalization of PTSA strategy.
6
Conclusion
This paper proposes a novel FR framework called TopoFR that aims to encode the critical structure information in large-scale face dataset into the latent space, Specially, TopoFR leverages structure alignment strategy PTSA and a hard sample mining strategy SDE. PTSA employs PH to reduce the topological structure discrepancy between the input and latent spaces, effectively mitigating structure collapse phenomenon and preserving structure information. SDE accurately identifies hard samples and guides the model to prioritize optimizing these samples, mitigating their adverse impact on the latent space's structure. Comprehensive experiments validate the superiority of our TopoFR.
7 Broader Impacts
would be good to mention that the utilization of face images do not have any privacy concern given the datasets have proper license and users consent to distribute biometric data for research purpose. We address the well-defined face recognition task and conduct experiments publicly available face datasets. Therefore, the propose method does not involve sensitive attributes and we do not notice any negative societal issues.
References
[1] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4690 4699, 2019.
[2] Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5265-527777 2018.
[3] Minchul Kim, Anil K Jain, and Xiaoming Liu. Adaface: Quality adaptive margin for face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18750-18759, 2022.
[4] Yuge Huang, Yuhan Wang, Ying Tai, Xiaoming Liu, Pengeheng Shen, Shaoxin Li, Jilin Li, and Feiyue Huang. Curricularface: adaptive curriculum learning loss for deep face recognition. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5901-5910, 2020.
[5] Qiang Meng, Shichao Zhao, Zhida Huang, and Feng Zhou. Magface: A universal representation for face recognition and quality assessment. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2021.
[6] Jie Chang, Zhonghao Lan, Changmao Cheng, and Yichen Wei. Data uncertainty learning in face recognition. Proceedings of the IEFF/VE conference on computer vision and pattern recognition, pages 177109799, 2020.
10

---

[7] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 815-823, 2015.
[8] Yaoyao Zhong and Weihong Deng. Face transformer for recognition. arXiv preprint arXiv:2103. /4803, 2021.
[9] Jun Dan, Yang Liu, Haoyu Xie, Jiankang Deng, Haoran Xie, Xuansong Xie, and Baigui Sun. Transface: Calibrating transformer training for face recognition from data-centric perspective. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 20642-
20653, 2023.
[10] Wenjie Zhu and Bo Peng. Manifold-basse aggregation clustering for unsupervised vehicle re-identification. Knowledge-Baud Bassd Systems, 235:107622 2022.
[11] Fan Wu, Hao Ma, Yun Guan, and Lixia Tian. Manifold-based unsupervised metric learning, with applications in individualized predictions based on functional connectivity. Biomedical Signal Processing and Control, 79: 104081, 2023.
[12] Kejie Jiang, Qiang Han, Xiuli Du and Pinghe Ni. A decentralized unsupervised structural con- dition diagnosis approach using deep auto-encoders. Computer-Aided Civil and Infrastructure Engineering. 36(6):711 712-22, 2021.
[13] Leonardo Tadeu Lopes and Daniel Carlos Guimarães Pedronette. Self-supervised clustering based on manifold learning and graph convolutional networks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5634-5643, 2023.
[14] Jun Dan, Tao Jin, Hao Chi, Yixuan Shen, Jiawang Yu, and Jinhai Zhou. Homda: High-order moment-based domain alignment for unsupervised domain adaptation. Knowledge-Bil -aased Systems, 261:1 10205, 2023.
[15] Jun Dan, Weiming Liu, Mushui Liu, Chunfeng Xie, Shunjie Dong Guofang Ma, Yanchao Tan, and Jiazheng Xing. Hogda: Boosting semi-supervised graph domain adaptation via high-order structure-guided adaptive feature alignmen. In ACM Multimedia 2024, 2024.
[16] Chi-Chong Wong and Chi-Man Vong. Persistent homology based graph convolution network for fine-grained 3d shape segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7098-7107, 2021.
[17] Qitian Wu, Chenxiao Yang, Wentao Zhao, Yixuan He, David Wipf, and Junchi Yan. Dif- former: Scalable (graph) transformers induced by energy constrained diffusion. arXiv preprint arXiv:2301.09474, 2023.
[18] Yuriy Mileyko, Sayan Mukherjee, and John Harer. Probability measures on the space of persistence diagrams. Inverse Problems, 27(12):124007, 2011.
[19] Afra Zomorodian and Gunnar Carlsson. Computing persistent homology. In Proceedings of the twentieth annual symposium on Computational geometry, pages 347-356, 2004.
[20] Herbert Edelsbrunner, John Harer, et al. Persistent homology-a a survey. Contemporary mathe- matics, 453(2)))257-28) 2008.
[21] Serguei Barannikov. The framed morse complex and its invariants. Advances in Sovier Mathematics, 21:93-116, 1994.
[22] Gunnar Carlsson. Topology and data. Bulletin the American Mathematical Society, 46(2):255- 308, 2009.
[23] Brianna Maze, Jocelyn Adams, James Dunean, Nathan Kalka, Tim Miller, Charles Otto, Anil K Jain, W Tyler Niggel, Janet Anderson, Jordan Cheney, et al. Tarpa janus benchmark-c: Face dataset and protocol. In 2018 international conference on biometrics (ICB), pages 155-165
IEEE, 2018.

---

[24] Jun Dan, Tao Jin, Hao Chi, Shunjie Dong, and Yixuan Shen. Uncertainty guided joint unbal- anced optimal transport for unsupervised domain adaptation. Neural Computing and Applica- lions, 35(7):5351-554)
2023.
[25] Shen Li, Jianqing Xu, Xiaqing Xu, Pengcheng Shen, Shaoxin Li, and Bryan Hooi. Spherical confidence learning for face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15629-15633 2021.
[26] Yichun Shi and Anil K Jain. Probabilistic face embeddings. In Proceedings of the the JEEE/CVF International Conference on Computer Vision, pages 6902-6911 6222-691,, 1, 2019.
[27] Jiankang Deng, Jia Guo, Xiang An, Zheng Zhu, and Stefanos Zafeiriou. Masked face recogni- tion challenge: The insightface track report. In Proceedings of the IEEF/CVE International Conference Computer Vision, pages 1437-1444, 2021.
[28] Mushui Liu, Yunlong Yu, Zhong Ji, Jungong Han, and Zhongfei Zhang. Tolerant self-distillation for image classification. Neural Nerworks, 2024.
[29] Mushui Liu, Fangtai Wu, Bozheng Li, Ziqian Lu, Yunlong Yu, and Xi Li. Envisioning class entity reasoning by large language models for few-shot learning, In AAAI, 2025.
[30] Yang Liu, Fei Wang, Jiankang Deng, Zhipeng Zhou, Baigui Sun, and Hao Li. Mogface: Towards deeper uppreciation face detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1093-41022 2022.
[31] Yang Liu, Jiankang Deng, Fei Wang, Lei Shang, Xuansong Xie, and Baigui Sun. Damofd: Digging into backbone design face detection. In The Eleventh International Conference Learning Representations, 2022.
[32] Yang Liu, Xu Tang. Junyu Han, Jingtuo Liu, Dinger Rui, and Xiang Wu. Hambox: Delving into mining high-quality anchors on face detection. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13043-13051 IEEE, 2020.
[33] Yang Liu and Xu Tang. Bfbox: Searching face-appropriate backbone and feature pyramid network for face detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13568-13577, 2020.
[34] Jiankang Deng. Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. Retinaface: Single-shot multi-level face localisation in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5203-5212, 2020.
[35] Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29, 2016.
[36] Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning ap- proach for deep face recognition. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VII 14, pages 499-515. Springer, 2016.
[37] Feng Wang, Jian Cheng, Weiyang Liu, and Haijun Liu. Additive margin softmax for face verification. IEEE Signal Processing Letters, 25(7):926 00066,9902220 2018
[38] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 212 22- 2017.
[39] Xingkun Xu, Yuge Huang. Pengcheng Shen, Shaoxin Li, Jilin Li, Feiyue Huang, Yong Li, and Zhen Cui. Consistent instance false positive improves fairness in face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 578-586, 2021.
[40] Jiankang Deng, Jia Guo, Tongliang Liu, Mingming Gong, and Stefanos Zafeiriou. Sub-center arcface: Boosting face recognition by large-scale noisy web faces. In Computer ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pages 741-757. Springer, 2020.
12

---

[41] Xiang An, Xuhan Zhu, Yuan Gao, Yang Xiao, Yongle Zhao, Ziyong Feng, Lan Wu, Bin Qin, Ming Zhang. Debing Zhang, et Partial fc: Training 10 million identities on a single machine. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1145-1449 2021.
[42] Pengyu Li, Biao Wang, and Lei Zhang. Virtual fully-connected layer: Training large-scale face recognition dataset with limited computational resources. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 13315-13324, 2021.
[43] Xiang An, Jiankang Deng, Jía Guo, Ziyong Feng, XuHan Zhu, Jing Yang. and Tongliang Liu. Killing two birds with one stone: Efficient and robust training of face recognition cnns by partial fc. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4042-4051, 2022.
[44] Jose A Perea and John Harer. Sliding windows and persistence: An application of topological methods to signal analysis. Foundations of Computational Mathematics, 15:799-8833 888, 2015.
[45] Christopher Tralie and Jose Perea. (quasi) periodicity quantification in video data, using topology. SIAM Journal on Imaging Sciences, 11(2):044-11077 2018.
[46] Mushui Liu, Bozheng Li, and Yunlong Yu. Omniclip: Adapting clip for video recognition with spatial-temporal omni-scale feature learning. ECAI, 2024.
[47] Gurjeet Singh, Facundo Memoli, Tigran Ishkhanov, Guillermo Sapiro, Gunnar Carlsson, and Dario L. Ringach. Topological analysis of population activity in visual cortex. Journal of vision, 888):11-11, 2008
[48] Yuri Dabaghian, Facundo Mémoli, Loren Frank, and Gunnar Carlsson. A topological paradigm for hippocampal spatial map formation using persistent homology. 2012.
[49] Yu-Min Chung, Chuan-Shen Hu, Austin Lawson, and Clifford Smyth. Topological approaches to skin disease image analysis. 2018 IEEE International Conference on Big Data (Big Data), pages 100-105. IEEE, 2018.
[50] Bastian Rieck and Heike Leitte. Persistent homology for the evaluation of dimensionality reduction schemes. In Computer Graphics Forum, volume 34, pages 431-440 Wiley Online Library, 2015.
[51] Bastian Rieck and Heike Leitte. Agreement analysis of quality measures for dimensionality reduction. In Topological Methods in Data Analysis and Visualization IV: Theory, Algorithmss, and Applications VI, pages 103-117. Springer, 2017.
[52] James Clough, Nicholas Byrne, Ilkay Oksuz, Veronika A Zimmer, Julia A Schnabel, and Andrew P King. A topological loss function for deep-learning based image segmentation using persistent homology. IEEE Trunsactions on Puttern Analysis and Machine Intelligence, 44(12):8766-8778 2020.
[53] Ekaterina Khramtsova, Guido Zuccon, Xi Wang. and Mahsa Baktashmotlagh. Rethinking persistent homology for visual recognition. In Topologicul, Algebraic and Geometric Learning Workshops 2022, pages 206-215. PMLR, 2022.
[54] Vinay Venkataraman, Karthikeyan Natesan Ramamurthy, and Pavan Turaga. Persistent ho- mology of attractors for action recognition. In 2016 IEEE international conference on image processing (ICIP), pages 4150-4154. IEEE, 2016.
[55] Danijela Horak, Simiao Yu, and Gholamreza Salimi-Khorshinn Topology distance: topology- based approach for evaluating generative adversarial networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7721-7728, 2021.
[56] Herbert Edelsbrunner, David Letscher, and Afra Zomorodian. Topological persistence and simplification. In Proceedings 41 41st annual symposium on foundations of computer science, pages 454-463. IEEE, 2000.
13

---

[57] Leopold Vietoris. Über den höheren zusammenhang kompakter räume und eine klasse von zusammenhangstreuen abbildungen. Mathematische Annalen, 97(1):454-47) 472, 1927.
[58] Herbert Edelsbrunner. Persistent homology: theory and practice. 2013.
[59]
David Cohen-Steiner, Herbert Edelsbrunner, and John Harer. Stability of persistence diagrams. In Proceedings of the twenty-first annual symposium on Computational geometry, pages 263-
271, 2005.
[60] Katharine Turner, Yuriy Mileyko, Sayan Mukherjee, and John Harer. Fréchet means for distributions of persistence diagrams. Discrete & Computational Geometry, 52:44-70, 70, 2014.
[61] Dimitri Bertsekas, A new algorithm for the assignment problem. Mathematical Programming. 21(1):152-171, 1981.
[62] Afrr Zomorodian. Fast construction of the vietoris-rips complex. Computers & Graphics, 34(3):263-27,, 2010.
[63] David Cohen-Steiner, Herbert Fdelsbrunner, and John Harer. Extending persistence using poincaré and lefschetz. duality. Foundations of Computational Mathematics, ((1):79-133, 2009.
[64] Michael Moor, Max Horn, Bastian Rieck, and Karsten Borgwardt. Topological autoencoders. In International conference on machine learning, pages 7045-7054. PMLR, 2020.
[65] Yichun Shi, Xiang Yu, Kihyuk Sohn, Manmohan Chandraker, and Anil Jain. Towards universal representation learning for deep face recognition. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6817-6826, 2020.
[66] Jun Dan, Tao Jin, Hao Chi, Shunjie Dong, Haoran Xie, Keying Cao, and Xinjing Yang. Trust- aware conditional adversarial domain adaptation with feature norm alignment. Neural Networks, 168:518-555 500,,5300500000000 2023.
[67] Weiming Liu, Chaochao Chen, Xinting Liao, Mengling Hu, Jiajie Su, Yanchao Tan, and Fan Wang. User distribution mapping modelling with collaborative filtering for cross domain recommendation. Proceedings of the ACM on Web Conference 2024, pages 334-343 2024.
[68] Ziqian Lu, Fengli Shen, Mushui Liu, Yunlong Yu, and Xi Li. Improving zero-shot generalization for clip with variational adapter. ECCV, 2024.
[69] Weiming Liu, Xiaolin Zheng, Chaochao Chen, Jiajie Su, Xinting Liao, Mengling Hu, and Yanchao Tan. Joint internal multi-interest exploration and external domain alignment for cross domain sequential recommendation. In Proceedings of the ACM Web Conference 2023, pages 383-394, 2023.
[70] Weiming Liu, Xiaolin Zheng, Jiajie Su, Longfei Zheng, Chaochao Chen, and Mengling Hu. Contrastive proxy kernel stein path alignment for cross-domain cold-start recommendation. IEEE Transactions on Knowledge and Data Engineering, 35(11):1 1230, 2023.
[71] Weiming Liu, Xiaolin Zheng, Jiajie Su, Mengling Hu, Yanchao Tan, and Chaochao Chen. Exploiting variational domain-invariant user embedding for partially overlapped cross domain recommendation. In Proceedings of the 45th International ACM SIGIR conference on research and development information retrieval, pages 312-321, 2022.
[72] Olivier Chapelle and Alexander Zien. Semi-supervised elassification by low density separation.
In International workshop on artificial intelligence and statistics, pages 57-64. PMLR, 2005.
[73] Weiming Liu, Xiaolin Zheng, Mengling Hu, and Chaochao Chen. Collaborative filtering with attribution alignment for beasew non-overlapped cross domain recommendation. In Proceedings of the ACM web conference 2022, pages 181-1190, 2022.
[74] Alessio De Angelis, Guido De Angelis, and Paolo Carbone. Using gaussian-uniform mixture models for robust time-interval measurement. IEEE Transactions on Instrumentation and Measurement, 64(12):3545-353 2015.
14

---

[75] Stéphane Lathuilière, Pablo Mesejo, Xavier Hameda-Pinedee and Radu Horaud. Deepgum: Learning deep robust regression with a gaussian caussian-umiform mixture model. In Proceedings of the European Conference on Computer Vision (ECCV), pages 202-217, 2018.
[76] Xiang Gu, Jian Sun, and Zongben Xu. Spherical space domain adaptation with robust pseudo- label loss. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 9101-9110, 10, 2020.
[77] Bin Dai, Shilin Ding, and Grace Wahba. Multivariate bernoulli distribution. 2013.
[78] Pietro Coretto and Christian Hennig. Robust improper maximum likelihood: tuning, compu- tation, and a comparison with other methods for robust gaussian clustering. Journal of the American Statistical Association, 11 (6516))1664-1666 2016.
[79] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection, Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[80] Zheng Zhu, Guan Huang. Jiankang Deng, Yun Ye, Junjie Huang, Xinze Chen, Jiagang Zhu, Tian Yang, Dalong Du, Jiwen Lu, et al. Webface260m: A benchmark for million-scale deep face recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(2):2627-26444 2022.
[81] Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Miller. Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In Workshop faces in RefllLiffe Images: detection, alignment, and recognition, 2008.
[82] Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou. Agedb: the first manually collected, in-the-wild wild age database. In proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 51-59, 2017.
[83] Soumyadip Sengupta, Jun-Cheng Chen, Carlos Castillo, Vishal M Patel, Rama Chellappa, and David W Jacobs. Frontal to profile face verification in the wild. In 20/6 IEEE winter conference on applications of computer vision (WACV), pages 1-9, IEEE, 2016.
[84] Tianyue Zheng and Weihong Deng. Cross-pose Ifw: A database for studying cross-pose face recognition in unconstrained environments. Beijing University of Posts and Telecommunications, Tech. Rep, 5(7):5, 2018.
[85] Tianyue Zheng, Weihong Deng, and Jiani Hu. Cross-age Ifw: A database for stuyying cross-age face recognition in unconstrained environments. arXiv preprint 1708.088 2017.
[86] Cameron Whitelam, Emma Taborsky, Austin Blanton, Brianna Maze, Jocelyn Adams, Tim Miller, Nathan Kalka, Anil K Jain, James A Duncan, Kristen Allen, al. larpa janus benchmark- b face dataset. In proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 90-98, 2017.
[87] Jiancan Zhou, Xi Jia, Qiufu Li, Linlin Shen, and Jinming Duan. Uniface: Unified cross-entropy loss for deep face recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 40730-20732 2023.
[88] Xiaobo Wang, Shifeng Zhang, Shuo Wang, Tianyu Fu, Hailin Shi, and Tao Mei. Mis-classified vector guided softmax loss for face recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12241-12248, 2020.
[89] Yonghyun Kim, Wonpyo Park, and Jongju Shin. Broadface: Looking at tens of thousands of people at once for face recognition. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX, pages 536-552, Springer, 2020.
[90] Jiaheng Liu, Yudong Wa, Yichao Wu, Chuming Li, Xiaolin Hu, Ding Liang, and Mengyu Wang. Dam: Discrepancy alignment metric for face recognition. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 3814-3823, 2021.
15

---

[91] Fadi Boutros, Naser Damer, Florian Kirchbuchner, and Arjan Kuijper. Elasticface: Elastic margin loss for deep face recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR) Workshops, pages 1578-1587, June 2022.
[92] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[93] Yuanqing Huang, Yinggui Wang, Le Yang, and Lei Wang. Enhanced face recognition using intra-class incoherence constraint. In The Twelfth International Conference on Learning Representations, 2023.
[94] Sheng Chen, Yang Liu, Xiang Gao, and Zhen Han. Mobilefacenets: Efficient cnns for accurate real-time face verification on mobile devices. In Biometric Recognition: 13th Chinese Con- ference, CCBR 2018, Urunqi, China, August 11-12, 2018, Proceedings 13, pages 428 438. Springer, 2018
[95] Yanbo Fan, Siwei Lyu, Yiming Ying, and Baogang Hu. Learning with average top-k loss. Advances neural information processing systems, 30, 2017.
[96] Anirudh Som, Kowshik Thopalli, Karthikeyan Natesan Ramamurthy, Vinay Venkataraman, Ankita Shukla, and Pavan Turaga. Perturbation robust representations of topological persistence diagrams. In Proceedings of the European Conference on Computer Vision (ECCV), pages 617-635, 2018.
[97] Matthew Kahle. Random geometric complexes. Discrete Computational Geometry, 45:553- 573, 2011.
[98] Omer Bobrowski and Matthew Kahle. Topology of random geometric complexes: a survey. Journal of applied and Computational Topology, 1:331-364, 2018
16

---

A Appendix
A.1 Implementation Details
Training Details. For MSIMV2 and Glint360K, models are trained using Pytorch 4 NVIDIA Tesla A100 GPUs, and a mini-batch of 128 images is assigned for each GPU. In the case of WehFace42M, train models (ResNet-200 backhone) using 64 NVIDIA Tesla V100 GPUs. We crop all images to 11 12 12, following the same setting as in ArcFace I 34
For the backbones,
we adopt ResNet-50, ResNet-100 and ResNet-200 [92] modified in M. We follow [ to employ ArcFace (s = 64 and m = 0.5) as the basic classification loss to train the TopoFR model. For the TopoFR' model trained by CosFace 22. we set the scale 64 and the cosine margin m of CosFace to 0.4. To optimize the models, we use Stochastic Gradient Descent (SGD) optimizer with momentum of 0.9 for both datasets. The weight decay for MSIMV is set 5e-4 and le-4 for Glint360K. The initial learning rate is set to 0.1 for both datesets. In terms of the balance coefficient a, we choose a 0.1 for experiments on R50 TopoFR, and 0.05 for experiments on R100 TopoFR and R200 TopoFR. During training, we apply RSP mechanism with a certain probability. Specially, for an original input sample x, the probability of undergoing RSP is Ç. and the probability of it remaining unchanged g- For the aeper-parameter ç, we choose 0.2. Notably, in our method, we focus on preserving the O-dimension homology Ho in the topological structure alignment loss Lea. Because preliminary experiments demonstrated that using the l-dimension or model's training time without clear performance gains. Code and pre-trained models are available at: https://gi cww httpe//eeeeceeeeeaccaaaaa/a/eeae
homology only increases
Evaluation Protocol on MFR-Ongoing Challenge. In the MFR-Ongoing Challenge, the trained models are submitted to and evaluated by the online server. Specially, "TAR @ FAR=1cc44 and "TAR @ FAR=le-5" are reported on the IJB-C. Furthermore, we report "TAR @ FAR=le-4 for Mask and Children test sets, and "TARR @ FAR=1e-6" for MR-ALL test set. The academic track of the ongoing MFR challenge leaderboard can be found onhttpp:/iccv21: mfr. com/ /eadeebbbrrdd///// academic More details about the MFR-Onoging Challenge can be found in Ref. [27].
A.2 More Results
A.2.1
Experiments Other Benchmarks
Results on CPLFW and CALFW. We utilize MS1MV2 as the training set, and compare our TopoFR with SOTA methods on CPLFW and CALFW benchmarks, as reported in 4 As can be seen, the proposed R50 TopoFR and R50 TopoFR greatly outperform the SOTA competitor R50 AdaFace on two benchmarks. Furthermore, under the ResNet- backbone, our TopoFR also achieves SOTA performance and surpasses AdaFace by 0.26% and 0.36% on CPLFW and CALFW respectively.
Table Verification accuracy (%) on CPLFW and CALFW.
Training Data
Method
Venue
CPLFW
CALFW
RS0, ArcFace
CVPRT9
91.76
95.14
R50, MagFace
3
CVPR21
92.49
95.88
R50, AdalFace
CVPR22
92.83
R50, TopoFR
NeurlPS24
93.36
96.24
R50. Topol'R
NeurIPS24
93.38
96.25
RI00. CosFace
2
CVPRIS
V2.26
95.75
1000 ArcFace
i
CVPR19
92.10
95.47
R100, MV-Softmax
88
AAA120
92.83
96.10
MSIMV2
R100. BroadFace 89
ECCV20
93.17
96.
R100, CurricularFace
CVPR20
93.13
96.20
R100, MagFace
CVPR21
92.87
96.15
1100, ArcFae 25
CVPR21
93.16
96.12
R R100, ElasticFace-Are- 91
CVPR22
93.28
96.17
R100, ElasticFace-Cors 91
CVPR22
93.23
96.18
R100, AdaFace B
CVPR22
93.53
96.08
R100, IIC-Adaface
ICLR24
93.48
1100. TopoFR
NeurIPS24
93.78
96.42
1100, TopoFR
NeurIPS24
93.79
96.44
17

---

A.2.2 Experiments on Light-weight Network
To demonstrate the universality of our method, we conduct some experiments on lightweight network MobileFaceNeet0.4555 [94), as shown in Table 5 We can observe that with the help of SDE and PTSA strategies, the MobileFaceNNet-O..4 implying the effectiveness and universality of our method.
model achieve higher recognition accuracy,
Table 5: Verification performance (%) on UB-C. MobileFaceNet refers to the MobileFaccNNet-04556 backbone
94
Training Data
Method
IJB-C (1e-6)
(1e-()
UB-C
MS IMV
FaciecFaccNet
81.75
90.13
93.42
MS MS
MobileFaceNet PTSA SDE
83.14
91.01
94.48
Glimt360K
MohileFaceNet
83.67
92.49
94.86
Glint360K
MobileFaceNett PTSA SDE
85.44
93.35
95.79
A.3 More Ablation Experiments and Analysis
1) Does Structure Information Improve Intra-class and Inter-class Relationships?: We conduct two pairs of ablative experiments to validate the relationships between topological structure alignment and intra-class distance constraint as well as inter-class distance constraint. The results are gathered in Table
We can find that integrating topological structure alignment with single inter-class or intra-class distance constraint both obtain additional significant performance gains. This indicates that topological structure alignment can provide the extra structure information of intra-class and inter- class relationships, thereby establishing clearer decision boundaries. Overall, these improved results demonstrate topological structure alignment implicitly encourages intra-class compactness and inter-class separability in the deep feature space.
Table 6: Verification performance (%) on IJB-C. Relationships between Topological Structure Information and Intra-class/ Inter-class constraints.
Training Data
Method
UJB-C (1e-6)
UB-C
DBBC (le-4)
MSTMV2
R100, ArcFace (w/o intra-class)
82.37
89.13
94,16
R100, ArcFace(w/o intra-class) topological constraint
83.69
91.48
94.86
MSIMV2
R100, ArcFace((lo inter-class)
82.72
89.51
94.47
R100, ArcFace(w/o inter-class) + topological constraint
84.53
92.36
95.12
2) Comparison with Previous Hard Sample Mining Strategies: To further demonstrate the superiority of our SDE strategy in mining hard samples, we compare it with existing hard sample mining strategies, including MV-Softmax proposed EHSM [9]
88] AT (95 loss, Focal loss 77 and the recently
The results gathered in Table 7 We observe that the proposed SDE strategy elearly outperforms than previous hard sample mining strategies, indicating that our SDE strategy is better able to measure sample difficulty and improve the model's generalization performance. This is because the SDE comprehensively considers the prediction uncertainty and label information (i.e., prediction probability of ground truth) when mining hard samples.
3) Effectiveness of ISA: Previous works often use Bottleneck distance and p- D-sssersteinn dis- tance to measure the distance between persistence diagrams x and Dē 18 60 the Bottleneck distance is defined (Dx D inf "=+=>>>>>> suP || xla) with g. ranging over all bijections between sets of persistent intervals in diagrams Dx and D² and I1-11oo denotes the -o-norm Equivalently, the p- p-Wasserstein distance is defined as (D* _D²) (inl K; D Σ=epx || - x( (())) However, the Bottleneck distance metric and p- Wasseesstein distance metric are sensitive to outliers (55 61 96 and will significantly increase the training time of FR models. This makes them unsuitable for tasks with extremely large-scale datasets.
Concretely,
18

---

Table Comparison with Previous Hard Sample Mining Strategies.
Training Data
Method
UB-C(le-6)
IBB-C
UBBCC (1c-4)
R100, ArcFace
85.65
92,69
95.74
R100, ArcFace + A
85,93
93.04
95.89
R100, ArcFace + MV-Softmax
86.47
93,38
95.94
MSIMV2
R100, ArcFace + Focal Loss
87.58
93.87
96.11
R100. ArcFace EHSM
88.30
94.19
96.18
R100, ArcFace SDE
89.23
94.65
96.49
Table 8: Comparison of TopoFR using Different Metrics.
Training Data
Method
Average Training Time Epoch
IJB-C (1e-4)
R 1000, TopoFR-B
4312.56s
96.76
MSIMV2
R100, TopoFR. W R100, TopoFR
4127.299
96.81
2729.28s
96.95
To demonstrate the stability and computational efficiency of our ISA strategy, we employ MS1MV2 as the training set, and compare TopoFR and its two variants the IJB-C benchmark. The variants of TopoFR are as follows: (1) TopoFR-B, the variant uses Bottleneck distance to compute the discrepancy between persistence diagrams. (2) TopoFR-W. the variant adopts I-Wasserstein distance to measure the discrepancy between persistence diagrams.
We provide the average training time per epoch of these models and their recognition performance on "TAR 1e-= of the IJBC benchmarks. The results presented in Table & reflect the following observations: (1) TopoFR outperforms TopoFR-B --B and TopoFR-W W on "TAR @ FAR=Ic-4", indicating the robustness of our ISA to outliers. (2) Compared with TopoFR- and TopoFR-W W. our proposed TopoFR has shorter training time, demonstrating the computational efficiency of ISA.
Effeet of Batch Size: We investigate the model's performance under different batch size, and the results are gathered in Table 9
We find that increasing the batch size does not lead a significant improvement in model's recognition accuracy. Additionally, larger batch size imposes a greater workload on GPUs. Therefore, to strike a balance between model's accuracy and and GPU computational load, we choose the batch size to 128 in our TopoFR model.
Table 9: The Performance of TopoFR Model Under Different Batch Size.
Training Data
Method
Batch Size
IJB-C (le-5)
IJB-C (le-4)
R100, TopoFR
128
95.23
96.95
MSIMV2
R100, TopoFR
256
95.22
96.91
R100, TopoFR
512
95.20
96.93
For detailed training time analysis, please refer to the Table 10 Due to the introduction of the strueture alignment strategy PTSA and hard sample mining strategy SDE, TopoFR models require longer training time (1.16x). Specially, compared to the vanilla R50 ArcFace model, our R50 TopoFR model requires about 2 seconds more training time per 100 steps, which does not significantly increase the training time but brings a large performance gain. And our 100 topoFR model requires about seconds extra training time per 100 steps than the vanilla R100 ArcFace model, which is not major increase in traing time but leads to significant performance ad advantage.
5) Training Time:
While for inference computation head, our method performs consistently with that of vanilla ArcFace model, since we adopt the same network architecture and data pre-process module.
Moreover, the results in Table 10 indicate that introducing SDE strategy (i.e., GUM) leads to significant performance improvements with only a small increase training time (i.e., R50 backbone: 0.2s / 100 steps. R 100 backbone: 0.25s/ / 100 steps), which is reasonable. Therefore, the addition of GUM does not bring too much computational burden and does not significantly increase the training time.
19

---

Table 10: Detailed Training Time of FR models.
Training Data
Metholl
Average Training Time/ 100 steps
Avenage Triining Time / Eaoch
UBBC (le-4)
R50, ArcPace
15.335
1745.925
92.52
MSIMV2
R50, ArcFace PTSA
17.57s
1999.196
96.25
R50, ArcFace PTSA SDE (R50, TopoFR)
17.77s
2020 900
96.49
KT00, ArcPace
20,16s
225/2.594 596
55.74
MSIMV2
R100, ArcFace PTSA
23,10s
2626.93s
96.68
R100, ArcFace PTSA (R100, TopoFR)
23.35s
2655.36x
6) Parameter Sensitivity: To demonstrate the effect of the paper-parameters a (i.e., the balance coefficient of the ISA loss L.a) and & (i.e., the probability of RSP mechanism), conduct additional experiments by setting different values of a and &. respectively. We use MSIMV2 dataset to train the R50 TopoFR and R 100 TopoFR models, and evaluate their performance the IJB-C benchmark. The results on MAR@FAR=1c-44 are shown in Figures 6a and 6b
We can observe that increases, the accuracy first rises and then falls. This is because an overly large a will cause the model to focus more on the structure alignment and less on the classification learning of the samples during training. Additionally, setting the parameter & to high value may introduce more unidentifiable face images and severely disturb the topological structure of the latent space, making it difficult for the ISA loss function L.a to converge.
Verification Accuracy (%6
IJB- (1e-4)
IJB (1e-4)
97.4
TopoFR
97.4
RSO TapoFR
97.2
R100 TopoFR
R100 TopoFR
97.0
97.0
96.8
96.8
Verification
96.4
96.4
96.2
96.2
96.0
96.0
0.01
0.05
0.1
0.15
0.2
0.25
0.1
0.2
0.3
0.4
0.5
(a)
(b) E
Figure 6: Parameter sensitivity analysis. (a) The effect of the hyper- (b) The effect of the hyper-parameter E.
7) Comparison of Structure Discrepancy: To further demonstrate the effectiveness of our PTSA strategy in preserving structure information, we utilize the Bottleneck distance metric (60 [8] to investigate the topological structure discrepancy between the input space and the latent space of ArcFace and TRpoFR on IJB-C benchmark, as illustrated Figure
We can observe that our TopoFR model significantly reduces the structure discrepancy (i.e., measured by the Bottleneck distance metric) between two spaces compared the vanilla ArcFace model, and effectively preserves the structure information hidden in the large-sclae dataset.
8) Visualization of Hard Samples: We conduct visual comparison experiment to visualize some hard samples that can be correctly classified by our method but cannot be correctly classified by existing method.
The visualization results illustrated Figure reffect the following observations: (1) Hard samples are usually blurry, low-contrast, occluded, or in unusual poses, so they are easily misclassified by existing method such as ArcFace model. (2) The ArcFace model assigns equal weight (i.e., 1) to each sample. While TopoFR model utilizes SDE strategy to adaptively assign weight (i.e., SDS to each sample based on its prediction uncertainty W1 and prediction accuracy w2. Specially, SDE will assign higher SDS w to hard samples, which can encourage the model to extract robust face features from these challenging samples, thereby effectively improving the model's generalization performance.
9) Is the topological structure constructed in the input space robust enough? i) In the input space, we first flatten the face images into vector features and then calculate their pair-wise distance matrix in order to construct Vietoris-Rips complex. And the dimension of face features in the pixel space is significantly higher than that of the features in the latent layer space. Notably, the expected k-th
20

---

ArcFace
100
TopoFR
80
Histogram
20
20
40
60
Bottleneck Distance on IJB-C
Figure 7: The topological structure discrepancy (i.e., measured by the Bottleneck distance metric) of R50 TopoFR and R50 ArcFace on the LJB-C benchmark.
Misclassified
Correctly Classified
ArcFace
TopoFR
SDS w
1.64
1.75
1.48
1.41
1.52
1.37
Figure 8: Visualization of hard samples.
Betti number E [r)] = c ( (:: data size, d: data dimension, cg: constant) [97 98 in topology theory also demonstrates that as long as the dimension of the data is sufficiently high, its underlying topological structure can be well constructed. Therefore, this can effectively capture the topological structure information hidden in the large-scale face datasets. ii) More importantly, during training, our RSP mechanism also effectively simulates the influence of multiple factors, such as lighting, occlusion, blur, etc., the training samples, making the constructed topological structure robust enough to noise.
Overall, based on the analysis above, we believe that the topological structure constructed in the input space sufficiently robust and can effectively guide the learning of the latent space structure.
A.4 Limitation
Our model might has the following two limitations:
(i) Due the introduction of the structure alignment strategy PTSA and hard sample mining strategy SDE, our TopoFR model requires longer training time (1.16x). For detailed training time analysis, please refer the Table TO in appendix.
(ii) This work aims to improve the generalization performance of models by leveraging the inherent structure information in large-scale face training data. However, large-scale face training datasets real-world scenarios inevitably contain a small amount of noisy labels. Our loss function does not assign any special treatment these mislabeled samples. Since our hard sample mining strategy SDE assigns larger weights to hard samples that are misclassified or have high prediction uncertainty, mislabeled images may be wrongly emphasized. We believe that future works will be able to simultancously address the challenges of structure alignment and label noise.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction section include the main claims made in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have discussed the limitations of this work. Please refer to the Section A.4 in the Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer:
[NA]
Justification:
This work does not involve any novel theoretical findings.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have provided the required code and pre-trained model for this paper. Please refer to the GitHub link provided at the end of the abstract.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
23

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have provided detailed implementation details (i.e., Section A.1 in the Appendix) and provided the data and source code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
Moolic//guide//Coeeettliiis itw.ooeeSuubiissiooooooooooogg for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wwtps/cccppplgguuuaaaeeeae Citeeee//ooSSSSeeSS..ee.a www.ssssiicggcooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided detailed implementation details (i.e., Section A.1 in the Appendix) and provided the data and source code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We follow the standard experimental principles of previous FR works and report the best results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We have provided detailed implementation details and training settings (i.e.,
Section A.1 in the Appendix)
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http::////nnapppp..cpcpoctt/t/ee
Question:
Answer: [Yes]
The research conducted in the paper fully adheres to the NeurIPS Code of Ethics.
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: It would be good to mention the utilization of face images do not have any privacy concern given the datasets have proper license users consent distribute biometric data for research purpose.
25

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The data and code used in this paper have obtained legal permissions.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
26

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
27

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "6BoFFB9lW5", "reviewer": "Reviewer_SWuk", "summary": "The paper introduces a new method to improve the topological structures of facial features in the latent space. By exploring the topological structure alignment in face recognition, the authors propose a new structural alignment strategy PTSA to align the structures of origin input space and feature space. The experimental results on various benchmarks have verified the effectiveness of the proposed method.", "questions": "Please refer to my weakness section. In addition, I have another question related to the Perturbation-guided Topological Structure Alignment. Are the persistence diagrams and persistence pairings computation in the ISA loss differentiable for backpropagation? Can the authors detail this computational process?", "limitations": "The authors have discussed their limitations in the appendix. However, it will be better if the authors discuss the broader impact of the proposed method.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Based on the persistent homology method, the paper provides a comprehensive analysis of the correlation between the number of data samples and topological structures in the latent space.\n- The proposed modules in the method, Perturbation-guided Topological Structure Alignment and Structure Damage Estimation, are well motivated and seem to improve the topological structure of facial features.\n- The authors achieve competitive results on the standard benchmarks, especially ranking #2 on the MFR-Ongoing leaderboard.", "weaknesses": "- While I acknowledge the competitive results of the proposed approach, the performance improvement of the method in several benchmarks remain minor. For example, the results of the method on IJB-B and IJB-C benchmarks are minor compared to prior methods on the same backbone. \n\n- The technical approach seems to be an incremental one. In particular, the authors adopt the ArcFace framework and introduce two additional modules on top of the ArcFace method, including Perturbation-guided Topological Structure Alignment and Structure Damage Estimation. The first module is basically a metric to compare the feature distribution between feature space and original input space, similar to the Gromov-Wasserstein distance. However, the ArcFace loss itself is also an efficient approach to learning the facial feature representation in the latent space. I am not sure whether the Perturbation-guided Topological Structure Alignment significantly improves the structure information or not.\n\n- Why the authors claim the proposed approach improve the topological structure of the feature space, I could not see this point has been verified via either experimental results or visualization. It would be better if the authors visualize the feature distributions of faces on the latent space (via PCA or T-SNE).\n\n\n- The author claims that \"to the best of our knowledge, how to effectively mine the potential structure information in large-scale face data has not investigated\". I do not agree with this claim since the other methods, e.g., ArcFace, SphereFace, CoseFace, etc, are also considered as approaches to improve the structures/distributions of facial features in the deep latent space via the well design loss functions. I think the statements made by the authors are over-claimed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer SWuk for the careful reading and the insightful comments, which are helpful in improving our paper. Our detailed point-by-point responses are provided below.\n\n**W1: The results of the method on IJB-B and IJB-C are minor.**  \n**A1:** Due to the characters limit, we will respond to you in the **global rebuttal area**. \n\n**W2: The technical approach seems to be an incremental one. The PTSA strategy is similar to the Gromov-Wasserstein distance.**  \n**A2:** Due to the characters limit, we will respond to you in the **global rebuttal area**.\n\n**W3: I am not sure whether the PTSA strategy significantly improves the structure information or not. Why the authors claim the proposed approach improve the topological structure of the feature space? It would be better to visualize the feature distributions.**  \n**A3:** As shown in Figures 2(a) and 2(b), we experimentally find that there are significant topological structure discrepancy between the input space and the latent space of existing FR models. This observation indicates that the topological structure information of the large-scale dataset is not effectively preserved in the latent space and may even be destroyed. Therefore, we propose the PTSA strategy to align the topological structure of two spaces. The visualization results in Figs 2(d), 5 and 7 have validated that with the help of our PTSA strategy, the topological structure discrepancy between two spaces have significantly decreased. In fact, the results in Figs 2(d), 5 and 7 can quantify topological structure discrepancy more objectively and accurately than t-SNE. Moreover, the ablation study results in Tab 8 of Appendix have also validated that the addition of PTSA can implicitly enhance the intra-class compactness and inter-class separability of facial features. \n\n**t-SNE:** Based on your insightful advice, we sample 10 identities face images from MS1MV2 dataset and utilize the t-SNE to visualize the facial features learned by ArcFace and ArcFace + PTSA. The results are shown in the Figure 1 of the **global response PDF file**. We can observe that using the proposed PTSA strategy to reduce the topological structure discrepancy between two spaces can greatly enhance the discriminative power of the learned facial features (e.g., with better inter-class separability and intra-class compactness), thereby boosting the FR model's generalization performance. The t-SNE results are consistent with our aforementioned experimental results.\n\n**W4: Other methods are also considered as approaches to improve facial features distributions in latent space via well design loss functions.**  \n**A4:** Existing FR methods, such as ArcFace, CosFace, SphereFace and CurricularFace, tended to design some margin-based softmax loss functions to explicitly encourage facial features with better inter-class separability and intra-class compactness in feature space. However, the experimental results in Figs 2(a) and 2(b) have shown that the latent space's topological structure of existing models may be destroyed, which limits FR model's generalization. Thus, unlike these existing FR works that directly force features to become more discriminative in the feature space, our PTSA strategy aims to leverage the intrinsic topological structure information in training dataset (i.e.,from the input pixel space) to guide the construction of the latent space's topological structure and the learning of facial features.  Detailed experiments have indicated that our method can effectively reduce topological structure discrepancy and boost model's generalization.\n\nIn general, leveraging the topological structure information in dataset to enhance the model's generalization is a different research direction from designing the margin-based softmax loss functions to improve facial features distribution in the deep feature space.\n\n**Q1: Are the persistence diagrams (PDs) and persistence pairings (PPs) computation in ISA loss differentiable for backpropagation? Detail this computational process.**  \n**A5:** Yes, the computation of PDs and PPs in ISA loss is differentiable for back-propagation. \n\n**1) Detailed computation process:** Persistent homology (PH) is a method used to analyze the topological structure information of complex point clouds (e.g., mini-batch data). In the original input space, we first flatten the mini-batch face images into vector features to model point clouds $\\mathcal{X}$, and then calculate their pairwise distance matrix to construct the Vietoris-Rips (VR) complex $\\mathcal{V}\\_\\{\\rho}(\\mathcal{X})$. Similarly, we can construct another VR complex $\\mathcal{V}\\_\\{\\rho}(\\widetilde{\\mathcal{Z}})$ for the perturbed latent features $\\widetilde{\\mathcal{Z}}$.\nAfter that, we use the Ripser package [S3] to implement the PH to extract topological features from two VR complexes, and obtain their corresponding PDs and PPs. The discrepancy between the input space's PD and latent space's PD represents the topological structure discrepancy of two spaces. However, directly calculating the distance between two PDs is computationally expensive. We thus turn to utilize PPs to estimate the discrepancy between two PDs. \nPPs contain indices of simplices that are related to the birth and death of the topological feature. In this case, we can calculate the discrepancy between two PDs by utilizing PPs to retrieve the differences between two pairwise distance matrices.\n\nRef.[S3] Ripser: Efficient Computation of Vietoris–Rips Persistence Barcodes. 2021.\n\n**2) Optimization of model params w.r.t $\\mathcal{L}_{sa}$:** Please refer to the 6th answer **A4** to **Reviewer GA4X**.\n\n\n\n**Q2: Discuss the broader impact.**  \n**A6:** Due to the characters limit, we will respond to you in the **global rebuttal area**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2LwShD7mhF", "reviewer": "Reviewer_68Et", "summary": "This paper uses the topological structure alignment in face recognition tasks, and it proposes a Perturbation-guided Topological Structure Alignment (PTSA) strategy to align the topology of input image space and latent space. In this paper, Persistent Homology(PH) is used to verify that the complexity of input space topology will increase rapidly when there are many face images. In addition, this paper also puts forward some ideas and implementation of data mining.", "questions": "- Is this topological alignment strategy only for face recognition tasks?\n- Is the process of alignment computationally intensive?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The paper adopts topological structure alignment in face recognition tasks.\n+ A new hard sample mining strategy SDE is proposed to reduce the adverse effects of hard samples on potential spatial structure.\n+ Experiments are sufficient, and the experimental results prove that the model has excellent performance.", "weaknesses": "- Formula 7 may require a more specific explanation.\n- The meaning of H0 in Figure 1, along with the distribution style and other differences, is best explained.\n- The hard sample mining strategy SDE in this paper seems to have little correlation with Perturbation-guided Topological Structure Alignment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate Reviewer 68Et for the thorough review of the paper and valuable comments that will aid in enhancing our paper.\n\n**W1: Formula 7 may require a more specific explanation.**   \n**A1:** Based on your valuable suggestion, we provide an explanation of how to estimate the parameter set $\\varphi = \\\\left \\\\{ \\pi, \\Sigma, \\Omega \\\\right \\\\}$ of the Gaussian-uniform mixture (GUM) model using EM algorithm.\n\nFirst, we transform the original GUM model (Eq.(3)) into a more easily solvable statistical distribution (Eq.(6)) using the Bernoulli distribution-based sampling method. In this way, the maximum likelihood model of Eq.(6) is formulated as: $\\max\\limits\\_\\{\\pi,\\Sigma,\\Omega} \\displaystyle\\prod_{i=1}^{n}p\\left(\\widehat{E}(\\widetilde{g}\\_\\{i})|\\widetilde{x}\\_\\{i}\\right )$.\nThen, we can use the EM algorithm to solve the maximum likelihood problem in order to estimate the parameter set $\\varphi$ of GUM. The specific iterative updating formulas of EM algorithm are shown in Eq.(7).\n\nSpecially, at each iteration, EM alternates between evaluating the expected log-likelihood (E-step) and updating the parameter set $\\varphi$ (M-step). In Eq.(7), the **E-step** aims to evaluate the posterior probability $h_{\\varphi}^{(t+1)}$ of an sample $\\widetilde{x}\\_\\{i}$ to be hard sample using the iterative formula $h\\_\\{\\varphi}^{(t+1)}(\\widetilde{x}\\_\\{i})$, where $(t+1)$ denotes the EM iteration index. The **M-step** updates the parameter set $\\varphi$ using the iterative formulas $\\pi^{(t+1)}$, $\\Sigma^{(t+1)}$ and $\\Omega^{(t+1)}$, where $\\eta_{1}$ and $\\eta_{1}$ are the first-order and second-order centered data moments, respectively.\n\n**W2: The meaning of H0 in Figure 1, along with the distribution style and other differences, is best explained.**  \n**A2:** As described in Sec.3 (lines 105-109), homology is an algebraic structure that analyzes the topological features of a simplicial complex in different dimension $j$, including connected components ($H_{0}$), cycles ($H_{1}$), voids ($H_{2}$), and higher-dimensional topological features ($H_{j},j\\geq 3$). By tracking the changes in topological features $H_{j}$ across different dimensions $j$ as the scale parameter $\\rho$ increases, we can obtain the multi-scale topological information of the underlying space.\n\nThus, $H_{0}$ is the $0$-th dimension homology in the persistence diagram, which captures the $0$-th dimension topological feature of the underlying space. Notably, low-dimension topological features (e.g., $H_{0}$) can roughly reflect the topological structure of the space, while high-dimension topological features (e.g., $H_{3}$, $H_{4}$) aim to capture the intricate details of the space's topological structure.\n\nMoreover, in topology theory, an increase in the number of high-dimensional topological features within a space corresponds to a more complex topological structure. As illustrated in Figs 1(a)-1(d), as the amount of face data increases, the persistence diagram contains more and more high-dimensional homology (e.g., $H_{3}$ and $H_{4}$), indicating that the input space contains an increasing number of high-dimensional topological features. Thus, this phenomenon shows that the topological structure of the input space is becoming more and more complex.\n\n**W3: The hard sample mining strategy SDE seems to have little correlation with Perturbation-guided Topological Structure Alignment (PTSA).**     \n**A3:** As stated in Sec.1, the large-scale face dataset contains rich topological structure information. However, we find that these structure information is not effectively preserved in the latent space of existing FR models, as verified in Fig 2 (lines 31-43). Existing studies on FR have overlooked this issue, which limits the generalization of FR models. To solve this problem, we propose the PTSA strategy. Specially, our PTSA strategy directly extracts the topological structure information in the face dataset from the input pixel space and then encodes these powerful structure information into the latent space by aligning the topological structures of the input and latent spaces.\n\nHowever, as stated in Sec 4.2 (lines 170-177), we experimentally find that some low-quality samples (e.g., hard samples) can easily be encoded to abnormal positions in the latent space during training, which disrupts the latent space’s topological structure and further hinder the alignment of topological structures. Thus, we propose the SDE strategy to identify hard samples with significant structure damage and effectively mitigate their adverse impact on the latent space’s topological structure by guiding them back to their appropriate positions during optimization.\n\nHence, our PTSA strategy and SDE strategy are complementary to each other. Additionally, the ablation study in Table 3 also validate the complementarity of two strategies.\n\n**Q1: Is this topological alignment strategy only for face recognition (FR) tasks?**   \n**A4:** In addition to FR systems, the proposed PTSA strategy can also be applied to Image Retrieval and Large Language Model (LLM), since these tasks all require large-scale datasets for training.\\\nExisting works on Image Retrieval and LLM seem to neglect the utilization of topological structure information hidden in large-scale datasets. Thus, we believe that applying the topological structure alignment technique to these tasks can effectively improve the representation power of features and the generalization ability of models.\n\n**Q2: Is the process of alignment computationally intensive?**   \n**A5:** In fact, we have provided an analysis of the model training time in our manuscript (lines 637-650). Please refer to Table 13 in the Appendix. For example, compared to the R50 ArcFace (Baseline), our R50 ArcFace + PTSA requires about 2.24 seconds more training time per 100 steps. These results indicate that introducing PTSA strategy leads to significant performance gains with only a small increase in training time, which is acceptable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4sdkR5Wkzw", "reviewer": "Reviewer_GA4X", "summary": "This paper proposes a novel framework named TopoFR for face recognition (FR). The authors first discover that existing FR models struggle to effectively preserve the structure information hidden in FR dataset, and provide three specific observations: \n\n(i) the topological structure in the input pixel space becomes increasingly complex as the amount of data increases; \n\n(ii) the structure difference between pixel space and latent feature space increases as the amount of data increases;\n \n(iii) the structure difference between two spaces decreases as the network architecture becomes deeper. \n\nConsidering that FR training datasets are typically massive and contain rich structure information, this paper aims to leverage these intrinsic structure information to enhance FR model's generalization performance. However, the authors notice that directly aligning the structures between latent space and input pixel space will result in inferior performance which might be overfitting. To solve this issue, TopoFR introduces a PTSA strategy to encode the structure information of input pixel space into latent space. Specifically, PTSA utilizes RSP to perturb the latent space structure and employs ISA to match the topological structures of two spaces, effectively mitigating structure discrepancy. Moreover, a novel hard sample mining strategy SDE is proposed to identify hard samples and minimize their impact on the latent space structure. Extensive experiments on various FR benchmarks demonstrate that TopoFR outperforms state-of-the-art methods. In addition, detailed ablation studies and analysis are conducted to provide a comprehensive understanding of the proposed method.", "questions": "1) EM algorithm is sensitive to the initialization of its parameters values. In Eq.(7), how to determine the initial values of parameters $\\pi$, $\\Sigma$, and $\\Omega$ in the GUM statistical distribution?\n \n2) In Eq.(2), does the ISA function loss $\\mathcal{L}_{sa}$ guide the optimization of the entire network or only specific components of the network?\n \n3) What distance criterion is used to construct the topological structure space of the data in Vietoris-Rips complex during model forward propagation? \n \n4) In addition, in topological data analysis, different dimensions of topological holes analyze the underlying data structure information at different levels. In ISA loss, what dimension of topological holes does the persistent homology (PH) capture and analyze?\n \n5) I am very curious about the reasons why the structure differences between the input pixel and latent spaces decrease as the network becomes deeper.", "limitations": "It's not discussed whether this method can be applied in other related FR methods, e.g. AdaFace. I suggest to further discuss this to increase the overall quality of this paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1) This paper is clearly written and well-motivated. The overall readability is high. Despite the significant progress made by existing FR models, these models rely on explicitly constraining inter-class and intra-class relationships. Unlike these existing works, this paper seeks to leverage the structure information in datasets to improve the FR model's generalization performance, which is a highly innovative attempt and can provide valuable insights for future research. Furthermore, it is novel to address the input-latent space structure degradation problem.\n \n2) It is a very interesting idea to employ the GUM statistical distribution in designing the hard sample mining strategy SDE from the outliers detection perspective. The structure damage score SDS is similar to the area of noise labels, which is not well investigated in FR.\n \n3) Experiments are good and the various benchmark results have produced state-of-the-art performance. Moreover, the model's ranking in the MFR competition shows its strong generalization performance.", "weaknesses": "There are some concerns that I would like the authors to address in order to enhance the overall comprehensiveness of the paper.\n \n1) It is not clear if this method can be applied to recent proposed loss, e.g., AdaFace [1]. I recommend adding experiments to further validate the universality of the proposed method.\n \n2) [Minor Comments]: Although the authors have discussed the model’s training and inference efficiency in Appendix, I suggest including comparisons of the model's GFLOPs. This would provide a more intuitive representation of the model’s computational cost.\n \nRef[1] Kim M, Jain A K, Liu X. Adaface: Quality adaptive margin for face recognition[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 18750-18759.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer GA4X for the careful reading of the manuscript and the related comments, which are helpful to improve our paper.\nOur detailed point-by-point responses are provided below.\n\n**W1: It is not clear if this method can be applied to recent proposed loss AdaFace.**  \n**A1:** According to your suggestion, we conduct additional experiments on AdaFace model to further validate the universality of our method, as shown in the following Table R1. \\\nIn comparison to the baseline model ArcFace, AdaFace is a highly advanced FR model. Consequently, integrating our method with AdaFace can notably enhance the model's verification accuracy across multiple benchmarks.\nThese improved results validate the universality of our method.\n\n**Table R1: Verification performance (%) on IJB-C, IJB-B, CPLFW and CALFW.**\n| Training Data | Method | IJB-C(1e-4) | IJB-B(1e-4) | CPLFW | CALFW |\n| ------ | ------ | ------ | ------ | ------ | ------ |\n|  | R50 AdaFace | 96.27 | 94.42 | 92.83 | 96.07 |\n| MS1MV2 | R50 AdaFace + PTSA | 96.51 | 95.25 | 93.36 | 96.28 |\n|  | R50 AdaFace + PTSA + SDE | **96.64** | **95.38** | **93.50** | **96.39** |\n\n**W2: Includes comparison of the model's GFLOPs.**  \n**A2:** Based on your valuable advice, we provide a comparison of model's GFLOPs, as shown in the following Table R2.\nAs can be seen, our TopoFR has the same GFLOPs as the vanilla ArcFace model (Baseline) and some existing popular FR models (e.g., MagFace, AdaFace and CurricularFace), since we adopt the same network architecture.\n\n**Table R2: Comparison of Model's GFLOPs.**\n| Method | GFLOPs |\n| ------ | ------ |\n| R50 ArcFace | 6.3 |\n| R50 MagFace | 6.3 |\n| R50 AdaFace | 6.3 |\n| **R50 TopoFR** | 6.3 |\n| R100 ArcFace | 12.1 |\n| R100 MagFace | 12.1 |\n| R100 AdaFace | 12.1 |\n| R100 CurricularFace | 12.1 |\n| **R100 TopoFR** | 12.1 |\n| R200 ArcFace | 23.4 |\n| R200 AdaFace | 23.4 |\n| **R200 TopoFR** | 23.4 |\n\n\n**Q1: In Eq.(7), how to determine the initial values of parameters $\\phi$, $\\Sigma$, and $\\Omega$ in the GUM statistical distribution ?**  \n**A3:** We refer to prior work [S4] to set the initial values for the EM algorithm used in our GUM parameter estimation process.\n\nRef.[S4] Using gaussian-uniform mixture models for robust time-interval measurement. TIM 2015.\n\n\n**Q2: In Eq.(2), does the ISA function loss $\\mathcal{L}_{sa}$ guide the optimization of the entire network or only specific components of the network ?**  \n**A4:** In fact, the ISA loss $\\mathcal{L}_{sa}$ is only responsible for optimizing the parameters of the feature extractor $\\mathcal{F}$.\n\nAs stated in Section 4.1 (lines 140-146), since computing the pairwise distance matrices $\\mathcal{M}^{\\mathcal{X}}$ and $ \\mathcal{M}^{\\mathcal{\\widetilde{Z}}}$ for Vietoris-Rips (VR) complexes $ \\mathcal{V}\\_\\{\\rho}(\\mathcal{X})$ and $ \\mathcal{V}\\_\\{\\rho}(\\mathcal{\\widetilde{Z}})$ is a differentiable process, the ISA loss $\\mathcal{L}_{sa}$ will perform gradient back-propagation through the two distance matrices to optimize the parameters of the feature extractor $\\mathcal{F}$.\n\n**Q3: What distance criterion is used to construct the topological structure space of the data in Vietoris-Rips complex during model forward propagation ?**  \n**A5:** As stated in Section 3 (lines 95-104), we utilize the Euclidean distance as the distance criterion to construct the Vietoris-Rips (VR) complex $\\mathcal{V}_{\\rho}$ and analyze the topological structure of the underlying space.\n\n**Q4: In ISA loss, what dimension of topological holes does the persistent homology (PH) capture and analyze ?**  \n**A6:** As described in Section A.1 of the Appendix (lines 547-550), we preserve and analyze 0-dimension topological holes $H_{0}$ (e.g., connected components) in our ISA loss $\\mathcal{L}_{sa}$. \nBecause some preliminary experiments have shown that using the 1-dimension or higher-dimension topological holes only increases model’s training time without bringing clear performance gains.\n\n\n**Q5: I am very curious about the reasons why the structure differences between the input pixel and latent spaces decrease as the network becomes deeper.**  \n**A7:** This is a very interesting question. When we obtained this finding in the experiment, we also contemplated its underlying reasons. \\\nIn our opinion, the deep neural network is actually a complex function mapping process, and each network layer can be regarded as a sub-function in this composite function. If the neural network is deeper, the mapping of each network layer will become smoother, and the data will lose less information during the mapping process. On the contrary, if the neural network is shallower, the mapping of each network layer will become sharper, and some intrinsic information of the data, such as structure information, is easily destroyed during the mapping process. Similar conclusions can also be found in the flow-based models [S5].\n\nRef. [S5] Variational inference with normalizing flows. ICML 2015.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear **Reviewer SWuk**, here is our response to some of the concerns you raised.

**W1: The results of the method on IJB-B and IJB-C are minor.**  
**A1:** (1) Notably, when using a shallow backbone such as ResNet-50, our method showcases remarkable performance gains on IJB-B, IJB-C and other benchmarks, as shown in Tabs 2 and 4, compared to the leading competitors. This is because the shallow backbone has larger topological structure discrepancy between the input and latent spaces (as verified in Figure 2(b)), and our method can effectively mitigate the structure discrepancy and boost the model's generalization performance. Furthermore, when using extremely large-scale datasets such as Glint360K and WebFace42M, our method can achieve more significant accuracy improvements, as shown in Tabs 1 and 2. Since larger-scale dataset contains richer topological structure information, which more clearly indicates the effectiveness of our method in using topological structure information to boost model's generalization.

(2) On some low-quality (e.g., contains many blurry face images) and challenging FR benchmarks, such as MFR-Ongoing, MegaFace, MegaFace-Refined, TinyFace, CPLFW and CALFW, our method can achieve clear performance gains compared to the competitors, as shown in Tabs 4, 5 and 6 of the Appendix. Additionally, as shown in Tab 7 of Appendix, our method can obtain a significant improvement in performance when combined with the lightweight face recognition (FR) network backbone MobileFaceNet.

(3) The accuracy of existing FR methods on the IJB-B and IJB-C benchmarks is nearing saturation, as these two datasets contain many high-quality face images. Our proposed method is still able to achieve performance gains, surpassing the ResNet-based SOTA competitor AdaFace [S1] and the ViT-based SOTA competitor TransFace [S2]. Notably, in FR task, a performance improvement of approximately 0.2% to 0.3% on the IJB-C (1e-4) and IJB-B (1e-4) benchmarks is quite substantial. While our R200 TopoFR trained with Glint360K achieves maximum performance gains of 0.49% on IJB-C (1e-4) and 0.47% on IJB-B (1e-4) compared to the SOTA method AdaFace, respectively. In general, our method achieves SOTA performance on various benchmarks using different backbones, implying its strong generalization ability.

Ref.[S1] Adaface: Quality adaptive margin for face recognition. CVPR 2022.\
Ref.[S2] Transface: Calibrating transformer training for face recognition from a data-centric perspective. ICCV 2023.

**W2: The technical approach seems to be an incremental one. The PTSA strategy is similar to the Gromov-Wasserstein (GW) distance.**  
**A2:** As mentioned in Sec.1 (lines 31-43), we are the first to discover that existing FR models struggle to effectively preserve the topological structure of face data in the latent space, and we provide some novel observations (as depicted in Figs 1 and 2).
However, existing FR studies have overlooked this issue, which severely limits the models' generalization. Based on this motivation, we are the first to propose aligning the topological structures of two spaces to improve the model's generalization performance. A series of experimental results have also shown that the topological structure information in the dataset is quite crucial for FR task. Thus, our method is not an incremental work, but a novel exploration. 

Moreover, the Gromov-Wasserstein (GW) distance metric is different from the topological structure distance metric. Specially, the topological structure models the relationships and connectivity between data by analyzing the evolution progress of the simplicial complex constructed from the data in high-dimensional metric space. While the GW distance is often used to align the mainfold structures of two feature distributions. Mainfold structure primarily models the similarity between data based on the similarity matrix, which is a different concept from topological structure. Thus, our PTSA strategy does not simply match feature distributions, but rather aligns the topological structures of the input and latent spaces in the high-dimensional metric space.


**Q2: It will be better if the authors discuss the broader impact of the proposed method.**  
**A6:** Thanks for your valuable advice.
It would be good to mention that the utilization of face images do not have any privacy concern given the datasets have proper license and users consent to distribute biometric data for research purpose. We address the well-defined face recognition task and conduct experiments on publicly available face datasets. Therefore, the propose method does not involve sensitive attributes and we do not notice any negative societal issues.

**t-SNE Visualization**: The t-SNE visualization results mentioned in A3 are shown in the Figure 1 of the global response PDF file.

**To all Reviewers:** If you have additional concerns, please let us know and we will do our best to address them. We appreciate your time and efforts in reviewing our work.

Meta-review: The rebuttal provided clarifications about the proposed method and its analysis that were useful for assessing the paper's contribution and responded adequately to most reviewer concerns. All reviewers recommend acceptance after discussion (with two weak accepts and one accept), and the ACs concur. The final version should include all reviewer comments, suggestions, and additional clarifications from the rebuttal.
Decision: Accept (poster)