ProxyFusion: Face Feature Aggregation Through Sparse Experts

Authors: bhavin  jawade
alexander  stone
deen  dayal
xiao  wang
srirangaraj  setlur
venu  govindaraju
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 20:43:11

Abstract: Face feature fusion is indispensable for robust face recognition, particularly in scenarios involving long-range, low-resolution media (unconstrained environments) where not all frames or features are equally informative. Existing methods often rely on large intermediate feature maps or face metadata information, making them incompatible with legacy biometric template databases that store pre-computed features. Additionally, real-time inference and generalization to large probe sets remains challenging. 
To address these limitations, we introduce a linear time O(N) proxy based sparse expert selection and pooling approach for context driven feature-set attention. Our approach is order invariant on the feature-set, generalizes to large sets, is compatible with legacy template stores, and utilizes significantly less parameters making it suitable real-time inference and edge use-cases. Through qualitative experiments, we demonstrate that ProxyFusion learns discriminative information for importance weighting of face features without relying on intermediate features. Quantitative evaluations on challenging low-resolution face verification datasets such as IARPA BTS3.1 and DroneSURF show the superiority of ProxyFusion in unconstrained long-range face recognition setting. 
Our code and pretrained models are available at: https://github.com/bhavinjawade/ProxyFusion

Main paper: 
 introduction: 
 Face recognition (FR) involves generating representations or templates from face images for 1:1 verification and 1:N identification between query media, known as the probe, and an enrolled biometric template, known as the gallery. Numerous studies have explored novel feature extraction architectures [10] and metric learning-based loss formulations [3, 17] for learning discriminative representations. But, feature extraction is typically the first step in building a face matching system; the second step involves creating robust templates using sets of images or videos of the same individual. This requires fusing or aggregating representations from different face images (or frames, in the case of videos) to obtain a unified template.Recently, there has been heightened interest in recognizing individuals under extremely challenging conditions, such as from long distances and high altitudes, exemplified by the IARPA BRIAR program. These scenarios introduce novel challenges, making face feature aggregation even more crucial since only a limited set of frames would contain discriminative identity information. In cross-distribution setting, such as low-resolution to high-resolution face matching where a significant distribution gap Figure 1 : We design our approach to solve three primary challenges (i) Cross-Domain Matching: Matching low-resolution, long-range faces with high-quality gallery faces. (ii) Linear Runtime Complexity: Ensuring our method's time complexity increases linearly with the number of features. (iii) Compatibility with Legacy Templates: Relying solely on final feature vectors for fusion to maintain compatibility with pre-enrolled feature stores that lack intermediate features or metadata. exists, face feature fusion becomes essential for selecting the most relevant gallery or probe frames for accurate matching.Existing face feature fusion methods face several key challenges: (i) generalization to a large probe set [9] , (ii) real-time inference with low computational cost, (iii) compatibility with legacy template stores, and (iv) cross-distribution matching capabilities [5] . Additionally, these methods should perform importance attribution based on feature quality and feature frequency. This work aims to train a fusion model that learns an order and length-invariant feature weighting strategy. Specifically, given an unordered set of N features, the model should return a set of scores that, when used to weigh the original features, produce an aggregated vector that robustly represents the feature set while maximizing identity-specific information.As observed by CAFace [9] , typical set-to-set attention mechanisms such as Multihead Attention (MHA) and other intra-set attention methods like RSA [12] or CoNAN [5] exhibit quadratic time complexity, O(N 2 ). This makes them unsuitable for feature aggregation when dealing with large probe sets. Furthermore, methods such as CAFace [9] require high-dimensional intermediate feature maps (H × W × C) from the face feature extractor to compute the style information for feature importance weighting. Other approaches, such as [14, 13] , leverage external metadata predictors to extract facial characteristics like pose, gender, and distance, which are then used for attributeconditioned aggregation. These methods are incompatible with existing biometric template stores which typically only store the penultimate features.In this work we propose a novel feature aggregation framework, ProxyFusion (PF) that utilizes K learnable proxies P, (where P i ∈ R K×D ) to implicitly represent latent facial attributes. These learnable proxies are used for selecting pooling experts based on relevancy scores. Inspired by works on mixture-of-experts [4] and its sparse variants [16] , we utilize only the most relevant K experts for each feature-set, thereby reducing the inference time parameters. The proposed pooling experts generate set-centers conditioned on the feature-set distribution. Divergence of input set features from set-centers represents their informativeness. We utilize the divergence to pool the N feature vectors into one aggregated representation. As we will discuss later, ProxyFusion approach performs order-invariant, size-agnostic feature aggregation without relying on high-dimensional intermediate feature maps or additional metadata. Through various qualitative experiments, we demonstrate that experts can learn distinctive face quality information. Since the effectiveness of face feature fusion is particularly evident in long-range, low-resolution settings, we conduct extensive experiments on the IARPA's BRIAR BTS3.1 dataset [1] , which includes videos and images collected in extreme unconstrained environments, along with their constrained counterparts. In addition to BTS3.1, we also conducted experiments on another unconstrained UAV cross-distribution dataset, DroneSURF [6] .In summary, the key contributions of this paper are:• We introduce ProxyFusion, a feature aggregation framework using learnable query embeddings to select pooling experts. The methods avoids the need for high-dimensional  
method: 
 The objective of our method is to correctly match the subject identities in probe media to the subject identities in gallery media. Our problem setting deviates from more conventional face recognition because we have additional challenges where the probe images can have significantly degraded quality as they exist in the long-range, low-resolution domain, where as the gallery images are in the close-range, high-resolution domain. Additionally, the probe feature set for a given subject can be extremely large, possibly in the hundreds of thousands. We define these two sets of images asI G = {G 1 , G 2 , .. . , G m } for the set of high quality gallery images, and I P = {P 1 , P 2 , . . . , P n } for the set of low quality probe images. Since a face feature set is composed of faces in different poses and quality, they have different degrees of discriminative identity information. The goal of aggregating features across the feature set {f i } N i=1 is to form a template vector t ∈ R K•d that best represents the subject's identity for cross-distribution matching.Figure 2 illustrates our proposed approach. Since our method strictly focuses on feature fusion of output embeddings, we pre-extract face features using frozen pretrained face detection and recognition backbones. The face detector is denoted as f D : I → D, where I is a set of frames and D is the set of detected face regions. For our face feature extractor, we denote the model as f E : D → E, which maps a cropped face region in D to a d-dimensional feature embedding in the set E = {f i } N i=1 , where f is a face feature vector.Our proposed feature fusion method is composed of two key parts. The first involves sparsely selecting relevant experts, and the second performs feature pooling using the chosen experts. In section 2.1 we discuss the Expert Network Selection strategy followed by the feature aggregation apporach using these experts.Given a set E of face features {f i } N i=1 , we define a set of learnable proxies {p j } K j=1 , where f i , p j ∈ R d . Here proxies are fixed dimensional embeddings that would represent latent information about facial characteristics required to decide which expert network should be utilized. We project each feature f i and proxy p j to a unified latent space using a shared projection layer, represented by thematrix W ∈ R d×d ′ : f ′ i = W • f i ∈ R d ′ , p ′ j = W • p j ∈ R d ′ where d ′is the output dimensionality of the W. We choose d ′ << d for parameter efficiency.Using f ′ i , p ′ j , we compute the Proxy Relevancy Scores denoted by r j ∈ R, by computing the similarities between each projected proxy p ′ j and all the projected features f ′ i , and accumulating these similarities across all features r j =N i=1 p ′ j • f ′ i .To perform expert network selection, we index the top-K values from the set of proxy relevancy scores {r j } K j=1 . We denote these indices as I top-k = {j 1 , j 2 , . . . , j k }, and use them to selectively activate a subset of expert networks { E j } K j=1 ⊆ {E j } K j=1 , where K < K. These selected expert networks will be used in the following stage for feature pooling.We subsampled proxies and their associated expert networks using their relevancy scores with respect to the feature-set in the previous stage. Here we describe our approach to use these sparsely selected expert networks to extract conditional set-centers for aggregating features into the final template vector representation. Motivated by mixture-of-experts [4] , we aim to learn mutually exclusive yet homogeneous experts that rank features differently based on learned implicit characteristics. We condition our expert networks over the learned subsampled proxies along with the cross-sample mean and variance across all dimensions within the feature set. More concretely, given a set of feature vectors {f i } N i=1 , where each vector f i ∈ R d , we compute the mean and variance vectors µ ∈ R d , σ 2 ∈ R d as follows:µ = 1 N N i=1 f i , σ 2 = 1 N N i=1 (f i -µ) 2 .then, we create our input feature distribution representation as x j = µ σ 2 p j , where x j ∈ R 3•d and denotes concatenation. For each x j , we infer through its corresponding expert network, determined by the associated proxy, to obtain the set-centers {c j } K j=1 , where c j = E j (x j ), ∀j ∈ {1, . . . , K}.The outputs of the expert networks, referred here as set-centers, are used to compute the divergence of each feature in the original set. These divergence scores are then utilized to compute feature importance. We compute the divergence scores as the un-normalized alignment between the features {f i } N i=1 and set-centers {c j } K j=1 given by c j •f i . Next, we softmax these divergence scores to compute weights as follows:a ij = exp(c j • f i ) N k=1 exp(c j • f k ), ∀i ∈ {1, . . . , N }, ∀j ∈ {1, . . . , K} Finally, we compute the weighted sum of the feature vectors for each set-center using the weights a ij :s j = N i=1 a ij f i , ∀j ∈ {1, . . . , K}where {s j } K j=1 is the set of reduced aggregated representation from the selected experts. We define the final template vector t as concatenation of s j given by: t = s 1 , s 2 , . . . , s K .Set Length and Order Invariance: Feature set length invariance refers to our method's ability to effectively handle feature sets of any size, while feature order invariance indicates that our method can process any permutation of features without affecting the outcome. Feature set length invariance arises through the application of mean and variance as a representation of feature-set distribution as also utilized by [5] . Mean and variance could generalize to varying feature lengths if the training-set consists of diversely sampled features. We facilitate this through our batch creation strategy.To construct a batch, we select a set of M subject identities {I j } M j=1 and subsample their respective probe and gallery feature sets. Let P i denote the probe feature set for identity I i and G i denote the respective gallery feature set. We define the subsets P i ⊆ P i and G i ⊆ G i , where the sizes of these subsets are uniformly sampled from the range| P i |, | G i | ∼ U(L, U ).For efficient computation, we zero-pad to the maximum feature set length for any identity within a batch to create uniformly sized tensors.The order independence of features is a direct consequence of the properties of mean and variance. This invariance also applies to our Proxy Relevancy Scores, {r j } K j=1 , because the order in which the feature set is multiplied with the proxies is not relevant. The sum is computed along the dimension of the feature vectors, identifying the most relevant proxy without being affected by feature order.We primarily optimize our aggregation experts and the proxies with the identification objective. The identification loss addresses the primary goal to bridge the distribution gap between the probe and gallery templates for a same identity while increasing the inter-class variance across different identities.The output of ProxyFusion is the template t. For all feature subsets in a batch we compute the identity loss using the supervised contrastive loss [7] . Let, B = (t P , t G ) be the batch consisting of all probe and gallery features then L id is defined as:L id = i∈B -1 |P(i)| p∈P(i) ln exp(t i • t ⊤ p /τ ) j∈A(i) exp(t i • t j /τ )where t i is the feature embedding at index i in the set the (t P , t G ). P(i) is the set of indices of samples with the same subject label as i and A(i) is the set of indices of all samples with identities different from subject at i. Here, τ is the softmax's temperature parameter.During training we want to supervise the experts to learn focus on mutually exclusive information through the proxy conditioning. Additionally, while computing the proxy relevancy scores, we want different proxies to attend over different facial characteristics. To achieve this, we propose an additional optimization criteria referred here as the proxy loss. We will fix K uniformly spaced equidistant vectors on the unit hypersphere in R K-1 . We choose K -1 dimensions because it is the lowest dimensional space such that our K proxies can be equidistant yet farthest apart from each other. We project the proxies p ∈ R d down using Wp ∈ R K-1 . Then, we fix K vectors {v i } K i=1 to be equidistant from each other as follows:v i = e i - 1 d d j=1 e j d d -1 , ∀d ∈ {1, . . . , K -1}, ∀i ∈ {1, . . . , K}where e i ∈ R K-1 are the standard basis vectors. Based on this the proxy loss is defined as:L Proxy = 1 K K i=1   ln (1 + exp (-α(s ii -λ))) + 1 |K -1| k∈K k̸ =i ln (1 + exp (β(s ik -λ)))    Where, s ik = (p i • v k ) / (∥p i ∥∥v k ∥)is the similarity between the i th proxy and k th fixed basis vector. λ is the threshold for the exponential function. The first part of this term enforces that proxies get closer to their respective basis vectors, while second part enforces that they move away from all other negative basis vectors.The final loss is given by L = L ID + γ • L Proxy , where γ is weightage of the proxy loss. 
dataset: 
 To illustrate the effectiveness of our aggregation technique in long-range, low-resolution, unconstrained matching scenarios, we selected challenging datasets featuring low-quality images and videos captured from long distances. Our experiments utilize the following datasets for training: (i) BRIAR Research Set 3 (BRS 3)[1]: This dataset is from IARPA's BRIAR program Phase 1, featuring videos and images from 170 participants in controlled and field settings. Controlled settings have high-resolution facial images at close range, while field settings include media captured from 100 to 500 meters away. For training CoNAN, we used 49,429 video clips and images from BRS 3, with 20,780 field-setting clips, 23,489 controlled-setting clips, and 5,160 images. Our method is trained on BRS 3 for fair comparison with methods like [5] trained on the same dataset. (ii) WebFace 4M [22] : Apart from BRIAR, we also present results by training our method WebFace 4M dataset. This is done to present fair evaluation with CAFace [9] which has been trained on WebFace 4M. Following CAFace [9] , we use their randomly sampled subset, consisting of 813, 482 images from 10, 000 identities to train our aggregation function. Following [5] , we evaluate our method on following two datasets: (i) BTS 3.1: This is the test set for IARPA BRIAR Phase 1 evaluation. We report results for the face-included treatment and control protocols. The treatment set has 5,822 probe videos from 260 subjects in uncontrolled settings, while the control set has 1,914 probe videos from 256 subjects in regulated settings. The BTS 3 gallery is split into Gallery 1 (47,925 clips/images, 485 subjects) and Gallery 2 (47,413 clips/images, 481 subjects), with 351 common distractor identities. (ii) DroneSURF: This dataset includes Active and Passive Surveillance settings, each with 100 videos and 786,000+ face annotations. Following [12] , we split subjects randomly: 60% (34 identities) for training/validation, 40% (24 identities) for testing. The dataset has 200 videos of 58 subjects, over 411,000 drone-captured frames. Results are based on the video-wise identification protocol.Face Detector and Alignment: We present results using two popular face detectors: MTCNN [21] and RetinaFace [2] . Unless otherwise specified, the results are reported using the RetinaFace. Feature Extractors: Following previous works [5] , [8] we report results using two pretrained frozen feature extractors -Adaface [3] and Arcface [3] . For Adaface [8] we use a ResNet-101 model pretrained on WebFace. Each face image is resized to 112x112x3 before feature extraction. For Arcface, we extract features using a MS1MV2 pre-trained ResNet-50 backbone. For fusion architecture trained using WebFace 4M Adaface features [8] , we utilize the precomputed features provided by previous workd [9] for fair comparison. Architecture and Hyperparameters: For the expert networks we utilize a three layer MLP with LeakyReLU activation and a dropout with probability of 0.5. More specifically, the first layer in the MLP projects from 1536 to 1024, the second layer from 1024 to 1024, and last layer from 1024 to 512. Overall the MLP has 3.14M learnable parameters. For supervised contrastive loss, we use a temperature of 0.1. The proxy loss weight γ = 0.01 and threshold λ = 0.1. We utilize the Adafactor with adaptive learning rates as the optimizer. For all SoTA experiments we utilize number of total proxies K as 11, and number of selected experts K as 4. We choose the number of identities in a batch M = 170 based on available GPU memory. We choose the bounds for probe and gallery subset sizes to be L = 100 and U = 1200. All experiments are performed on 1 x A6000 48GB NVIDIA GPU. Most experiments require nearly 2 hours with precomputed features.Weight Visualizations and Interpretability: In Fig. 3 , we illustrate the weightings assigned by the set-centers extracted from each selected expert for the face embeddings. This visualization offers insights into the aggregation function's capacity to discern the informativeness of faces. Notably, the model operates solely on frozen precomputed features within the feature set, with no access to intermediate features or metadata.As evidenced in Fig. 3 , each expert attributes higher weights to faces that exhibit more discriminative identity information, such as frontal or profile views, in both the gallery and probe sets. Conversely, the model assigns minimal weight to poor-quality crops, such as images of the back of the head or inaccurate detections. In the case of probes, which are collected in unconstrained, long-range settings, only a limited number of frames in a video contain significant facial information. Hence, the model assigns considerably higher weights to these informative frames within the probe set (e.g., Expert 10 assigns approximately 0.007205 to a frontal face).In contrast, the gallery set comprises numerous high-quality, informative faces, resulting in the model assigning relatively lower weights to these frames due to their abundance. Consequently, weight assignments decrease more gradually in the gallery set compared to the probe set. Additionally, each expert provides subtly different weightings to the same frames, enhancing the overall representation of the face. This analysis demonstrates that the ProxyFusion model adeptly discerns the informativeness of facial features in an interpretable yet effective manner.In Figure 4 , we present the face verification performance in terms of TAR@FAR=10 -2 , plotted against the number of proxies K (X-axis) and the selected number of experts K (Y-axis). The results1 indicate that increasing the number of proxies generally enhances model performance. However, this improvement plateaus around 10-12 proxies, and further increasing the number of proxies beyond 13 leads to overfitting. This overfitting likely occurs because the model, with a higher number of experts during training, becomes more prone to fitting the training distribution and training subjects too closely, thus failing to generalize well to unseen subjects. 4 also demonstrates that an increase in the number of selected experts generally enhances performance. However, this improvement comes at the expense of longer inference times. As the number of selected experts increases, the number of parameters required during inference also rises, leading to increased Table 3 : Verification Performance (TAR (%) @FAR=%) for face included treatment and control protocols of the BTS 3.1 dataset. All faces are detected and algined using RetinaFace face detector.Face Included Treatment Face Included Control Feature Dataset 10 -1 10 -2 10 -3 10 -4 10 -1 10 -2 10 -3 10 -4 GAP [11] Adaface [8] 1 we present the contribution of the L Proxy to the overall performance. We observe that though the contribution of L Proxy is marginal when the number of selected experts is small, there are performance improvements at modest N . We believe this is due to the decorrelation of proxies, which results in extraction of diverse information from the feature set. To validate our algorithm's claimed linear time complexity we perform GFLOPs analysis of our method against increasing sizes of the featureset. As can be observed from Fig. 5 , we start with small feature-set size of 100 and scale it up to 1 million. Our fusion model's (in blue) GLOPs increases linearly with the increasing number of features in the set. On contrary, GFLOPs increase quadratical for models like CoNAN [5] and [9] . Moreover, these with single-shot input of large feature-sets to these models, the memory footprint increases quadratically due to intra-set attention which leads to out-of-memory issues for these models beyond an N of 21000. These experiments are performed on 1 NVIDIA A6000 averaged over 3 runs.Table 2 : Rank-1 accuracy (%) for video-wise identification on DroneSURF dataset.Trained On DroneSURF Feature Active Passive GAP [11] Adaface [8] 46.87 7.29 NAN [20] Adaface [8] 65.62 6.25 MCN [19] Adaface [8] 72.92 8.33 CoNAN [5] Adaface [8] 80.21 13.54 Ours Adaface [8] 83.33 13.54 Trained On BRS (Cross-Dataset Evaluation) GAP [11] Adaface [8] 46.87 7.29 NAN [20] Adaface [8] 80.21 8.33 MCN [19] Adaface [8] 79.16 10.41 CoNAN [5] Adaface [8] As can be observed from table 3 and 4 our method significantly outperforms other linear time methods such as NAN [20] and MCN [19] while providing a significant jump over naive averaging or Global Average Pooling (GAP) [11] . Our method also achieves on-par performance to quadratic time complexity methods such as CoNAN [5] and CAFace [9] . When trained on Web-Face, our approach outperforms CAFace [9] on most FAR thresholds without utilzing intermediate features unlike [9] . When compared to CoNAN [5] , our method performs outperforms it on Face included control setting while being significantly faster (See Fig. 5 , having much lower runtime memory requirements, and comparable inference time parameters (13.6M for CoNAN and 12.4M for ProxyFusion). Similar trend can be observed on DroneSURF dataset (refer Table 2 ). All SotA experiments are performed several times and we present the average results. We calculated Standard Errors of the Mean (SEM) across the multiple observations and found the SEM to very low in the range of 0.008-0.09 showing the statistical significance of the experiments.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Justification:Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
related work: 
 Methods based on Intermediate Feature Maps and Metadata: Leveraging intermediate feature maps for feature fusion, methods like the nonlocal neural network [18] and RSA [12] show promise in modeling intra-set relationships through detailed spatial analysis. These methods utilize intermediate feature maps U i of size C m × H × W to capture complementary information and refine spatial relationships. CAFace [9] addresses the computational issue of these approaches with a two-stage feature aggregation method, using these feature maps as style information. This involves assigning N inputs to M global cluster centers and fusing clustered features. CAFace generates an affinity map with N 2 complexity [15] . The requirement for intermediate feature maps makes these methods incompatible with legacy biometric systems lacking precomputed intermediate features. In contrast, our method relies solely on the penultimate feature representation, typically used for generating biometric templates. Furthermore, Metadata extracted from face images can serve as an alternative source for inferring face quality. [14] introduced a feature aggregation method that integrates metadata (e.g., yaw, pitch, face size) and utilizes a siamese network to determine the relative quality correlations among face images in a set. Additionally, metadata can facilitate fine-grained matching across galleries and probes by identifying faces with similar characteristics. For instance, TADPool [13] adapted both probe and gallery features by considering attributes like face yaw and roll, employing an attention block to pool probe features based on their compatibility with selected gallery features.Linear Time Feature Fusion Methods: Previous works have proposed aggregation methods with linear time complexity against the number of features in set. In [20] , a network architecture with two cascaded attention blocks is presented, which evaluates the significance of features in an image set and uses these scores for feature aggregation. Recently, [15] employed a differentiable coreset selection approach, using learned metrics and a Gumbel-Softmax distribution to optimize the selection of a small, representative subset, which is then enriched via self and cross-attention mechanisms thereby reducing computational complexity by decreasing set-size. [19] introduces a multicolumn network that assigns weights to images within a set based on their visual quality, assessed through a self-quality assessment module. This network then dynamically adjusts these weights according to each image's relative content quality compared to others in the set.We address key challenges in feature fusion: (i) real-time inference, (ii) cross-distribution matching, (iii) generalization to large feature sets, and (iv) compatibility with legacy feature stores. Our proxy-relevancy based approach for expert selection, combined with a feature pooling approach, ensures robust multifaceted feature aggregation with minimal inference time and no need for extra metadata or intermediate features. Ablation studies demonstrate our method's superior performance and efficiency, outpacing existing approaches in both speed and effectiveness for real-time applications. Potential Negative Societal Impacts: We are meticulous about training or testing our models only on datasets with approved IRB and consent from involved human subjects. This is why we do not use IJB-B/C datasets, as they contain web-sourced faces without consent or IRB approval. Conversely, datasets like IARPA BRIAR and [6] which we utilize are collected under IRB with subject consent. Limitations: While our method effectively discerns feature informativeness via sparse experts, relying solely on feature-set distribution statistics overlooks fine-grained intra-set relationships. We can enhance this by hierarchically merging them with intra-set methods, achieving further gains with limited computational cost.• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We include all of the information necessary to reproduce our experiments.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
open access to data and code: 
 Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We will publicly release the code and trained models. The data can be requested from IARPA and IIT Jodhpur.Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: We follow all of the guidelines in the NeurIPS code of ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We have included this discussion in our paper.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Justification: There are no particular safe guards, yet we only train our models on datasets with approved IRB and consent from involved subjects. Therefore there is a limited risk of misuse of our pretrained models.Answer: [No] Justification: The datasets we are using are provided external entities. We sign required agreements and licenses to acquire these datasets. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.38th Conference on Neural Information Processing Systems (NeurIPS 2024).It should be noted here that, given the number of experimental runs required to perform this analysis we only indicate the results after 10 th epoch in this analysis, which may not reflect the best performance of the model. 
 
 Appendix:  


Decision: This submission proposed a face recognition framework for feature aggregation to deal with long-range, low-resolution cases. The reviewers recognized the merits of proposed method. Meanwhile, some concerns on presentation and writing are pointed.  3/4 reviewers are positive to this submission in general, while reviewer Cxak has some concerns on optimization details, potential drawback on feature extraction. After reading the submission, comments, and rebuttal, the AC agree with most of the reviewers and suggest accept this submission.

Reviews: review: paper_summary: paper_summary: The authors propose a method for fusing face features extracted from sets of images.
Several proxies are defined that are trained to be specialized for certain aspects of faces, and the most relevant for a given set of face images are obtained.
Importance weights for different samples are estimated for each selected proxy, and a template stores a combination of weighted feature averages.
Experiments on challenging datasets with a wide variety of viewing conditions indicate largely improved performance with linear time complexity in the number of samples per set.
strength_weakness: strength_weakness: 1. There are a few important details of the proposed algorithm missing in the paper:

   a) One part of the proposed method, i.e., the details of the expert network E_j are missing in the paper -- how large are the layers? 

   b) The proxy loss is defined over a set of negative samples. There is no word spent on how to select such negative samples -- yet, the selection is of high importance. Also, this loss includes a lambda value that is not detailed.

   c) Both gallery and probe templates finally consists of a concatenation of several averaged features. The authors need to clarify how the final score between these sets of templates is computed.


2. Some parts of the paper include non-standard solutions that should be considered to update:

   a) In order to compute valuable means and standard deviations in section 2.2, the features would need to reside in Euclidean space. However, modern deep FR networks optimize for cosine space, and deep features typically do not follow Euclidean distributions. Why would this be a good idea?

   b) Relatedly, the expert networks obtain mean, standard deviation and proxy centers. It is not shown in the paper that the variances provide any improvements over only using the means. A better handling of mean and proxy center than a simple concatenation is likely possible.

   c) The proxy center projections v are selected to be equidistant in Euclidean space, while they result from a projection that rather lives in cosine space. It is not clear where the expression is coming from or why such a difficult expression needs to be defined, a simple set of K-dimensional basis vectors e_i would likely be sufficient.

   d) The models used for feature extraction are not well-suited for low-resolution and highly distorted faces. Better models will likely improve the utility of the feature fusion approach.


3. There are some improvements on the paper:

   a) The introduction is slightly repetitive and could be shortened.

   b) At several places in the paper, it is mentioned that the method is compatible with existing template storage solutions. The practical relevance for such compatibility is questionable. In a typical template storage, a single template is stored as an aggregation over several samples already. The authors should not overemphasize this characteristic of their method.

   c) The authors claim their algorithm to have a time complexity of O(N). However, the complexity is rather O(N*K). Depending on the choice of K, this might be a relevant deviation from O(N).

   d) In line 138 and following, the authors write about batch creation strategies. It is not entirely clear if this is applied during test time as well, or whether this is a training time issue only. Finally, the authors claim to include diverse features, but random selection does not ensure diversity.

   e) The equation for the standard deviation in section 2.2 is slightly off, the factor should be 1/(N-1).

   f) In line 170, the scores s_ik should make use of the projected proxies Wp, not the proxies themselves.

   g) The figures should be included as vector graphics to allow zooming. Especially figure 3 is of too low quality to read.

   h) In the caption of figure 5, N represents the total number of probe features. In section 2.2, N is the number of images for one identity, while in section 2.3, N represents the number of identities.

   i) The paper should be carefully checked for spelling mistakes.
questions: questions: The most important question is how negative samples are obtained for the proxy loss. The authors should elaborate on this. Also, the strategy on handling several features per template needs to be revealed. Afterward, I am happy to increase my rating.
limitations: limitations: The authors took special care about using only datasets with IRB-consented data. They should be aware, however, that the pre-trained networks that they exploit for feature extraction have been trained on web-scraped datasets which did not include IRB-approved subjects.
paper_summary: paper_summary: The authors introduce a novel approach for face feature fusion, addressing typical scenarios such as low-quality probes and high-quality or different domain gallery sets of faces. They describe alternative approaches and conclude that these are currently limited because they are either not compatible with legacy biometric databases or have real-time performance issues.

The authors provide a method for feature aggregation through Sparse Experts. The overall pipeline includes three steps: Feature Extraction -> Expert Network Selection -> Sparse Expert Network Feature Aggregation. Each step is described in detail.

Optimization is done using a two-component loss: Loss for the identification objective + proxy loss. The proxy loss aims to make different proxies attend to different facial characteristics. The authors present experimental justifications for the second loss in the experimental section.

The authors describe the experiments conducted. Two datasets were used for training: BRIAR Research Set 3 and WebFace 4M. Evaluation was performed on BTS 3.1 and DroneSURF. The authors provide all details about these steps. They also present visualizations showing that each expert assigns higher weights to faces with more discriminative identity information and minimal weights to poor-quality images. Experimentally, they found that the optimal setup is to use 4 Selected Experts and 12 total Proxies/Experts.

The authors compared their method with various state-of-the-art alternatives and demonstrated that their method either produces strong results, achieving SOTA performance, or performs on par with quadratic-time methods, despite being a linear-time method.

Additionally, the authors justify that their method is feature order and set length invariant.
strength_weakness: strength_weakness: The authors commit to releasing the code but did not attach it. While the method is clear and straightforward, the overall implementation can be complex, which could hinder reproducibility. Therefore, I believe the code is a crucial part of this work and should be shared along with the article, especially since this work does not involve heavy neural models or long training procedures.

Without a full code review and the ability to run it, there may not be sufficient proof of reproducibility. Additionally, the authors' response to Question 7, 'Experiment Statistical Significance,' may not fully address the issue, as comparison with alternative methods alone is not evidence of statistical significance.

Furthermore:

> (ii) DroneSURF: This dataset includes Active and Passive Surveillance settings, each with 100 videos and 786,000+ face annotations. Following [12], we split subjects randomly: 60% (34 identities) for training/validation, 40% (24 identities) for testing. The dataset has 200 videos of 58 subjects, over 411,000 drone-captured frames. Results are based on the 203 video-wise identification protocol.

It is unclear how the random split can be compared to other methods. If the split was done as in [12], this information should have been provided. Additionally, I suggest that the splits be released so future authors can make clear comparisons.
questions: questions: 1. Could the authors provide the code and necessary weights at this review stage to enhance clarity and reproducibility?
2. Could the authors provide more detailed information about the DroneSURF random splits or share the splits themselves?
3. Could the authors clarify about the statistical significance?
limitations: limitations: There is one limitation described:

> While our method effectively discerns feature informativeness via sparse experts, relying solely on feature-set distribution statistics overlooks fine-grained intra-set relationships.
paper_summary: paper_summary: The paper proposes a novel feature fusion technique for unconstrained face recognition via sparse experts. The authors propose a expert network selection mechanism using the $k$ proxies, $p_j$, and all the $N$ precomputed face features, $f_i$. The top $\hat{k}$ selected experts are used by the 3d set centers to produce discriminative features per expert that finally forms the template. Two losses are proposed, a supervised contrastive loss ($L_{id}$) to ensure high interclass and low intra class separability between probe and gallery, and a proxy loss ($L_{proxy}$) to ensure the experts learn mutually exclusive information. The authors validate their method primarily on DroneSurf and BRIAR dataset on different TAR@FAR=thresholds. They also provide visualization for the learned experts and GFLOPS analysis to support the linear time complexity claim.
strength_weakness: strength_weakness: ### 1. The presentation of the method end experiment section could be greatly improved. There are some texts that is unclear that certainly affects the final rating of the paper. 

a.  Line 97, What are proxies in this context and how are they constructed initially? Are they randomly initialized? Or are they some form of router network? The paper also seems to use proxies/learnable queries (line 68) interchangeably which is further confusing.

b.  Line 125, Divergences represent the dissimilarity between two distributions. Can the authors please expand on how the dot product between two feature vector is a divergence?

c.  Table 1, What dataset has this been tested on? Is this where the authors employed $\hat{K}=4$ as the optimum experts?

d.  What is the success rate of both face detectors for probe images? At 500m with several noise factors, these standard detectors might encounter higher degree of failure.

e.  Figure 4 is a bit confusing as to what exactly $x-axis$ is, as the experts/proxies are used interchangeably.

f.  It would also add value if authors are able to add TAR@FAR values per each distance category for probe BTS 3.1. I am assuming this would involved categorizing the results which should not be a completely new experiment.


### 2. The paper does not include enough description to make it reproducible.

a.  Line  141, what were the chosen $L$ and $U$ values in the experiments?

b.  Line 143, what was the suitable $M$ found to be?

c.  Line 146, pad them with what value?

d.  What is the final $d$ and $d'$ ?


### 3. I also have few minor corrections to suggest to further improve the quality of the paper.

a.  Paragraph 139 and 143 seems to convey the same meaning. Can be combined into one.

b. SCL equation for $L_{id}$ uses the transpose $T$ inconsistently.

c. Line 187, I think it is supposed to be "For training *ProxyFusion*..."

d. Other errors:

 * Typo in line 90, *refere*
* Line 125, 'that is used to compute...'
* Line 154, "primary goal bridge...."
* Line 282, "these with single shot..."
questions: questions: I have mentioned most of the questions in the weakness section. The most important ones to address would be 1 and 2.
limitations: limitations: The authors adequately address the limitations about the proposed methods and use of only IRB approved datasets for experiementation. 

For future work, it would be interesting to compare the method against [1], which also attempts to achieve similar goals. This however, is a suggestion, and in no way affects my final rating for this conference.


[1] Shapira, Gil, and Yosi Keller. "FaceCoresetNet: Differentiable Coresets for Face Set Recognition." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 5. 2024.
paper_summary: paper_summary: This paper presents a novel face recognition framework that aims to address the challenges of feature fusion in long-range, low-resolution scenes. The authors propose a linear time complexity approach that is compatible with traditional biometric template databases and does not require additional metadata or intermediate feature maps.ProxyFusion utilises a set of learnable agents to implicitly represent potential facial attributes and selects the most relevant experts to focus on the feature set, thus significantly reducing inference time and parameters. The method is order and size invariant, making it very robust for real-time applications and large probe sets. Extensive experiments on the IARPA BRIAR BTS3.1 and DroneSURF datasets demonstrate the superiority of ProxyFusion in unconstrained long-range face recognition settings.
strength_weakness: strength_weakness: 1. The optimization of the sparse experts is ambiguous. In the section of methods, the authors do not provide any details on how to optimize the sparse experts. 
2. The paper mentions that relying solely on feature-set distribution statistics might overlook fine-grained intra-set relationships. This suggests that the model may not capture all the nuances present within a set of features.
3. The performance of ProxyFusion, like any face recognition system, is likely to depend heavily on the quality of the input data. Poor quality inputs could degrade performance.
4. The paper does not mention the model's robustness to adversarial attacks, which is an important consideration in the field of biometrics and face recognition.
questions: questions: Please refer to the weaknesses.
limitations: limitations: Please refer to the weaknesses.

Comments: title: Running the code
Thanks for the response
Thanks for your response
Response to rebuttal
Thanks for your response
Read Acknowledgment
comment: Thank you for taking the time to review our submission and for your feedback. We would like to clarify that our repository's code is fully functional and has been tested. To demonstrate this, we have attached screenshots showing our training and evaluation code running on the Github repo. We have added screenshots to Github repo since we couldn't add images to openreview's comments section. We are more than willing to assist you in getting the code to run on your end.

To better understand the issues you encountered, could you please provide further details on the following:
1. The code requires access to external datasets that are provided by their respective authors through license and agreement. Have you procured these datasets (example: BRS, BTS) and extracted face embeddings according to the instructions provided in our ReadMe file?
2. After creating the HDF5 file for the precomputed embeddings from the dataset, does it follow the same structure as mentioned in our ReadMe? 
3. Could you please share the minor errors you mentioned about in your comment? 

We are committed to provide any assistance you might need. Thank you for your understanding and time.
I have reviewed the authors' response and explanation, and I appreciate their efforts. However, the code provided is not functional, which is a significant issue. The errors, though minor, indicate that the code has not been executed successfully. This impacts the overall quality of the submission. As a result, my evaluation remains unchanged.
Thanks to the reviewer for their valuable comments, constructive feedback, and time.
The authors have addressed all my comments well. 

My initial point 2(a) was just a hint for a different processing, the explanation of the method in the paper is clear. The authors should note that unnormalized deep features typically are not normal distributed, so modeling the features with a normal distribution (which assumes Euclidean space) might not be optimal.
Thank you for your response. We appreciate the time and the valuable feedback you have provided. We are glad that our explanations and code could provide better clarity. As you have suggested, we will ensure that the points about the definition of proxies, their purpose, and the contrastive-like explanation behind the proxy loss equation are covered in detail in the final version with the same level of clarity.
I have read the rebuttals from the Authors. I thank them for the thorough explanation for each question. 

For 1a, After reading through the explanations here and having gone through the code provided, I have a better understanding about proxies. Please ensure the below points are covered or expanded in the paper so that the same level of clarity is reflected in the final version.

* Definition of proxies and its purpose at the beginning of 2.1.
* The contrastive-like Intuition behind the proxy loss equation.