Title: ProxyFusion: Face Feature Aggregation Through Sparse Experts
Venue: NeurIPS 2024
Accepted: yes
Keywords: Feature Fusion

Face Recognition

Pooling
Abstract: Face feature fusion is indispensable for robust face recognition, particularly in scenarios involving long-range, low-resolution media (unconstrained environments) where not all frames or features are equally informative. Existing methods often rely on large intermediate feature maps or face metadata information, making them incompatible with legacy biometric template databases that store pre-computed features. Additionally, real-time inference and generalization to large probe sets remains challenging. 
To address these limitations, we introduce a linear time O(N) proxy based sparse expert selection and pooling approach for context driven feature-set attention. Our approach is order invariant on the feature-set, generalizes to large sets, is compatible with legacy template stores, and utilizes significantly less parameters making it suitable real-time inference and edge use-cases. Through qualitative experiments, we demonstrate that ProxyFusion learns discriminative information for importance weighting of face features without relying on intermediate features. Quantitative evaluations on challenging low-resolution face verification datasets such as IARPA BTS3.1 and DroneSURF show the superiority of ProxyFusion in unconstrained long-range face recognition setting. 
Our code and pretrained models are available at: https://github.com/bhavinjawade/ProxyFusion

ProxyFusion: Face Feature Aggregation Through Sparse Experts

Abstract

Face feature fusion is indispensable for robust face recognition, particularly in scenarios involving long-range, low-resolution media (unconstrained environments) where not all frames or features are equally informative. Existing methods often rely on large intermediate feature maps or face metadata information, making them incompatible with legacy biometric template databases that store pre-computed features. Additionally, real-time inference and generalization to large probe sets remains challenging. To address these limitations, we introduce a linear time O ( N ) proxy based sparse expert selection and pooling approach for context driven feature-set attention. Our approach is order invariant on the feature-set, generalizes to large sets, is compatible with legacy template stores, and utilizes significantly less parameters making it suitable real-time inference and edge usecases. Through qualitative experiments, we demonstrate that ProxyFusion learns discriminative information for importance weighting of face features without relying on intermediate features. Quantitative evaluations on challenging lowresolution face verification datasets such as IARPA BTS3.1 and DroneSURF show the superiority of ProxyFusion in unconstrained long-range face recognition setting. Our code and pretrained models are available at: https://github.com/ bhavinjawade/ProxyFusion

1 Introduction

Face recognition (FR) involves generating representations or templates from face images for 1:1 verification and 1:N identification between query media, known as the probe , and an enrolled biometric template, known as the gallery . Numerous studies have explored novel feature extraction architectures [10] and metric learning-based loss formulations [3, 17] for learning discriminative representations. But, feature extraction is typically the first step in building a face matching system; the second step involves creating robust templates using sets of images or videos of the same individual. This requires fusing or aggregating representations from different face images (or frames, in the case of videos) to obtain a unified template.

Recently, there has been heightened interest in recognizing individuals under extremely challenging conditions, such as from long distances and high altitudes, exemplified by the IARPA BRIAR program. These scenarios introduce novel challenges, making face feature aggregation even more crucial since only a limited set of frames would contain discriminative identity information. In cross-distribution setting, such as low-resolution to high-resolution face matching where a significant distribution gap

Complexity

Linear

with Legacy

Compatible

Matching

Domain

Cross-

ComplexitywithLargeProbeSet

ProxyFusion

X

X

Features

Probe

Template

Features

[e.g. CoNAN, RSA]

Intra-set Attention

[e.g. MARN]

/X>

Probe Set

Small

Others

Probe Set

Large

Features

Gallery

Style Based Methods

X

X

[e.g. TADPool,

ProxyFusion

MFAN]

ProxyFusion

ProxyFusion (Ours)

exists, face feature fusion becomes essential for selecting the most relevant gallery or probe frames for accurate matching.

Existing face feature fusion methods face several key challenges: (i) generalization to a large probe set [9], (ii) real-time inference with low computational cost, (iii) compatibility with legacy template stores, and (iv) cross-distribution matching capabilities [5]. Additionally, these methods should perform importance attribution based on feature quality and feature frequency. This work aims to train a fusion model that learns an order and length-invariant feature weighting strategy. Specifically, given an unordered set of N features, the model should return a set of scores that, when used to weigh the original features, produce an aggregated vector that robustly represents the feature set while maximizing identity-specific information.

As observed by CAFace [9], typical set-to-set attention mechanisms such as Multihead Attention (MHA) and other intra-set attention methods like RSA [12] or CoNAN [5] exhibit quadratic time complexity, O ( N 2 ) . This makes them unsuitable for feature aggregation when dealing with large probe sets. Furthermore, methods such as CAFace [9] require high-dimensional intermediate feature maps ( H × W × C ) from the face feature extractor to compute the style information for feature importance weighting. Other approaches, such as [14, 13], leverage external metadata predictors to extract facial characteristics like pose, gender, and distance, which are then used for attributeconditioned aggregation. These methods are incompatible with existing biometric template stores which typically only store the penultimate features.

In this work we propose a novel feature aggregation framework, ProxyFusion (PF) that utilizes K learnable proxies P , (where P i ∈ R K × D ) to implicitly represent latent facial attributes. These learnable proxies are used for selecting pooling experts based on relevancy scores. Inspired by works on mixture-of-experts [4] and its sparse variants [16], we utilize only the most relevant ̂ K experts for each feature-set, thereby reducing the inference time parameters. The proposed pooling experts generate set-centers conditioned on the feature-set distribution. Divergence of input set features from set-centers represents their informativeness. We utilize the divergence to pool the N feature vectors into one aggregated representation.

As we will discuss later, ProxyFusion approach performs order-invariant, size-agnostic feature aggregation without relying on high-dimensional intermediate feature maps or additional metadata. Through various qualitative experiments, we demonstrate that experts can learn distinctive face quality information. Since the effectiveness of face feature fusion is particularly evident in long-range, low-resolution settings, we conduct extensive experiments on the IARPA's BRIAR BTS3.1 dataset [1], which includes videos and images collected in extreme unconstrained environments, along with their constrained counterparts. In addition to BTS3.1, we also conducted experiments on another unconstrained UAV cross-distribution dataset, DroneSURF [6].

In summary, the key contributions of this paper are:

We introduce ProxyFusion, a feature aggregation framework using learnable query embeddings to select pooling experts. The methods avoids the need for high-dimensional

FeatureExtraction

ExpertNetworkSelection

SparseExpertNetworkFeatureAggregation

Proxies Q

(Learnable Queries)

Experts

Networks

6E

W

D x I

（μo²,P）

E

DxT

Selection Indices

Expert

i

E

R1x D

RNx D

FE

RNxD

W

Pooling

ArgMax

E

X

Summation

Weighted

Scores

R1xD

RKxD

1 x P

Divergence

Scores

(P' x F’)

S

Identity F

Ek

R1xD

intermediate feature maps or additional metadata making it compatible with existing biometric template stores.

By selecting the top experts for each feature set inspired by the mixture-of-experts approach, ProxyFusion significantly reduces inference time parameters.

Extensive qualitative and quantitative experiments on challenging datasets (IARPA BRIAR BTS3.1 and DroneSURF) show ProxyFusion's effectiveness in long-range, low-resolution face matching, improving FR performance in extreme environments.

2 Method

The objective of our method is to correctly match the subject identities in probe media to the subject identities in gallery media. Our problem setting deviates from more conventional face recognition because we have additional challenges where the probe images can have significantly degraded quality as they exist in the long-range, low-resolution domain, where as the gallery images are in the close-range, high-resolution domain. Additionally, the probe feature set for a given subject can be extremely large, possibly in the hundreds of thousands. We define these two sets of images as I G = { G 1 , G 2 , . . . , G m } for the set of high quality gallery images, and I P = { P 1 , P 2 , . . . , P n } for the set of low quality probe images. Since a face feature set is composed of faces in different poses and quality, they have different degrees of discriminative identity information. The goal of aggregating features across the feature set { f i } N i =1 is to form a template vector t ∈ R ̂ K · d that best represents the subject's identity for cross-distribution matching.

Figure 2 illustrates our proposed approach. Since our method strictly focuses on feature fusion of output embeddings, we pre-extract face features using frozen pretrained face detection and recognition backbones. The face detector is denoted as f D : I → D , where I is a set of frames and D is the set of detected face regions. For our face feature extractor, we denote the model as f E : D → E , which maps a cropped face region in D to a d -dimensional feature embedding in the set E = { f i } N i =1 , where f is a face feature vector.

Our proposed feature fusion method is composed of two key parts. The first involves sparsely selecting relevant experts, and the second performs feature pooling using the chosen experts. In section 2.1 we discuss the Expert Network Selection strategy followed by the feature aggregation apporach using these experts.

2.1 Expert Network Selection

Given a set E of face features { f i } N i =1 , we define a set of learnable proxies { p j } K j =1 , where f i , p j ∈ R d . Here proxies are fixed dimensional embeddings that would represent latent information about facial characteristics required to decide which expert network should be utilized. We project each feature f i and proxy p j to a unified latent space using a shared projection layer, represented by the

matrix W ∈ R d × d ′ :

where d ′ is the output dimensionality of the W . We choose d ′ << d for parameter efficiency.

Using f ′ i , p ′ j , we compute the Proxy Relevancy Scores denoted by r j ∈ R , by computing the similarities between each projected proxy p ′ j and all the projected features f ′ i , and accumulating these similarities across all features r j = ∑ N i =1 ( p ′ j · f ′ i ) .

To perform expert network selection, we index the top-̂ K values from the set of proxy relevancy scores { r j } K j =1 . We denote these indices as I topk = { j 1 , j 2 , . . . , j k } , and use them to selectively activate a subset of expert networks { ̂ E j } ̂ K j =1 ⊆ { E j } K j =1 , where ̂ K < K . These selected expert networks will be used in the following stage for feature pooling.

2.2 Sparse Expert Network Feature Aggregation

We subsampled proxies and their associated expert networks using their relevancy scores with respect to the feature-set in the previous stage. Here we describe our approach to use these sparsely selected expert networks to extract conditional set-centers for aggregating features into the final template vector representation. Motivated by mixture-of-experts [4], we aim to learn mutually exclusive yet homogeneous experts that rank features differently based on learned implicit characteristics. We condition our expert networks over the learned subsampled proxies along with the cross-sample mean and variance across all dimensions within the feature set. More concretely, given a set of feature vectors { f i } N i =1 , where each vector f i ∈ R d , we compute the mean and variance vectors µ ∈ R d , σ 2 ∈ R d as follows:

then, we create our input feature distribution representation as x j = [ µ ⊕ σ 2 ⊕ p j ] , where x j ∈ R 3 · d and ⊕ denotes concatenation. For each x j , we infer through its corresponding expert network, determined by the associated proxy, to obtain the set-centers { c j } K j =1 , where c j = ̂ E j ( x j ) , ∀ j ∈ { 1 , . . . , ̂ K } .

The outputs of the expert networks, referred here as set-centers, are used to compute the divergence of each feature in the original set. These divergence scores are then utilized to compute feature importance. We compute the divergence scores as the un-normalized alignment between the features { f i } N i =1 and set-centers { c j } ̂ K j =1 given by c j · f i . Next, we softmax these divergence scores to compute weights as follows:

Finally, we compute the weighted sum of the feature vectors for each set-center using the weights a ij : N

where { s j } ̂ K j =1 is the set of reduced aggregated representation from the selected experts. We define the final template vector t as concatenation of s j given by: t = [ s 1 , s 2 , . . . , s ̂ K ] .

Set Length and Order Invariance: Feature set length invariance refers to our method's ability to effectively handle feature sets of any size, while feature order invariance indicates that our method can process any permutation of features without affecting the outcome. Feature set length invariance arises through the application of mean and variance as a representation of feature-set distribution as also utilized by [5]. Mean and variance could generalize to varying feature lengths if the training-set consists of diversely sampled features. We facilitate this through our batch creation strategy.

To construct a batch, we select a set of M subject identities { I j } j M =1 and subsample their respective probe and gallery feature sets. Let P i denote the probe feature set for identity I i and G i denote the respective gallery feature set. We define the subsets ̂ P i ⊆ P i and ̂ G i ⊆ G i , where the sizes of these

subsets are uniformly sampled from the range | ̂ P i | , | ̂ G i | ∼ U ( L, U ) . For efficient computation, we zero-pad to the maximum feature set length for any identity within a batch to create uniformly sized tensors.

The order independence of features is a direct consequence of the properties of mean and variance. This invariance also applies to our Proxy Relevancy Scores , { r j } K j =1 , because the order in which the feature set is multiplied with the proxies is not relevant. The sum is computed along the dimension of the feature vectors, identifying the most relevant proxy without being affected by feature order.

2.3 Optimization

We primarily optimize our aggregation experts and the proxies with the identification objective. The identification loss addresses the primary goal to bridge the distribution gap between the probe and gallery templates for a same identity while increasing the inter-class variance across different identities.

The output of ProxyFusion is the template t . For all feature subsets in a batch we compute the identity loss using the supervised contrastive loss [7]. Let, B = ( t P , t G ) be the batch consisting of all probe and gallery features then L id is defined as:

where t i is the feature embedding at index i in the set the ( t P , t G ) . P ( i ) is the set of indices of samples with the same subject label as i and A ( i ) is the set of indices of all samples with identities different from subject at i . Here, τ is the softmax's temperature parameter.

During training we want to supervise the experts to learn focus on mutually exclusive information through the proxy conditioning. Additionally, while computing the proxy relevancy scores, we want different proxies to attend over different facial characteristics. To achieve this, we propose an additional optimization criteria referred here as the proxy loss. We will fix K uniformly spaced equidistant vectors on the unit hypersphere in R K -1 . We choose K -1 dimensions because it is the lowest dimensional space such that our K proxies can be equidistant yet farthest apart from each other. We project the proxies p ∈ R d down using Wp ∈ R K -1 . Then, we fix K vectors { v i } K i =1 to be equidistant from each other as follows:

where e i ∈ R K -1 are the standard basis vectors. Based on this the proxy loss is defined as:

̸

Where, s ik = ( p i · v k ) / ( ∥ p i ∥∥ v k ∥ ) is the similarity between the i th proxy and k th fixed basis vector. λ is the threshold for the exponential function. The first part of this term enforces that proxies get closer to their respective basis vectors, while second part enforces that they move away from all other negative basis vectors.

The final loss is given by L = L ID + γ · L Proxy, where γ is weightage of the proxy loss.

3 Experiments

3.1 Dataset

To illustrate the effectiveness of our aggregation technique in long-range, low-resolution, unconstrained matching scenarios, we selected challenging datasets featuring low-quality images and

videos captured from long distances. Our experiments utilize the following datasets for training: (i) BRIAR Research Set 3 (BRS 3)[1]: This dataset is from IARPA's BRIAR program Phase 1, featuring videos and images from 170 participants in controlled and field settings. Controlled settings have high-resolution facial images at close range, while field settings include media captured from 100 to 500 meters away. For training CoNAN, we used 49,429 video clips and images from BRS 3, with 20,780 field-setting clips, 23,489 controlled-setting clips, and 5,160 images. Our method is trained on BRS 3 for fair comparison with methods like [5] trained on the same dataset. (ii) WebFace 4M [22]: Apart from BRIAR, we also present results by training our method WebFace 4M dataset. This is done to present fair evaluation with CAFace [9] which has been trained on WebFace 4M. Following CAFace [9], we use their randomly sampled subset, consisting of 813, 482 images from 10, 000 identities to train our aggregation function. Following [5], we evaluate our method on following two datasets: (i) BTS 3.1: This is the test set for IARPA BRIAR Phase 1 evaluation. We report results for the face-included treatment and control protocols. The treatment set has 5,822 probe videos from 260 subjects in uncontrolled settings, while the control set has 1,914 probe videos from 256 subjects in regulated settings. The BTS 3 gallery is split into Gallery 1 (47,925 clips/images, 485 subjects) and Gallery 2 (47,413 clips/images, 481 subjects), with 351 common distractor identities. (ii) DroneSURF: This dataset includes Active and Passive Surveillance settings, each with 100 videos and 786,000+ face annotations. Following [12], we split subjects randomly: 60% (34 identities) for training/validation, 40% (24 identities) for testing. The dataset has 200 videos of 58 subjects, over 411,000 drone-captured frames. Results are based on the video-wise identification protocol.

3.2 Implementation Details

Face Detector and Alignment: We present results using two popular face detectors: MTCNN [21] and RetinaFace [2]. Unless otherwise specified, the results are reported using the RetinaFace. Feature Extractors: Following previous works [5], [8] we report results using two pretrained frozen feature extractors - Adaface [3] and Arcface [3]. For Adaface [8] we use a ResNet-101 model pretrained on WebFace. Each face image is resized to 112x112x3 before feature extraction. For Arcface, we extract features using a MS1MV2 pre-trained ResNet-50 backbone. For fusion architecture trained using WebFace 4M Adaface features [8], we utilize the precomputed features provided by previous workd [9] for fair comparison.

Architecture and Hyperparameters: For the expert networks we utilize a three layer MLP with LeakyReLU activation and a dropout with probability of 0.5. More specifically, the first layer in the MLP projects from 1536 to 1024, the second layer from 1024 to 1024, and last layer from 1024 to 512. Overall the MLP has 3.14M learnable parameters. For supervised contrastive loss, we use a temperature of 0.1. The proxy loss weight γ = 0 . 01 and threshold λ = 0 . 1 . We utilize the Adafactor with adaptive learning rates as the optimizer. For all SoTA experiments we utilize number of total proxies K as 11, and number of selected experts ̂ K as 4. We choose the number of identities in a batch M = 170 based on available GPU memory. We choose the bounds for probe and gallery subset sizes to be L = 100 and U = 1200 . All experiments are performed on 1 x A6000 48GB NVIDIA GPU. Most experiments require nearly 2 hours with precomputed features.

3.3 Discussion and Ablation

Weight Visualizations and Interpretability: In Fig. 3, we illustrate the weightings assigned by the set-centers extracted from each selected expert for the face embeddings. This visualization offers insights into the aggregation function's capacity to discern the informativeness of faces. Notably, the model operates solely on frozen precomputed features within the feature set, with no access to intermediate features or metadata.

As evidenced in Fig. 3, each expert attributes higher weights to faces that exhibit more discriminative identity information, such as frontal or profile views, in both the gallery and probe sets. Conversely, the model assigns minimal weight to poor-quality crops, such as images of the back of the head or inaccurate detections. In the case of probes, which are collected in unconstrained, long-range settings, only a limited number of frames in a video contain significant facial information. Hence, the model assigns considerably higher weights to these informative frames within the probe set (e.g., Expert 10 assigns approximately 0.007205 to a frontal face).

In contrast, the gallery set comprises numerous high-quality, informative faces, resulting in the model assigning relatively lower weights to these frames due to their abundance. Consequently, weight

Expert9

Expert10

0002910:

0.002175

0.003397

0.002990

0.002582

Gallery

0.0021283

0.001768

0.001361

0.002269

0.003174

0.002722

D

0.002709

0.003829

0.003269

0.001817

0.001365

0.002149

0.000913

Expert2

0.001029

Expert6

Expert10

Expert2

0.002883

0.001442

0.007205

0.005764

0.004323

0.000001

sauosg

0.003708

0.004944

0.002472

0.001236

0.006180

0.000001

Probes

0.002314

0.005778

0.003468

0.001159

0.004623

0.003428

0.001148

0.005707

0.004567

0.002288

0.000004

Expert5

0.000008

Expert 4

Experts

5

63.7

64.0

64.1

64.1

64.2

64.4

64.7

64.9

66.2

64.9

66.0

65.5

lectec

1

64.1

64.6

64.7

64.8

65.1

65.1

65.5

65.9

64.0

64.1

64.1

65.0

64.5

s

3

62.7

62.8

63.3

63.3

63.4

63.7

63.5

63.6

63.5

63.5

63.5

63.7

64.0

63.5

qunn

er

2

62.1

62.2

62.2

62.3

63.5

64.2

64.2

64.2

64.6

64.8

64.3

63.2

63.3

63.0

62.5

2

3

4

5

6

7

8

9

10

11

12

13

14

NumberofTotalProxies/TotalExperts

assignments decrease more gradually in the gallery set compared to the probe set. Additionally, each expert provides subtly different weightings to the same frames, enhancing the overall representation of the face. This analysis demonstrates that the ProxyFusion model adeptly discerns the informativeness of facial features in an interpretable yet effective manner.

Effect of Number of Proxies: In Figure 4, we present the face verification performance in terms of TAR@FAR= 10 -2 , plotted against the number of proxies K (X-axis) and the selected number of experts ̂ K (Y-axis). The results 1 indicate that increasing the number of proxies generally enhances model performance. However, this improvement plateaus around 10-12 proxies, and further increasing the number of proxies beyond 13 leads to overfitting. This overfitting likely occurs because the model, with a higher number of experts during training, becomes more prone to fitting the training distribution and training subjects too closely, thus failing to generalize well to unseen subjects.

Effect of Number of Selected Ex-

perts: Similarly, Figure 4 also demonstrates that an increase in the number of selected experts generally enhances performance. However, this improvement comes at the expense of longer inference times. As the number of selected experts increases, the number of parameters required during inference also rises, leading to increased

inference time. Our observations indicate that setting the number of selected experts, ̂ K , to 4 achieves performance close to the best model while significantly reducing computation time.

Contribution of Proxy Loss L Proxy : In Table 1 we present the contribution of the L Proxy to the overall performance. We observe that though the contribution of L Proxy is marginal when the number of selected experts is small, there are performance improvements at modest N . We believe this is due to the decorrelation of proxies, which results in extraction of diverse information from the feature set.

3.4 Inference Time and Computational Cost:

To validate our algorithm's claimed linear time complexity we perform GFLOPs analysis of our method against increasing sizes of the featureset. As can be observed from Fig. 5, we start with small feature-set size of 100 and scale it up to 1 million. Our fusion model's (in blue) GLOPs increases linearly with the increasing number of features in the set. On contrary, GFLOPs increase quadratical for models like CoNAN [5] and [9]. Moreover, these with single-shot input of large feature-sets to these models, the memory footprint increases quadratically due to intra-set attention

ComparisonofGFLOPsforDifferentMethods

0.0

0.80358

Mean

0.47369

NAN

0.5

MCN

CoNAN

1.0

CAFace

Log(GFLOPs)

Proxy

1.5

2.0

0.00673

2.5

0.00261

0.00132

-3.0

0.00066

3.5

100

5001000

50001000020000

50000100000

1000000

which leads to out-of-memory issues for these models beyond an N of 21000. These experiments are performed on 1 NVIDIA A6000 averaged over 3 runs.

3.5 Comparison to SotA methods

In table 3 we compare our results with the stateof-the-art (SOTA) in face verifcation using the RetinaFace face detector and Adaface Features on BTS3.1 Dataset. In table 4 we present results using MTCNN face detector and Adaface and Arcface feature extractors on BTS3.1. As mentioned earlier, for fair comparison to methods such as [9], in first half of table 4 we train our model on WebFace dataset using the precomputed Adaface features released by [9].

As can be observed from table 3 and 4 our method significantly outperforms other linear time methods such as NAN[20] and MCN[19] while providing a significant jump over naive

∗ Inter. denotes whether the method depends on intermediate features for feature fusion. If a method requires intermediate features, it is not compatible with legacy templates.

averaging or Global Average Pooling (GAP) [11]. Our method also achieves on-par performance to quadratic time complexity methods such as CoNAN [5] and CAFace [9]. When trained on WebFace, our approach outperforms CAFace [9] on most FAR thresholds without utilzing intermediate features unlike [9]. When compared to CoNAN [5], our method performs outperforms it on Face included control setting while being significantly faster (See Fig. 5, having much lower runtime memory requirements, and comparable inference time parameters (13.6M for CoNAN and 12.4M for ProxyFusion). Similar trend can be observed on DroneSURF dataset (refer Table 2). All SotA experiments are performed several times and we present the average results. We calculated Standard Errors of the Mean (SEM) across the multiple observations and found the SEM to very low in the range of 0.008-0.09 showing the statistical significance of the experiments.

4 Related Work

Methods based on Intermediate Feature Maps and Metadata: Leveraging intermediate feature maps for feature fusion, methods like the nonlocal neural network [18] and RSA [12] show promise in modeling intra-set relationships through detailed spatial analysis. These methods utilize intermediate feature maps U i of size C m × H × W to capture complementary information and refine spatial relationships. CAFace [9] addresses the computational issue of these approaches with a two-stage feature aggregation method, using these feature maps as style information. This involves assigning N inputs to M global cluster centers and fusing clustered features. CAFace generates an affinity map with N 2 complexity [15]. The requirement for intermediate feature maps makes these methods incompatible with legacy biometric systems lacking precomputed intermediate features. In contrast, our method relies solely on the penultimate feature representation, typically used for generating biometric templates. Furthermore, Metadata extracted from face images can serve as an alternative source for inferring face quality. [14] introduced a feature aggregation method that integrates metadata (e.g., yaw, pitch, face size) and utilizes a siamese network to determine the relative quality correlations among face images in a set. Additionally, metadata can facilitate fine-grained matching across galleries and probes by identifying faces with similar characteristics. For instance, TADPool [13] adapted both probe and gallery features by considering attributes like face yaw and roll, employing an attention block to pool probe features based on their compatibility with selected gallery features.

Linear Time Feature Fusion Methods: Previous works have proposed aggregation methods with linear time complexity against the number of features in set. In [20], a network architecture with two cascaded attention blocks is presented, which evaluates the significance of features in an image set and uses these scores for feature aggregation. Recently, [15] employed a differentiable coreset selection approach, using learned metrics and a Gumbel-Softmax distribution to optimize the selection of a small, representative subset, which is then enriched via self and cross-attention mechanisms thereby reducing computational complexity by decreasing set-size. [19] introduces a multicolumn network that assigns weights to images within a set based on their visual quality, assessed through a self-quality assessment module. This network then dynamically adjusts these weights according to each image's relative content quality compared to others in the set.

Conclusion

We address key challenges in feature fusion: (i) real-time inference, (ii) cross-distribution matching, (iii) generalization to large feature sets, and (iv) compatibility with legacy feature stores. Our proxy-relevancy based approach for expert selection, combined with a feature pooling approach, ensures robust multifaceted feature aggregation with minimal inference time and no need for extra metadata or intermediate features. Ablation studies demonstrate our method's superior performance and efficiency, outpacing existing approaches in both speed and effectiveness for real-time applications.

Potential Negative Societal Impacts: We are meticulous about training or testing our models only on datasets with approved IRB and consent from involved human subjects. This is why we do not use IJB-B/C datasets, as they contain web-sourced faces without consent or IRB approval. Conversely, datasets like IARPA BRIAR and [6] which we utilize are collected under IRB with subject consent.

Limitations: While our method effectively discerns feature informativeness via sparse experts, relying solely on feature-set distribution statistics overlooks fine-grained intra-set relationships. We can enhance this by hierarchically merging them with intra-set methods, achieving further gains with limited computational cost.

Acknowledgments

This research is based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via 2022-21102100001. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

This works is also partially supported by the the National AI Research Institutes program by the National Science Foundation (NSF) and the Institute of Education Sciences (IES), U.S. Department of Education, through Award #2229873. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF, the IES, or the U.S. Department of Education.

References

David Cornett, Joel Brogan, Nell Barber, Deniz Aykac, Seth Baird, Nicholas Burchfield, Carl Dukes, Andrew Duncan, Regina Ferrell, Jim Goddard, et al. Expanding accurate person recognition to new altitudes and ranges: The briar dataset. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 593-602, 2023.

Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. Retinaface: Single-shot multi-level face localisation in the wild. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5202-5211, 2020.

Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019.

William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research , 23(120):1-39, 2022.

Bhavin Jawade, Deen Dayal Mohan, Dennis Fedorishin, Srirangaraj Setlur, and Venu Govindaraju. Conan: Conditional neural aggregation network for unconstrained face feature fusion. In 2023 IEEE International Joint Conference on Biometrics (IJCB) , pages 1-9, 2023.

Isha Kalra, Maneet Singh, Shruti Nagpal, Richa Singh, Mayank Vatsa, and P. B. Sujit. Dronesurf: Benchmark dataset for drone-based face recognition. In 2019 14th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2019) , pages 1-7, 2019.

Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems , 33:18661-18673, 2020.

Minchul Kim, Anil K. Jain, and Xiaoming Liu. Adaface: Quality adaptive margin for face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 18750-18759, June 2022.

Minchul Kim, Feng Liu, Anil K Jain, and Xiaoming Liu. Cluster and aggregate: Face recognition with large probe set. Advances in Neural Information Processing Systems , 35:36054-36066, 2022.

Minchul Kim, Yiyang Su, Feng Liu, Anil Jain, and Xiaoming Liu. Keypoint relative position encoding for face recognition. arXiv preprint arXiv:2403.14852 , 2024.

Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. arXiv preprint arXiv:1312.4400 , 2013.

Xiaofeng Liu, Zhenhua Guo, Site Li, Lingsheng Kong, Ping Jia, Jane You, and BVK Kumar. Permutation-invariant feature restructuring for correlation-aware image set-based recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4986-4996, 2019.

Nishant Sankaran, Deen Dayal Mohan, Sergey Tulyakov, Srirangaraj Setlur, and Venugopal Govindaraju. Tadpool: Target adaptive pooling for set based face recognition. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021) , pages 1-8, 2021.

Nishant Sankaran, Sergey Tulyakov, Srirangaraj Setlur, and Venu Govindaraju. Metadatabased feature aggregation network for face recognition. In 2018 International Conference on Biometrics (ICB) , pages 118-123, 2018.

Gil Shapira and Yosi Keller. Facecoresetnet: Differentiable coresets for face set recognition. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 4748-4756, 2024.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538 , 2017.

Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2018.

Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2018.

Weidi Xie and Andrew Zisserman. Multicolumn networks for face recognition. arXiv preprint arXiv:1807.09192 , 2018.

Jiaolong Yang, Peiran Ren, Dongqing Zhang, Dong Chen, Fang Wen, Hongdong Li, and Gang Hua. Neural aggregation network for video face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4362-4371, 2017.

Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE signal processing letters , 23(10):14991503, 2016.

Zheng Zhu, Guan Huang, Jiankang Deng, Yun Ye, Junjie Huang, Xinze Chen, Jiagang Zhu, Tian Yang, Dalong Du, Jiwen Lu, et al. Webface260m: A benchmark for million-scale deep face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly state the contributions, and our paper is about these contributions. Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Yes, we discuss our limitations in the conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We include all of the information necessary to reproduce our experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will publicly release the code and trained models. The data can be requested from IARPA and IIT Jodhpur.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide all training and evaluation details in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Comparison to baseline methods like mean and state of the art methods provides information about the statistical relevance of our experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We include these details in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We follow all of the guidelines in the NeurIPS code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have included this discussion in our paper.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [No]

Justification: There are no particular safe guards, yet we only train our models on datasets with approved IRB and consent from involved subjects. Therefore there is a limited risk of misuse of our pretrained models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We explicitly state in what existing assets we are using, and properly attribute them.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We document the new assets we have created and we will release the documents with the assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [No]

Justification: The datasets we are using are provided external entities. We sign required agreements and licenses to acquire these datasets.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [Yes]

Justification: Yes, all of the datasets we are using have IRB approvals. See potential negative societal impact section of our conclusion.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel face recognition framework that aims to address the challenges of feature fusion in long-range, low-resolution scenes. The authors propose a linear time complexity approach that is compatible with traditional biometric template databases and does not require additional metadata or intermediate feature maps.ProxyFusion utilises a set of learnable agents to implicitly represent potential facial attributes and selects the most relevant experts to focus on the feature set, thus significantly reducing inference time and parameters. The method is order and size invariant, making it very robust for real-time applications and large probe sets. Extensive experiments on the IARPA BRIAR BTS3.1 and DroneSURF datasets demonstrate the superiority of ProxyFusion in unconstrained long-range face recognition settings.\n\nSTRENGTHS:\n1. In this paper, the authors propose a novel feature aggregation framework to address key challenges in face recognition, especially in low-resolution and long-range scenes. \n2. The runtime of ProxyFusion's feature aggregation method is linearly related to the number of features, which is very effective for real-time reasoning and processing large probe sets.\n\nWEAKNESSES:\n1. The optimization of the sparse experts is ambiguous. In the section of methods, the authors do not provide any details on how to optimize the sparse experts. \n2. The paper mentions that relying solely on feature-set distribution statistics might overlook fine-grained intra-set relationships. This suggests that the model may not capture all the nuances present within a set of features.\n3. The performance of ProxyFusion, like any face recognition system, is likely to depend heavily on the quality of the input data. Poor quality inputs could degrade performance.\n4. The paper does not mention the model's robustness to adversarial attacks, which is an important consideration in the field of biometrics and face recognition.\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nPlease refer to the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel feature fusion technique for unconstrained face recognition via sparse experts. The authors propose a expert network selection mechanism using the $k$ proxies, $p_j$, and all the $N$ precomputed face features, $f_i$. The top $\\hat{k}$ selected experts are used by the 3d set centers to produce discriminative features per expert that finally forms the template. Two losses are proposed, a supervised contrastive loss ($L_{id}$) to ensure high interclass and low intra class separability between probe and gallery, and a proxy loss ($L_{proxy}$) to ensure the experts learn mutually exclusive information. The authors validate their method primarily on DroneSurf and BRIAR dataset on different TAR@FAR=thresholds. They also provide visualization for the learned experts and GFLOPS analysis to support the linear time complexity claim.\n\nSTRENGTHS:\n* The application of mixture of experts acts as a way of diffentiable sampling to select the right experts and intrinsically, the right feature for aggregation. \n* The paper presents a strong evidence emperically, especially with the BRIAR dataset, which is a highly noisy and a challenging dataset. \n* The authors detail the contribution on all aspects that includes - interpretability of weights for set-centers, effects of number of proxies, effects of expert selection, and finally, the proxy loss.\n\nWEAKNESSES:\n### 1. The presentation of the method end experiment section could be greatly improved. There are some texts that is unclear that certainly affects the final rating of the paper. \n\na.  Line 97, What are proxies in this context and how are they constructed initially? Are they randomly initialized? Or are they some form of router network? The paper also seems to use proxies/learnable queries (line 68) interchangeably which is further confusing.\n\nb.  Line 125, Divergences represent the dissimilarity between two distributions. Can the authors please expand on how the dot product between two feature vector is a divergence?\n\nc.  Table 1, What dataset has this been tested on? Is this where the authors employed $\\hat{K}=4$ as the optimum experts?\n\nd.  What is the success rate of both face detectors for probe images? At 500m with several noise factors, these standard detectors might encounter higher degree of failure.\n\ne.  Figure 4 is a bit confusing as to what exactly $x-axis$ is, as the experts/proxies are used interchangeably.\n\nf.  It would also add value if authors are able to add TAR@FAR values per each distance category for probe BTS 3.1. I am assuming this would involved categorizing the results which should not be a completely new experiment.\n\n\n### 2. The paper does not include enough description to make it reproducible.\n\na.  Line  141, what were the chosen $L$ and $U$ values in the experiments?\n\nb.  Line 143, what was the suitable $M$ found to be?\n\nc.  Line 146, pad them with what value?\n\nd.  What is the final $d$ and $d'$ ?\n\n\n### 3. I also have few minor corrections to suggest to further improve the quality of the paper.\n\na.  Paragraph 139 and 143 seems to convey the same meaning. Can be combined into one.\n\nb. SCL equation for $L_{id}$ uses the transpose $T$ inconsistently.\n\nc. Line 187, I think it is supposed to be \"For training *ProxyFusion*...\"\n\nd. Other errors:\n\n * Typo in line 90, *refere*\n* Line 125, 'that is used to compute...'\n* Line 154, \"primary goal bridge....\"\n* Line 282, \"these with single shot...\"\n\nQUESTIONS:\nI have mentioned most of the questions in the weakness section. The most important ones to address would be 1 and 2.\n\nLIMITATIONS:\nThe authors adequately address the limitations about the proposed methods and use of only IRB approved datasets for experiementation. \n\nFor future work, it would be interesting to compare the method against [1], which also attempts to achieve similar goals. This however, is a suggestion, and in no way affects my final rating for this conference.\n\n\n[1] Shapira, Gil, and Yosi Keller. \"FaceCoresetNet: Differentiable Coresets for Face Set Recognition.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 5. 2024.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a novel approach for face feature fusion, addressing typical scenarios such as low-quality probes and high-quality or different domain gallery sets of faces. They describe alternative approaches and conclude that these are currently limited because they are either not compatible with legacy biometric databases or have real-time performance issues.\n\nThe authors provide a method for feature aggregation through Sparse Experts. The overall pipeline includes three steps: Feature Extraction -> Expert Network Selection -> Sparse Expert Network Feature Aggregation. Each step is described in detail.\n\nOptimization is done using a two-component loss: Loss for the identification objective + proxy loss. The proxy loss aims to make different proxies attend to different facial characteristics. The authors present experimental justifications for the second loss in the experimental section.\n\nThe authors describe the experiments conducted. Two datasets were used for training: BRIAR Research Set 3 and WebFace 4M. Evaluation was performed on BTS 3.1 and DroneSURF. The authors provide all details about these steps. They also present visualizations showing that each expert assigns higher weights to faces with more discriminative identity information and minimal weights to poor-quality images. Experimentally, they found that the optimal setup is to use 4 Selected Experts and 12 total Proxies/Experts.\n\nThe authors compared their method with various state-of-the-art alternatives and demonstrated that their method either produces strong results, achieving SOTA performance, or performs on par with quadratic-time methods, despite being a linear-time method.\n\nAdditionally, the authors justify that their method is feature order and set length invariant.\n\nSTRENGTHS:\nThe idea of the method is quite clear. The algorithm has a linear time complexity, while alternatives with similar quality have quadratic time complexity. The method is invariant to set length and feature order. It can be applied even to legacy databases, which is highly important.\nThe paper is clearly written, and the results seem to be very promising. The visualizations are well drawn, clear, and insightful.\n\nWEAKNESSES:\nThe authors commit to releasing the code but did not attach it. While the method is clear and straightforward, the overall implementation can be complex, which could hinder reproducibility. Therefore, I believe the code is a crucial part of this work and should be shared along with the article, especially since this work does not involve heavy neural models or long training procedures.\n\nWithout a full code review and the ability to run it, there may not be sufficient proof of reproducibility. Additionally, the authors' response to Question 7, 'Experiment Statistical Significance,' may not fully address the issue, as comparison with alternative methods alone is not evidence of statistical significance.\n\nFurthermore:\n\n> (ii) DroneSURF: This dataset includes Active and Passive Surveillance settings, each with 100 videos and 786,000+ face annotations. Following [12], we split subjects randomly: 60% (34 identities) for training/validation, 40% (24 identities) for testing. The dataset has 200 videos of 58 subjects, over 411,000 drone-captured frames. Results are based on the 203 video-wise identification protocol.\n\nIt is unclear how the random split can be compared to other methods. If the split was done as in [12], this information should have been provided. Additionally, I suggest that the splits be released so future authors can make clear comparisons.\n\nQUESTIONS:\n1. Could the authors provide the code and necessary weights at this review stage to enhance clarity and reproducibility?\n2. Could the authors provide more detailed information about the DroneSURF random splits or share the splits themselves?\n3. Could the authors clarify about the statistical significance?\n\nLIMITATIONS:\nThere is one limitation described:\n\n> While our method effectively discerns feature informativeness via sparse experts, relying solely on feature-set distribution statistics overlooks fine-grained intra-set relationships.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method for fusing face features extracted from sets of images.\nSeveral proxies are defined that are trained to be specialized for certain aspects of faces, and the most relevant for a given set of face images are obtained.\nImportance weights for different samples are estimated for each selected proxy, and a template stores a combination of weighted feature averages.\nExperiments on challenging datasets with a wide variety of viewing conditions indicate largely improved performance with linear time complexity in the number of samples per set.\n\nSTRENGTHS:\n+ The paper is written well, and the motivation of the proposed method is clearly stated.\n+ The results show improved performance over state-of-the-art feature fusion approaches on difficult datasets.\n+ Ablation studies show the importance of most parts of the pipeline, and provide insights on relevant parameters of the model.\n+ Besides a few minor mistakes, there are no apparent errors in the math.\n\nWEAKNESSES:\n1. There are a few important details of the proposed algorithm missing in the paper:\n\n   a) One part of the proposed method, i.e., the details of the expert network E_j are missing in the paper -- how large are the layers? \n\n   b) The proxy loss is defined over a set of negative samples. There is no word spent on how to select such negative samples -- yet, the selection is of high importance. Also, this loss includes a lambda value that is not detailed.\n\n   c) Both gallery and probe templates finally consists of a concatenation of several averaged features. The authors need to clarify how the final score between these sets of templates is computed.\n\n\n2. Some parts of the paper include non-standard solutions that should be considered to update:\n\n   a) In order to compute valuable means and standard deviations in section 2.2, the features would need to reside in Euclidean space. However, modern deep FR networks optimize for cosine space, and deep features typically do not follow Euclidean distributions. Why would this be a good idea?\n\n   b) Relatedly, the expert networks obtain mean, standard deviation and proxy centers. It is not shown in the paper that the variances provide any improvements over only using the means. A better handling of mean and proxy center than a simple concatenation is likely possible.\n\n   c) The proxy center projections v are selected to be equidistant in Euclidean space, while they result from a projection that rather lives in cosine space. It is not clear where the expression is coming from or why such a difficult expression needs to be defined, a simple set of K-dimensional basis vectors e_i would likely be sufficient.\n\n   d) The models used for feature extraction are not well-suited for low-resolution and highly distorted faces. Better models will likely improve the utility of the feature fusion approach.\n\n\n3. There are some improvements on the paper:\n\n   a) The introduction is slightly repetitive and could be shortened.\n\n   b) At several places in the paper, it is mentioned that the method is compatible with existing template storage solutions. The practical relevance for such compatibility is questionable. In a typical template storage, a single template is stored as an aggregation over several samples already. The authors should not overemphasize this characteristic of their method.\n\n   c) The authors claim their algorithm to have a time complexity of O(N). However, the complexity is rather O(N*K). Depending on the choice of K, this might be a relevant deviation from O(N).\n\n   d) In line 138 and following, the authors write about batch creation strategies. It is not entirely clear if this is applied during test time as well, or whether this is a training time issue only. Finally, the authors claim to include diverse features, but random selection does not ensure diversity.\n\n   e) The equation for the standard deviation in section 2.2 is slightly off, the factor should be 1/(N-1).\n\n   f) In line 170, the scores s_ik should make use of the projected proxies Wp, not the proxies themselves.\n\n   g) The figures should be included as vector graphics to allow zooming. Especially figure 3 is of too low quality to read.\n\n   h) In the caption of figure 5, N represents the total number of probe features. In section 2.2, N is the number of images for one identity, while in section 2.3, N represents the number of identities.\n\n   i) The paper should be carefully checked for spelling mistakes.\n\nQUESTIONS:\nThe most important question is how negative samples are obtained for the proxy loss. The authors should elaborate on this. Also, the strategy on handling several features per template needs to be revealed. Afterward, I am happy to increase my rating.\n\nLIMITATIONS:\nThe authors took special care about using only datasets with IRB-consented data. They should be aware, however, that the pre-trained networks that they exploit for feature extraction have been trained on web-scraped datasets which did not include IRB-approved subjects.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}